Airflow uses mysql database and LocalExecutor concurrent scheduling
This article mainly introduces "airflow uses mysql database and LocalExecutor concurrent scheduling". In daily operation, I believe many people have doubts about airflow using mysql database and LocalExecutor concurrent scheduling. Xiaobian consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the doubts of "airflow using mysql database and LocalExecutor concurrent scheduling". Next, please follow the editor to study!
Mysql-airflow executes on mysql
Create database airflow;-create a database
GRANT all privileges on airflow.* TO 'airflow'@'%' IDENTIFIED BY' 123456login;-- Grant all permissions of the database airflow to the airflow user with a password of 123456 and the user can log in to any IP segment
FLUSH PRIVILEGES;-refresh permissions
Set explicit_defaults_for_timestamp = 1
Modify airflow configuration file airflow.cfg
Sql_alchemy_conn = mysql://airflow:123456@mysqlIP/airflow
Initialize the database
Airflow initdb
If you have previously initialized with sqllite, you can use airflow resetdb
Airflow-LocalExecutor
Modify the executor to schedule tasks concurrently with airflow
Modify the configuration file airflow.cfg
Executor = LocalExecutor
Restart scheduler (default concurrent scheduling 16)
Distributed problem
1. Need to use synchronization method to solve the synchronization problem between dags directory and worker-- to be solved
two。 To solve the single point problem of scheduler, please refer to apache-airflow scheduling system https://blog.csdn.net/somezz/article/details/82749495
At this point, the study on "airflow uses mysql database and LocalExecutor concurrent scheduling" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!
