티스토리 뷰

공부

[airflow] loop dags

승가비 2022. 12. 28. 00:11
728x90
variables = ['first', 'second', 'third']

def run_dag_task(variable):
  task = dag_task(variable)
  return task
task_arr=[]
task_arr.append(run_dag_task(variable[0]))

for variable in variables[1:]:
  task=run_dag_task(variable)
  task_arr[-1]>>task
  task_arr.append(task)

https://stackoverflow.com/questions/70002086/how-to-run-tasks-sequentially-in-a-loop-in-an-airflow-dag

 

How to run tasks sequentially in a loop in an Airflow DAG?

Say I have this snippet of an example DAG: variables = ['first', 'second', 'third'] def run_dag_task(variable): task = dag_task(variable) return task for variable in variables: run_dag_task(

stackoverflow.com

 

728x90
저작자표시 비영리

'공부' 카테고리의 다른 글

[benchmark] `ORC` vs `Parquet`  (0) 2022.12.28
[spark] sql `SHOW PARTITIONS` & `SELECT * FROM db_name."table_name$partitions" ORDER BY column_name DESC`  (0) 2022.12.28
[spark] --name {appName}  (0) 2022.12.28
[sh] grep exclude word `grep -v 'exclude_word' file`  (0) 2022.12.28
[spark] text files  (0) 2022.12.20
댓글