티스토리 뷰

공부

[airflow] retry & failover

승가비 2023. 1. 28. 04:22
728x90

https://stackoverflow.com/questions/54598163/airflow-task-failure-retry-workflow

 

Airflow Task failure/retry workflow

I have retry logic for tasks and it's not clear how Airflow handles task failures when retries are turned on. Their documentation just states that on_failure_callback gets triggered when a task fa...

stackoverflow.com

https://github.com/apache/airflow/discussions/29201

 

[failover] restart from before step. · Discussion #29201 · apache/airflow

I used emr step sensor. I often get a hive sql related task failure alarm. If you do it again immediately, the task succeeds. So I want to re-run the retries option if it fails by calling emr step ...

github.com

 

728x90
저작자표시 비영리

'공부' 카테고리의 다른 글

[docker] stop & rm & run  (0) 2023.01.31
[airflow] airflow.operators.trigger_dagrun  (0) 2023.01.28
[log4j] SLF4J: Class path contains multiple SLF4J bindings  (0) 2023.01.28
[spark] group by -> reduce by  (0) 2023.01.28
[slack] multiple image upload  (0) 2023.01.28
댓글