티스토리 뷰

공부

[airflow] helm chart

승가비 2023. 4. 17. 20:14
728x90

https://artifacthub.io/packages/helm/airflow-helm/airflow/8.7.0

 

airflow 8.7.0 · airflow-helm/airflow-helm

Airflow Helm Chart (User Community) - the standard way to deploy Apache Airflow on Kubernetes with Helm

artifacthub.io

airflow:
  airflow:
    image:
      repository: 123123.dkr.ecr.ap-northeast-2.amazonaws.com/airflow
      tag: base-0.0.0-230417174017
    executor: KubernetesExecutor
    config:
      AIRFLOW__CORE__LOAD_EXAMPLES: false
      AIRFLOW__LOGGING__REMOTE_LOGGING: true
      AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: s3://bucket/airflow/logs
      AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID: aws_default
      AIRFLOW__WEBSERVER__EXPOSE_CONFIG: false
      AIRFLOW__WEBSERVER__DEFAULT_UI_TIMEZONE: Asia/Seoul
      AIRFLOW__WEBSERVER__BASE_URL: https://airflow.com
    connections:
      - id: aws_default
        type: aws
        login: ${ACCESS_KEY_ID}
        password: ${SECRET_ACCESS_KEY}
        extra: |
          {
            "region_name": "ap-northeast-2"
          }
    variables:
      - key: key1
        value: value1
    connectionsTemplates:
      ACCESS_KEY_ID:
        kind: secret
        name: secret
        key: AWS_ACCESS_KEY_ID
      SECRET_ACCESS_KEY:
        kind: secret
        name: secret
        key: AWS_SECRET_ACCESS_KEY
    extraEnv:
      - name: AWS_DEFAULT_REGION
        value: ap-northeast-2
      - name: AWS_ACCESS_KEY_ID
        valueFrom:
          secretKeyRef:
            name: secret
            key: AWS_ACCESS_KEY_ID
      - name: AWS_SECRET_ACCESS_KEY
        valueFrom:
          secretKeyRef:
            name: secret
            key: AWS_SECRET_ACCESS_KEY
  externalDatabase:
    type: postgres
    host: {host}
    port: 5432
    database: airflow
    userSecret: secret
    userSecretKey: POSTGRES_USER
    passwordSecret: secret
    passwordSecretKey: POSTGRES_PASSWORD
    properties:
  web:
    replicas: 1
    service:
      type: NodePort
  dags:
    gitSync:
      enabled: true
      repo: https://github.com/seunggabi/airflow.git
      branch: main
      revision: HEAD
      syncWait: 60
  flower:
    enabled: false
  workers:
    enabled: false
  postgresql:
    enabled: false
  redis:
    enabled: false

https://github.com/airflow-helm/charts/issues/34

 

airflow logging to S3 · Issue #34 · airflow-helm/charts

We have been encountering some issues with the logging to S3. Seems like in some cases where the dag fails the log is missing as well and we get the following error. Has anyone come across this iss...

github.com

https://github.com/seunggabi/pr-airflow-helm-charts

 

GitHub - seunggabi/pr-airflow-helm-charts: The User-Community Airflow Helm Chart is the standard way to deploy Apache Airflow on

The User-Community Airflow Helm Chart is the standard way to deploy Apache Airflow on Kubernetes with Helm. Originally created in 2018, it has since helped thousands of companies create production-...

github.com

https://airflow.apache.org/docs/helm-chart/stable/parameters-ref.html#database

 

Parameters reference — helm-chart Documentation

 

airflow.apache.org

https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/monitoring/log-persistence.md#s3-bucket

 

GitHub - airflow-helm/charts: The User-Community Airflow Helm Chart is the standard way to deploy Apache Airflow on Kubernetes w

The User-Community Airflow Helm Chart is the standard way to deploy Apache Airflow on Kubernetes with Helm. Originally created in 2018, it has since helped thousands of companies create production-...

github.com

https://github.com/airflow-helm/charts/issues/413

 

edit connection causes page crash · Issue #413 · airflow-helm/charts

What is the bug? When I click on edit connection, I get the following exception: Something bad has happened. Please consider letting us know by creating a bug report using GitHub. Python version: 3...

github.com

https://todaycodeplus.tistory.com/52

 

Airflow - Airflow on kubernetes(K8S) 설치 - 1

1. Airflow를 도입하게 된 이유 - Azure Functions (Serverless SaaS)를 이용해서 데이터를 처리하는 과정에서 적은 리소스로 많은 작업을 처리해야 하는 상황이 발생 당시 3.5Gb의 적은 메모리 용량으로 다양

todaycodeplus.tistory.com

https://airflow.apache.org/docs/helm-chart/stable/production-guide.html#extending-and-customizing-airflow-image

 

Production Guide — helm-chart Documentation

 

airflow.apache.org

 

728x90
댓글