Stream Apache Airflow (3.0.2) Logs to Azure Blob

Question

here I'm trying to stream (pushed) my airflow logs to azure blob instead of keeping it in container folder.

Current Issue that I'm facing: even after putting everything as mentioned in official document and following online threads, nothing works for my project.

current folder structure:

project_03/
├── config/
├── dags/
│   └── hello_world_dag.py
├── logs/
├── plugins/
├── requirements.txt
├── Dockerfile
├── .env
└── docker-compose.yml

DockerFile contain:

FROM apache/airflow:3.0.2
COPY requirements.txt /
RUN pip install --no-cache-dir -r /requirements.txt

here is my docker-compose.yml code:

x-airflow-common:
  &airflow-common

  build: .
  environment:
    &airflow-common-env
    AIRFLOW__CORE__EXECUTOR: CeleryExecutor
    AIRFLOW__CORE__AUTH_MANAGER: airflow.providers.fab.auth_manager.fab_auth_manager.FabAuthManager
    AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
    AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow
    AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
    AIRFLOW__CORE__FERNET_KEY: ''
    AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
    AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
    AIRFLOW__CORE__EXECUTION_API_SERVER_URL: 'http://airflow-apiserver:8080/execution/'

    AIRFLOW__SCHEDULER__ENABLE_HEALTH_CHECK: 'true'

    _PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}

    AIRFLOW__LOGGING__REMOTE_LOGGING: 'true'
    AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID: 'azure_blob_conn'
    AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: 'wasb://[email protected]/airflow/logs'
    AIRFLOW_CONN_AZURE_BLOB_CONN: 'wasb://ccmytest:<account-key>@ccmytest.blob.core.windows.net' 
    AIRFLOW__LOGGING__LOGGING_LEVEL: 'INFO'

    AIRFLOW_CONFIG: '/opt/airflow/config/airflow.cfg'

not sure what I'm missing but every other config that mentioned online I have put everything, but still it's not pushing the logs. if any workaround for this issue, that will be really helpful.

This configuration works for me AIRFLOW__LOGGING__REMOTE_LOGGING: 'true' AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID: 'azure-blob-conn-id' AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: 'foldername-in-container' do not forget to add the blob storage connection in the airflow interface. — Krengifo
– Krengifo, Commented Jul 4 at 23:07

Joost Döbken · Accepted Answer · 2025-07-24 11:03:52Z

0

This configuration works for me:

- AIRFLOW__LOGGING__REMOTE_LOGGING=True
- AIRFLOW__LOGGING__LOGGING_LEVEL=INFO
- AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID=wasb_default
- AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER=wasb://logs
- AIRFLOW__AZURE_REMOTE_LOGGING__REMOTE_WASB_LOG_CONTAINER=<container-name>
- AIRFLOW_CONN_WASB_DEFAULT={"conn_type":"wasb","login":"<account-name>","password":"<account-key>"}

The odd part is the AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER MUST start with wasb meaning in our blob container end up with another wasb/ folder...

answered Jul 24 at 11:03

Joost Döbken

4,1576 gold badges50 silver badges96 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Stream Apache Airflow (3.0.2) Logs to Azure Blob

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related