Airflow with Docker connect to a local Host PostgreSQL

Question

I am running airflow with docker.

I want to go query some data in my PostgresSQl which on my local host.

this is my connection in my dag:

def queryPostgresql():
    conn_string="dbname='datawarehouse' host='localhost' user='postgres' password='admin'"
    conn=db.connect(conn_string)
    df=pd.read_sql("select name,city from test",conn)
    df.to_csv('data.csv')
    print("-------Data Saved------")

I am adding a connection to airflow:

psycopg2.OperationalError: could not connect to server: Connection refused
    Is the server running on host "localhost" (127.0.0.1) and accepting
    TCP/IP connections on port 5432?

Is it possible to query in my PgSQL from my Airflow Docker? Should I instead install PgSQL in docker?

clarj · Accepted Answer · 2021-10-16 16:39:20Z

5

When you try to access localhost in Airflow, it's trying to connect to Postgres running on the Airflow container, which is not there. localhost from a container does not route to the Docker host machine by default.

Couple of options:

Connect to the Docker host with host.docker.internal instead of localhost
Run Airflow and Postgres in a Docker Compose network and connect via container name (psql etc.)

There are a few other methods with hosts files etc. but the above is likely your easiest options.

answered Oct 16, 2021 at 16:39

clarj

1,0536 silver badges15 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Airflow with Docker connect to a local Host PostgreSQL

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related