Flink Scheduled JDBC Source

Question

I have the following definition of the JDBC source in Apache Flink.

    val jdbcSource = JdbcSource.builder<LoggedInEvent>()
    .setDBUrl("jdbc:postgresql://db:5432/postgres")
    .setSql("SELECT player_id, past_logins FROM user_initial_data")
    .setUsername("postgres")
    .setPassword("example")
    .setTypeInformation(TypeInformation.of(PlayerLoggedInEvent::class.java))
    .setResultExtractor { LoggedInEvent(it.getInt(1).toString(), it.getInt(2), Instant.now().toEpochMilli()) }
    .build()

val snapshotsStream = env.fromSource(jdbcSource, WatermarkStrategy.noWatermarks(), "LoggedInSnapshots")

Currently I'm experiencing two issues with this solution:

I can't schedule this to execute every N seconds, so is there any simple way to do it with existing tooling?
This is realted to #1, but this executes only once and job finishes. I want this to be scheduled and run continuously within the same job.

David Anderson · Accepted Answer · 2025-04-03 11:33:28Z

1

Flink does not provide this sort of scheduling, or polling.

On the other hand, Kafka Connect does this support this: https://docs.confluent.io/kafka-connectors/jdbc/current/source-connector/source_config_options.html

answered Apr 3 at 11:33

David Anderson

44.3k4 gold badges41 silver badges73 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

ashur Apr 3 at 11:46

Would it make sense to create my own source with scheduling based on current implementation of JdbcSource?

Collectives™ on Stack Overflow

Flink Scheduled JDBC Source

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related