Skip to content

FLINK-37686 - Do not fail on NOT_OWNED deleted files#28801

Open
francis-a wants to merge 1 commit into
apache:masterfrom
francis-a:bugfix-FLINK-37686-dont-fail-on-deleted-file-checks
Open

FLINK-37686 - Do not fail on NOT_OWNED deleted files#28801
francis-a wants to merge 1 commit into
apache:masterfrom
francis-a:bugfix-FLINK-37686-dont-fail-on-deleted-file-checks

Conversation

@francis-a

Copy link
Copy Markdown

What is the purpose of the change

This PR aims to fix the issue documented in https://issues.apache.org/jira/browse/FLINK-37686 where remote objects cleaned up by the JM cause checkpoint failures even when the object is marked as NOT_OWNED.

Brief change log

  • Added a FileNotFoundException catch block to ForStFlinkFileSystem::listStatus
  • In this catch block we verify the object is NOT_OWNED
  • If true, we assume the JM removed this object and delete it using the FileMappingManager
  • Tests have been added to cover this case as well as ensuring the exception still bubbles up if the object is not NOT_OWNED

Verifying this change

This change added tests and can be verified as follows:

  • Added two new tests to ForStFlinkFileSystemTest, testListStatusDropsDeletedNotOwnedEntryAfterGiveUpOwnership and testListStatusStillFailsForDeletedDbOwnedFile
  • These tests cover the NOT_OWNED and verify the removed object is also removed from the FileMappingManager, the other test verifies the baseline hasn't changed for other object types

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
  • The serializers: (yes / no / don't know)
  • The runtime per-record code paths (performance sensitive): (don't know)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
  • The S3 file system connector: (no)

Documentation

  • Does this pull request introduce a new feature? (no)

Was generative AI tooling used to co-author this PR?
  • Yes

Generated-by: [Claude Opus]

@flinkbot

flinkbot commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@AlexYinHan

Copy link
Copy Markdown
Contributor

@francis-a Thanks for the PR. LGTM. The orphan files deleted by the JM are abandoned by ForSt. They should be removed and not cause failures during a full scan. @Zakelly Could you take a look?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants