docs: note S3-compatible endpoints for s3_endpoint config - #1235
Draft
goanpeca wants to merge 1 commit into
Draft
Conversation
Signed-off-by: Gonzalo Peña-Castellanos <goanpeca@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
offline_store.s3.s3_endpointis passed straight through to the Hadoopfs.s3a.endpointproperty, so it already works with any S3-compatible endpoint and not only AWS. That is not obvious from the current documentation, which showss3.amazonaws.comas the only example and describes the setting as "Required if using AWS S3 Storage as an offline store."This adds one clarifying line in each of the two places the option is documented: the
feathr_config.yamlworkspace template and the environment variable table indocs/how-to-guides/feathr-configuration-and-env.md. Comments and documentation only. No behavior change, no new config keys, no new dependencies.Per CONTRIBUTING.md, documentation-only changes do not need an accompanying GitHub issue.
How was this PR tested?
There is no new code path to exercise, so testing was limited to confirming the edited files are still well formed and that the statement being added is accurate.
I confirmed
feathr_config.yamlstill parses withyaml.safe_loadand thatoffline_store.s3.s3_endpointstill resolves tos3.amazonaws.com, unchanged. I also confirmed the edited Markdown table row still has the same three columns as its neighbors. No links were added, so the broken-link checker is unaffected.The chain the new note describes:
client.pyreadsoffline_store__s3__s3_endpoint,_get_s3_config_stremits it asS3_ENDPOINTin the HOCON job config, andS3ResourceInfoSetter.setupHadoopConfigsets that value asfs.s3a.endpointon the Spark Hadoop configuration.Does this PR introduce any user-facing changes?
Documentation wording only. The
s3_endpointdefault and all runtime behavior are unchanged. Readers of either the config template or the environment variable reference now see that the endpoint can point at any S3-compatible service.