Skip to content

Migrate Azure blob storage to the 1.0 Azure SDK - #6910

Open
dayaffe wants to merge 5 commits into
mainfrom
david-yaffe/azure-sdk-1.0-migration
Open

dayaffe wants to merge 5 commits into
mainfrom
david-yaffe/azure-sdk-1.0-migration

Conversation

@dayaffe

@dayaffe dayaffe commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Description

Migrates the Azure Blob Storage backend from the legacy 0.21 Azure SDK to the 1.0 SDK. Based on #6693 by @siva-abstract-security.

Fixes #6672: indexers using AKS workload identity lost storage access 24 hours after startup. azure_identity 1.0 re-reads the token file, so tokens keep refreshing.

The 1.0 SDK dropped shared key auth and create_credential(), so both are rebuilt here. Uploads now carry a CRC64 checksum that Azure verifies, and each download is a single streamed request.

Behavior changes:

  • The Azure CLI is only used with AZURE_CREDENTIAL_KIND=azurecli, not as a fallback.
  • AZURE_CLIENT_ID alone selects a user-assigned managed identity.
  • An unknown AZURE_CREDENTIAL_KIND is an error, and token credentials need an https endpoint.
  • AZURE_FEDERATED_TOKEN is replaced by AZURE_FEDERATED_TOKEN_FILE.

How was this PR tested?

Fixes #6672: azure_identity 1.0 re-reads the federated token file, so
workload identity no longer fails 24 hours after startup. Shared key
signing and credential selection, which 1.0 dropped, are rebuilt here.

Based on #6693 by Siva.

Co-authored-by: Siva <ssellathurai@abstract.security>
@dayaffe
dayaffe requested a review from a team as a code owner October 19, 2026 18:54
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-19T20:11:03.453301Z 347987d New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1686dd7e6e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread quickwit/quickwit-storage/src/object_storage/azure_blob_storage.rs Outdated
Comment thread quickwit/quickwit-storage/src/object_storage/azure_blob_storage.rs
Comment thread quickwit/quickwit-storage/src/object_storage/azure_credentials.rs
An empty range skipped the request, so a missing blob read as empty instead of NotFound.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 157da8cb44

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread quickwit/quickwit-storage/src/object_storage/azure_credentials.rs
Put Block rejects an id whose length differs from a blob's leftover uncommitted blocks, so a write after an interrupted 0.21 multipart upload of the same blob, such as a file-backed metastore.json, failed until those blocks expired.
@dayaffe

dayaffe commented Oct 19, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 👍

Reviewed commit: 1601ef5487

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

The SDK wraps failures such as DNS or Entra ID errors in a generic "non-transport error occurred which will not be retried", and only that message reached the storage error.
@dayaffe dayaffe added area:storage S3, GCS, Azure and local storage type:bug Fixes something that is broken labels Oct 22, 2026
@dayaffe
dayaffe requested a review from nadav-govari October 23, 2026 18:49

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:storage S3, GCS, Azure and local storage type:bug Fixes something that is broken

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Azure: indexer fails with storage error(kind=Unauthorized) exactly 24h after startup when using Workload Identity

1 participant