Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions crates/feldera-types/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -581,6 +581,19 @@ pub struct SyncConfig {
#[schema(default = default_retention_min_age)]
#[serde(default = "default_retention_min_age")]
pub retention_min_age: u32,

/// A read-only bucket used as a fallback checkpoint source.
///
/// When the pipeline has no local checkpoint and `bucket` contains no
/// checkpoint either, it will attempt to fetch the checkpoint from this
/// location instead. All connection settings (`endpoint`, `region`,
/// `provider`, `access_key`, `secret_key`) are shared with `bucket`.
///
/// The pipeline **never writes** to `read_bucket`.
///
/// Must point to a different location than `bucket`.
#[serde(default)]
pub read_bucket: Option<String>,
}

fn default_pull_interval() -> u64 {
Expand All @@ -603,6 +616,15 @@ Standby mode requires `start_from_checkpoint` to be set.
Consider setting `start_from_checkpoint` to `"latest"`."#.to_owned());
}

if let Some(ref rb) = self.read_bucket
&& rb == &self.bucket
{
return Err(
"invalid sync config: `read_bucket` and `bucket` must point to different locations"
.to_owned(),
);
}

Ok(())
}
}
Expand Down
21 changes: 19 additions & 2 deletions docs.feldera.com/docs/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,19 +8,35 @@ import TabItem from '@theme/TabItem';

# Changelog


<Tabs>
<TabItem className="changelogItem" value="enterprise"
label="Enterprise">


## Unreleased

Backward-incompatible Delta Lake output connector change. The new `max_retries` setting configures
the number of times the connector retries failed Delta Lake operations like writing Parquet files
and committing transactions. The setting is unset by default, causing the connector to retry
indefinitely. This behavior prevents data loss due to transient or permanent write errors.

### Checkpoint sync: `read_bucket` and checkpoint loading priority

The checkpoint sync configuration now supports a `read_bucket` field — a read-only
fallback bucket used to seed a pipeline's initial state. The pipeline **never writes**
to `read_bucket`.

Checkpoint sources are now resolved in priority order:
1. **Local checkpoint** — if the pipeline already has a local checkpoint, it resumes
from that without contacting any remote bucket.
2. **`bucket`** — the pipeline's own S3-compatible bucket. If a checkpoint is found
here, it is used and `read_bucket` is ignored.
3. **`read_bucket`** — consulted only when both local storage and `bucket` are empty.
This allows a new pipeline to seed from another pipeline's checkpoint, avoiding
a full backfill. `read_bucket` must point to a different location than `bucket`.

Refer to the [checkpoint sync documentation](/pipelines/checkpoint-sync#checkpoint-resolution-priority) for details.


## v0.263.0

Added connector error list to input/output connector stats.
Expand Down Expand Up @@ -334,4 +350,5 @@ import TabItem from '@theme/TabItem';
<TabItem className="changelogItem" value="oss" label="Open Source">
[Release notes](https://github.com/feldera/feldera/releases/) for the Open Source edition can be found on github.
</TabItem>

</Tabs>
Loading
Loading