Skip to content

Backend: DynamoDB#106

Open
lzukowski wants to merge 16 commits intomainfrom
feature/dynamodb-backend
Open

Backend: DynamoDB#106
lzukowski wants to merge 16 commits intomainfrom
feature/dynamodb-backend

Conversation

@lzukowski
Copy link
Copy Markdown
Collaborator

No description provided.

lzukowski and others added 8 commits February 8, 2026 14:43
Implements a new DynamoDB backend for python-event-sourcery that provides:

Core Features:
- Event storage and retrieval with versioning support
- Multi-tenancy support via tenant_id partitioning
- Optimistic concurrency control for versioned streams
- Automatic table creation for events, streams, and snapshots
- Integration with pytest backend selection (--backends=dynamodb)

Architecture:
- Extends base Backend class (non-transactional by design)
- Uses boto3 for AWS DynamoDB integration
- Supports DynamoDB Local for development/testing
- Follows existing backend patterns for consistency

Current Limitations:
- No subscription support (not implemented)
- No outbox pattern support (not implemented)
- No global position tracking
- Eventual consistency model

Testing:
- 78% code coverage with core functionality tests passing
- BDD-style behavioral tests added
- All standard event store tests pass for implemented features

This backend is designed for serverless AWS architectures where
DynamoDB's scalability and pay-per-request model are preferred
over transactional guarantees.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
Implement missing functionality for DynamoDB event store backend:

- Add subscription strategy with subscribe_to_all, subscribe_to_category, and subscribe_to_events methods
- Implement outbox storage strategy with transactional entry processing
- Add global position counter using atomic DynamoDB updates
- Create GSI for efficient position-based queries in subscriptions
- Handle empty tenant IDs by converting to "default" for DynamoDB compatibility
- Implement gap detection for out-of-order event handling
- Add retry management for outbox entries with configurable attempts

The implementation follows established patterns from SQLAlchemy and Django backends
while adapting to DynamoDB's NoSQL characteristics and constraints.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
Gap detection is not needed for DynamoDB backend because:
- Positions are allocated atomically via _allocate_positions() before events are written
- DynamoDB backend is non-transactional (extends Backend, not TransactionalBackend)
- No transaction rollbacks means no gaps can occur in the position sequence
- Batch writes are atomic - all events succeed or all fail

This simplifies the subscription implementation and removes unnecessary latency
from waiting for gaps that cannot exist in DynamoDB's architecture.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
Fixed 33 failing tests across 4 main areas:

1. Subscription Implementation (25 tests)
   - Fixed stream_id_hex storage to store only hex value, not full string representation
   - Fixed category handling - empty strings now properly converted to None
   - Fixed event ordering and position tracking
   - Implemented proper timeout handling for batch subscriptions
   - Fixed Decimal to int conversions for positions and versions

2. Snapshot Support (4 tests)
   - Implemented snapshot loading in fetch_events method
   - Returns only snapshot and events after it when snapshot exists
   - Fixed version filtering with proper start/stop range handling

3. Stream Name Uniqueness (1 test)
   - Added validation to prevent same stream name with different UUID
   - Considers categories - same name with different category is allowed
   - Raises AnotherStreamWithThisNameButOtherIdExists when appropriate

4. Category Handling (3 tests)
   - Modified partition key to include category when present
   - Ensures streams with same UUID but different categories are separate
   - Fixed stream metadata storage and retrieval for categories

Additional fixes:
- Fixed outbox ordering by sorting items by position
- Added proper Attr import for DynamoDB conditions
- Marked in_transaction tests to skip for non-transactional DynamoDB backend

All 123 tests now pass for DynamoDB backend.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
Removed dead code and simplified logic:
- Removed unnecessary try-except blocks that just re-raised exceptions
- Removed unused _get_stream_version method
- Simplified snapshot range checking to match SQL backends logic
- Removed redundant stream name check in _check_stream_name_uniqueness
- Removed unreachable break statements and None checks
- Cleaned up subscription implementation after gap detection removal

Coverage increased from 86% to 99%. Remaining uncovered lines are:
- Configuration handling in __init__.py (legitimate options)
- Edge case where snapshot exists but is outside query range
- Snapshot deletion when removing streams (not tested)

These are legitimate code paths that require specific test scenarios.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
Simplified table creation by always ensuring tables exist during backend
configuration. This makes the API cleaner and removes an unnecessary
configuration option.

Changes:
- Removed create_tables from DynamoDBConfig
- Tables are now always created idempotently during configure()
- Used contextlib.suppress for cleaner error handling
- Added documentation about migration considerations
- Updated test fixtures to remove create_tables parameter

Coverage improved from 98% to 99%. The remaining 2 uncovered lines handle
cases where resources already exist, which is expected in production.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
@lzukowski lzukowski closed this Feb 9, 2026
@lzukowski lzukowski deleted the feature/dynamodb-backend branch February 9, 2026 18:07
@lzukowski lzukowski restored the feature/dynamodb-backend branch February 9, 2026 18:09
@lzukowski lzukowski reopened this Feb 9, 2026
lzukowski and others added 8 commits February 9, 2026 21:28
Remove all inline comments from DynamoDB backend implementation while preserving
public method docstrings for API documentation.
- Fix line length and formatting in BDD test utilities
- Clean up whitespace and imports in outbox test configuration
- Update pytest skip markers with proper formatting
- Improve readability of test function signatures and skip decorators
Remove unnecessary try/except import handling for boto3 since it's declared
as an optional dependency in pyproject.toml. This simplifies the code and
removes redundant skip logic.
Fix regression introduced during comment removal where filterer function
was incorrectly wrapped in lambda, breaking outbox filtering functionality.
This restores 100% test coverage.
- Import boto3 directly in __init__.py instead of TYPE_CHECKING block
- Use direct boto3 type annotations instead of string literals
- Keep internal module imports in TYPE_CHECKING to avoid circular imports
- Remove unused TYPE_CHECKING imports where possible
- Remove all TYPE_CHECKING conditional imports from DynamoDB backend
- Create separate config.py module for DynamoDBClient and DynamoDBConfig to avoid circular imports
- Replace private module imports with public API:
  * event_sourcery._event_store.* → event_sourcery.* and event_sourcery.interfaces
  * Use OutboxFiltererStrategy from interfaces instead of private module
- All tests pass with 100% coverage maintained
- Import boto3 directly without TYPE_CHECKING guards
- Alphabetize imports and group them properly
- Remove redundant blank lines
- Fix import ordering to follow PEP 8 conventions
- Clean up formatting inconsistencies across all files

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Replace list concatenation with unpacking syntax for better performance
- Refactor complex insert_events method by extracting helper methods
- Remove unused variables and add proper type annotations
- Fix MyPy type errors with proper null checks and type ignores
- Add noqa comments for test credentials to suppress security warnings

All linting (ruff) and type checking (mypy) now pass successfully.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

### Implementation Approach

1. **Non-Transactional Nature**: Clearly document that this backend does not provide transactional guarantees
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about TransactWrite?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TransactWriteItems provides atomicity (all-or-nothing) for a predefined set of up to 100 write operations, but it doesn't satisfy what we mean by "transactional" in the context of TransactionalBackend.

Our concept of transactional means a database transaction that stays open while events are persisted, in-transaction listeners are dispatched synchronously, and those listeners can perform additional writes (e.g. read model projections) within the same transaction. The whole thing can then be committed or rolled back as a unit.

With TransactWriteItems, you must know all writes upfront. It's a closed batch. To make this backend work as TransactionalBackend, all operations performed by listeners would need to be collected and passed back to us rather than executed directly against DynamoDB, so we could bundle everything into a single TransactWriteItems call. That introduces additional abstractions and complexity that we don't think are justified.

That's why this backend assumes it is atomic but not transactional.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

introduces additional abstractions and complexity that we don't think are justified.

Makes sense, I was just surprised a bit it was not mentioned in the ADR.
Thanks for the explanation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants