Conversation
Implements a new DynamoDB backend for python-event-sourcery that provides: Core Features: - Event storage and retrieval with versioning support - Multi-tenancy support via tenant_id partitioning - Optimistic concurrency control for versioned streams - Automatic table creation for events, streams, and snapshots - Integration with pytest backend selection (--backends=dynamodb) Architecture: - Extends base Backend class (non-transactional by design) - Uses boto3 for AWS DynamoDB integration - Supports DynamoDB Local for development/testing - Follows existing backend patterns for consistency Current Limitations: - No subscription support (not implemented) - No outbox pattern support (not implemented) - No global position tracking - Eventual consistency model Testing: - 78% code coverage with core functionality tests passing - BDD-style behavioral tests added - All standard event store tests pass for implemented features This backend is designed for serverless AWS architectures where DynamoDB's scalability and pay-per-request model are preferred over transactional guarantees. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
Implement missing functionality for DynamoDB event store backend: - Add subscription strategy with subscribe_to_all, subscribe_to_category, and subscribe_to_events methods - Implement outbox storage strategy with transactional entry processing - Add global position counter using atomic DynamoDB updates - Create GSI for efficient position-based queries in subscriptions - Handle empty tenant IDs by converting to "default" for DynamoDB compatibility - Implement gap detection for out-of-order event handling - Add retry management for outbox entries with configurable attempts The implementation follows established patterns from SQLAlchemy and Django backends while adapting to DynamoDB's NoSQL characteristics and constraints. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
Gap detection is not needed for DynamoDB backend because: - Positions are allocated atomically via _allocate_positions() before events are written - DynamoDB backend is non-transactional (extends Backend, not TransactionalBackend) - No transaction rollbacks means no gaps can occur in the position sequence - Batch writes are atomic - all events succeed or all fail This simplifies the subscription implementation and removes unnecessary latency from waiting for gaps that cannot exist in DynamoDB's architecture. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
Fixed 33 failing tests across 4 main areas: 1. Subscription Implementation (25 tests) - Fixed stream_id_hex storage to store only hex value, not full string representation - Fixed category handling - empty strings now properly converted to None - Fixed event ordering and position tracking - Implemented proper timeout handling for batch subscriptions - Fixed Decimal to int conversions for positions and versions 2. Snapshot Support (4 tests) - Implemented snapshot loading in fetch_events method - Returns only snapshot and events after it when snapshot exists - Fixed version filtering with proper start/stop range handling 3. Stream Name Uniqueness (1 test) - Added validation to prevent same stream name with different UUID - Considers categories - same name with different category is allowed - Raises AnotherStreamWithThisNameButOtherIdExists when appropriate 4. Category Handling (3 tests) - Modified partition key to include category when present - Ensures streams with same UUID but different categories are separate - Fixed stream metadata storage and retrieval for categories Additional fixes: - Fixed outbox ordering by sorting items by position - Added proper Attr import for DynamoDB conditions - Marked in_transaction tests to skip for non-transactional DynamoDB backend All 123 tests now pass for DynamoDB backend. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
Removed dead code and simplified logic: - Removed unnecessary try-except blocks that just re-raised exceptions - Removed unused _get_stream_version method - Simplified snapshot range checking to match SQL backends logic - Removed redundant stream name check in _check_stream_name_uniqueness - Removed unreachable break statements and None checks - Cleaned up subscription implementation after gap detection removal Coverage increased from 86% to 99%. Remaining uncovered lines are: - Configuration handling in __init__.py (legitimate options) - Edge case where snapshot exists but is outside query range - Snapshot deletion when removing streams (not tested) These are legitimate code paths that require specific test scenarios. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
Simplified table creation by always ensuring tables exist during backend configuration. This makes the API cleaner and removes an unnecessary configuration option. Changes: - Removed create_tables from DynamoDBConfig - Tables are now always created idempotently during configure() - Used contextlib.suppress for cleaner error handling - Added documentation about migration considerations - Updated test fixtures to remove create_tables parameter Coverage improved from 98% to 99%. The remaining 2 uncovered lines handle cases where resources already exist, which is expected in production. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
Remove all inline comments from DynamoDB backend implementation while preserving public method docstrings for API documentation.
- Fix line length and formatting in BDD test utilities - Clean up whitespace and imports in outbox test configuration - Update pytest skip markers with proper formatting - Improve readability of test function signatures and skip decorators
Remove unnecessary try/except import handling for boto3 since it's declared as an optional dependency in pyproject.toml. This simplifies the code and removes redundant skip logic.
Fix regression introduced during comment removal where filterer function was incorrectly wrapped in lambda, breaking outbox filtering functionality. This restores 100% test coverage.
- Import boto3 directly in __init__.py instead of TYPE_CHECKING block - Use direct boto3 type annotations instead of string literals - Keep internal module imports in TYPE_CHECKING to avoid circular imports - Remove unused TYPE_CHECKING imports where possible
- Remove all TYPE_CHECKING conditional imports from DynamoDB backend - Create separate config.py module for DynamoDBClient and DynamoDBConfig to avoid circular imports - Replace private module imports with public API: * event_sourcery._event_store.* → event_sourcery.* and event_sourcery.interfaces * Use OutboxFiltererStrategy from interfaces instead of private module - All tests pass with 100% coverage maintained - Import boto3 directly without TYPE_CHECKING guards
- Alphabetize imports and group them properly - Remove redundant blank lines - Fix import ordering to follow PEP 8 conventions - Clean up formatting inconsistencies across all files 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
- Replace list concatenation with unpacking syntax for better performance - Refactor complex insert_events method by extracting helper methods - Remove unused variables and add proper type annotations - Fix MyPy type errors with proper null checks and type ignores - Add noqa comments for test credentials to suppress security warnings All linting (ruff) and type checking (mypy) now pass successfully. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
|
|
||
| ### Implementation Approach | ||
|
|
||
| 1. **Non-Transactional Nature**: Clearly document that this backend does not provide transactional guarantees |
There was a problem hiding this comment.
TransactWriteItems provides atomicity (all-or-nothing) for a predefined set of up to 100 write operations, but it doesn't satisfy what we mean by "transactional" in the context of TransactionalBackend.
Our concept of transactional means a database transaction that stays open while events are persisted, in-transaction listeners are dispatched synchronously, and those listeners can perform additional writes (e.g. read model projections) within the same transaction. The whole thing can then be committed or rolled back as a unit.
With TransactWriteItems, you must know all writes upfront. It's a closed batch. To make this backend work as TransactionalBackend, all operations performed by listeners would need to be collected and passed back to us rather than executed directly against DynamoDB, so we could bundle everything into a single TransactWriteItems call. That introduces additional abstractions and complexity that we don't think are justified.
That's why this backend assumes it is atomic but not transactional.
There was a problem hiding this comment.
introduces additional abstractions and complexity that we don't think are justified.
Makes sense, I was just surprised a bit it was not mentioned in the ADR.
Thanks for the explanation.
No description provided.