[MISC] Improve dev experience by adding a compose debug override - #1765
Conversation
Summary by CodeRabbit
WalkthroughAdds a debug-focused Docker Compose override enabling debugpy on multiple Python services with per-service debug ports, and a comprehensive development sample compose that adds frontend, new services, many V2 workers, updated dev runtime/watch commands, and a simplified README debugging section. Changes
Sequence Diagram(s)mermaid Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes 🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 inconclusive)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Fix all issues with AI agents
In `@docker/compose.debug.yaml`:
- Around line 52-67: Remove the duplicate port mapping "3001:3001" from the
platform-service ports list in the docker compose debug file so only "5680:5680"
remains here (the "3001:3001" binding is already declared in the other Compose
override), i.e., edit the platform-service ports section to delete the
"3001:3001" entry and leave the "5680:5680" mapping to avoid confusion and keep
port definitions consolidated.
In `@docker/sample.compose.override.yaml`:
- Around line 42-79: The backend service's hot-reload isn't enabled: update the
backend service command (the entrypoint/command that runs gunicorn in the
backend service) to include the --reload gunicorn flag so code synced by the
develop.watch sync actions takes effect (i.e., add "--reload" to the existing
gunicorn options in the command string); alternatively, if you prefer
restart-based reloads, change the develop.watch action entries under the backend
service from "action: sync" to "action: sync+restart" so the container restarts
after file syncs.
652a715 to
f5020c6
Compare
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Fix all issues with AI agents
In `@docker/README.md`:
- Around line 115-124: The README debug ports table is out of sync with
compose.debug.yaml: update the table in docker/README.md to exactly match the
port assignments defined in compose.debug.yaml (ensure backend remains 5678,
change platform-service to 5680, prompt-service to 5681, runner to 5679), remove
or mark x2text-service as not configured if compose.debug.yaml has no debug
entry for it, and add the V2 worker debug port range (5682–5688) reflecting the
compose.debug.yaml entries; verify all service names match the keys in
compose.debug.yaml and update any captions or notes to reflect these corrected
ports.
In `@docker/sample.compose.override.yaml`:
- Around line 301-304: The service worker-log-history-scheduler-v2 is missing
the standard develop.watch configuration used by other V2 workers; update the
service block for worker-log-history-scheduler-v2 to include the same
develop.watch settings (e.g., bind mounts, polling/ignore patterns, commands
used by other V2 workers) so it matches the other V2 workers, or if omission is
intentional add a short inline comment in the worker-log-history-scheduler-v2
service explaining why live development/watch is not needed.
- Around line 23-24: Add a note documenting the Docker Compose version
requirement for the !override directive (used where !override appears for
services like reverse-proxy) by adding a brief header comment at the top of this
compose override file and/or updating the main README to state "Docker Compose
2.24.4+ required" (or equivalent), so developers know that the !override syntax
requires Compose 2.24.4 or newer.
…ry, add V2 workers support - Move debugpy to optional compose.debug.yaml for cleaner default dev setup - Update compose.override.yaml with memory-optimized settings (1 worker, 2 threads) - Add V2 workers configuration with build definitions - Move V1 workers to optional workers-v1 profile - Use modern uv run python -Xfrozen_modules=off pattern for services - Updated README with compose.debug.yaml usage instructions Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
f5020c6 to
74c9991
Compare
- Add Docker Compose 2.24.4+ requirement note for !override directive - Add comment explaining why worker-log-history-scheduler-v2 has no watch Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Fix port order: runner=5679, platform=5680, prompt=5681 - Remove x2text-service (not in compose.debug.yaml) - Add V2 workers debug ports (5682-5688) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Remove db image override and V1 worker command overrides from sample file. Users should reference compose.override.yaml for actual dev setup. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
|
* [MISC] Improve Docker dev experience: separate debugpy, optimize memory, add V2 workers support - Move debugpy to optional compose.debug.yaml for cleaner default dev setup - Update compose.override.yaml with memory-optimized settings (1 worker, 2 threads) - Add V2 workers configuration with build definitions - Move V1 workers to optional workers-v1 profile - Use modern uv run python -Xfrozen_modules=off pattern for services - Updated README with compose.debug.yaml usage instructions Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * [MISC] Add Docker Compose version requirement and scheduler comment - Add Docker Compose 2.24.4+ requirement note for !override directive - Add comment explaining why worker-log-history-scheduler-v2 has no watch Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * [MISC] Fix debug ports table in README - Fix port order: runner=5679, platform=5680, prompt=5681 - Remove x2text-service (not in compose.debug.yaml) - Add V2 workers debug ports (5682-5688) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * [MISC] Simplify sample.compose.override.yaml Remove db image override and V1 worker command overrides from sample file. Users should reference compose.override.yaml for actual dev setup. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
…1806) * refactor: Add dynamic plugin loading for enterprise components ## What - Add dynamic plugin loading support to OSS codebase - Enable enterprise components to be loaded at runtime without modifying tracked files ## Why - Enterprise code was overwriting git-tracked OSS files causing dirty git state - Need clean separation between OSS and enterprise codebases - OSS should work independently without enterprise components ## How - `unstract_migrations.py`: Uses try/except ImportError to load from `pluggable_apps.migrations_ext` - `api_hub_usage_utils.py`: Uses try/except ImportError to load from `plugins.verticals_usage` - `utils.py`: Uses try/except ImportError to load from `pluggable_apps.manual_review_v2` and `plugins.workflow_manager.workflow_v2.rule_engine` - `backend.Dockerfile`: Conditional install of `requirements.txt` if present ## Can this PR break any existing features. If yes, please list possible items. If no, please explain why. - No. The changes add optional plugin loading that gracefully falls back to default behavior when plugins are not present. Existing OSS functionality is preserved. ## Database Migrations - None ## Env Config - None ## Relevant Docs - None ## Related Issues or PRs - None ## Dependencies Versions - None ## Notes on Testing - OSS build: Verify app starts and works without enterprise plugins - Enterprise build: Verify plugins are loaded and function correctly 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * refactor: Use get_plugin() for API Hub usage utilities 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Refactor random sampling logic in utils.py Removed redundant import of random and exception handling for manual_review_v2. Signed-off-by: Hari John Kuriakose <hari@zipstack.com> * fix: Add Traefik port labels and clean up service ignore list - Add explicit loadbalancer port labels for backend (8000) and frontend (3000) services in docker-compose to ensure proper Traefik routing - Rename spawned_services to ignored_services for clarity - Extend ignored_services list to include tool-classifier, tool-text_extractor, and worker-unified services that don't need environment setup 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: Update frontend Docker config for nginx serving Update Traefik port label to 80 to match nginx and fix Dockerfile to use BUILD_CONTEXT_PATH for the runtime config script in both dev and prod stages. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: Use ARG instead of ENV for BUILD_CONTEXT_PATH in frontend Dockerfile Convert BUILD_CONTEXT_PATH from environment variable to build argument for proper Docker multi-stage build support. ARGs must be declared globally and re-declared in each stage that needs them. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * feat: Add HubSpot integration plugin for contact event tracking - Add new integrations plugin category under backend/plugins/integrations/ - Create HubSpot plugin with event-based contact updates - Track user milestone events: project creation, document upload, prompt run, tool export, and API deployment - Plugin validates is_first_for_org flag and first org member status - Remove unused hubspot_signup_api() stub from authentication_service - Update subscription_helper to use plugin pattern for form submissions 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * [FIX] Fix HITL review screen showing "Never expires" despite TTL being set (#1785) Fix two interacting bugs that prevented TTL from propagating to HITL queue records: 1. WorkflowUtil.get_hitl_ttl_seconds was an OSS stub that always returned None. Now delegates to get_hitl_ttl_seconds_by_workflow via try/except import, falling back to None in OSS environments. 2. _push_to_queue_for_api_deployment never fetched TTL. Now mirrors the connector path by calling WorkflowUtil.get_hitl_ttl_seconds and passing ttl_seconds through to _create_queue_result and _enqueue_to_packet_or_regular_queue. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * refactor: Add dynamic plugin loading for enterprise components (#1736) * refactor: Add dynamic plugin loading for enterprise components - Add dynamic plugin loading support to OSS codebase - Enable enterprise components to be loaded at runtime without modifying tracked files - Enterprise code was overwriting git-tracked OSS files causing dirty git state - Need clean separation between OSS and enterprise codebases - OSS should work independently without enterprise components - `unstract_migrations.py`: Uses try/except ImportError to load from `pluggable_apps.migrations_ext` - `api_hub_usage_utils.py`: Uses try/except ImportError to load from `plugins.verticals_usage` - `utils.py`: Uses try/except ImportError to load from `pluggable_apps.manual_review_v2` and `plugins.workflow_manager.workflow_v2.rule_engine` - `backend.Dockerfile`: Conditional install of `requirements.txt` if present - No. The changes add optional plugin loading that gracefully falls back to default behavior when plugins are not present. Existing OSS functionality is preserved. - None - None - None - None - None - OSS build: Verify app starts and works without enterprise plugins - Enterprise build: Verify plugins are loaded and function correctly 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * refactor: Use get_plugin() for API Hub usage utilities 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Refactor random sampling logic in utils.py Removed redundant import of random and exception handling for manual_review_v2. Signed-off-by: Hari John Kuriakose <hari@zipstack.com> * fix: Add Traefik port labels and clean up service ignore list - Add explicit loadbalancer port labels for backend (8000) and frontend (3000) services in docker-compose to ensure proper Traefik routing - Rename spawned_services to ignored_services for clarity - Extend ignored_services list to include tool-classifier, tool-text_extractor, and worker-unified services that don't need environment setup 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: Update frontend Docker config for nginx serving Update Traefik port label to 80 to match nginx and fix Dockerfile to use BUILD_CONTEXT_PATH for the runtime config script in both dev and prod stages. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: Use ARG instead of ENV for BUILD_CONTEXT_PATH in frontend Dockerfile Convert BUILD_CONTEXT_PATH from environment variable to build argument for proper Docker multi-stage build support. ARGs must be declared globally and re-declared in each stage that needs them. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: typo in ignored services var name * fix: handle script execution via entrypoint --------- Signed-off-by: Hari John Kuriakose <hari@zipstack.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * feat: Add auth error code for forbidden emails (#1789) * feat: add auth error code and frontend error display * fix: run frontend dev server on port 80 and add signup handler - Set PORT=80 env var in frontend Dockerfile development stage - Change EXPOSE from 3000 to 80 to match production nginx - Add handleSignup function and pass to LoginForm component 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: add ARG declaration to Dockerfile stages using BUILD_CONTEXT_PATH SonarQube flagged that ARG must be declared in each Docker build stage where it is used. Added the missing ARG BUILD_CONTEXT_PATH to both development and builder stages. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> * [MISC] Improve dev experience by adding a compose debug override (#1765) * [MISC] Improve Docker dev experience: separate debugpy, optimize memory, add V2 workers support - Move debugpy to optional compose.debug.yaml for cleaner default dev setup - Update compose.override.yaml with memory-optimized settings (1 worker, 2 threads) - Add V2 workers configuration with build definitions - Move V1 workers to optional workers-v1 profile - Use modern uv run python -Xfrozen_modules=off pattern for services - Updated README with compose.debug.yaml usage instructions Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * [MISC] Add Docker Compose version requirement and scheduler comment - Add Docker Compose 2.24.4+ requirement note for !override directive - Add comment explaining why worker-log-history-scheduler-v2 has no watch Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * [MISC] Fix debug ports table in README - Fix port order: runner=5679, platform=5680, prompt=5681 - Remove x2text-service (not in compose.debug.yaml) - Add V2 workers debug ports (5682-5688) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * [MISC] Simplify sample.compose.override.yaml Remove db image override and V1 worker command overrides from sample file. Users should reference compose.override.yaml for actual dev setup. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> * [FIX] Optimize queries made by worker and retry config of worker base client (#1798) * Fix production queryset performance and retry amplification Resolves 39.8s GET /internal/v1/file-execution/<uuid>/ latency by: - Removing 7 debug COUNT(*) full table scans from get_queryset() - Adding get_object() O(1) PK lookup override in ViewSet - Adding @with_cache decorators to pipeline fetch methods - Adding pipeline_data_key() to prevent cache key collisions - Fixing urllib3 retry amplification: clear status_forcelist, let app-level retries handle status codes - Use config defaults instead of hardcoded retry values Root cause: Count queries on every request + retry storm (urllib3 × app-level). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Address PR #1798 review comments - Chain exception context in get_object() (Ruff B904) - Fix cache key collision: get_workflow_definition() now uses CacheType.WORKFLOW_DEFINITION instead of CacheType.WORKFLOW to avoid type mismatch with get_workflow() sharing the same cache key - Include check_active in get_pipeline_data() cache key to prevent active-status bypass when check_active=False is cached first - Refactor status() and update_hash() to use self.get_object() instead of duplicating manual queryset + org filtering; add except APIException pass-through so NotFound propagates as 404 not 500 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * UN-2971 [FEAT] Pass selectedProduct to login/signup API for OAuth product scope (#1803) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Hari John Kuriakose <hari@zipstack.com> Co-authored-by: Claude <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: Scope HubSpot milestone count checks to current organization PromptStudioOutputManager and DocumentManager lack DefaultOrganizationManagerMixin, so .objects.count() was counting across ALL organizations. Filter through the tool FK to CustomTool (which is org-scoped) to get correct per-org counts. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [REFACTOR] Extract HubSpot notification logic into shared utility Move all _notify_hubspot_* methods from views into a shared utils/hubspot_notify.py module with a single notify_hubspot_event() function, reducing duplication across prompt_studio views and api_deployment_views. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Signed-off-by: Hari John Kuriakose <hari@zipstack.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: vishnuszipstack <117254672+vishnuszipstack@users.noreply.github.com> Co-authored-by: Chandrasekharan M <117059509+chandrasekharan-zipstack@users.noreply.github.com> Co-authored-by: vishnuszipstack <vishnu@zipstack.com> Co-authored-by: Gayathri <142381512+gaya3-zipstack@users.noreply.github.com>



What
compose.debug.yamlfor debugpy supportsample.compose.override.yamlwith memory-optimized settings (1 worker, 2 threads)workers-v1profileWhy
How
compose.debug.yamlwith debugpy configuration for all Python services and V2 workers (ports 5678-5688)sample.compose.override.yamlwith new version usinguv run python -Xfrozen_modules=offpatternCan this PR break any existing features. If yes, please list possible items. If no, please explain why.
No breaking changes. Changes are additive and optional:
compose.debug.yamlsample.compose.override.yamlis backward compatibleworkers-v1profileworkers-v2profileDatabase Migrations
None
Env Config
None
Relevant Docs
/docker/README.md/docker/docker-compose.yamlRelated Issues or PRs
None
Dependencies Versions
None
Notes on Testing
Tested locally:
VERSION=test docker compose -f docker-compose.yaml -f compose.override.yaml watch- worksVERSION=test docker compose -f docker-compose.yaml -f compose.override.yaml -f compose.debug.yaml watch- debugpy ports accessibleScreenshots
N/A
Checklist