refactor: Add dynamic plugin loading for enterprise components - #1736
Merged
Conversation
## What - Add dynamic plugin loading support to OSS codebase - Enable enterprise components to be loaded at runtime without modifying tracked files ## Why - Enterprise code was overwriting git-tracked OSS files causing dirty git state - Need clean separation between OSS and enterprise codebases - OSS should work independently without enterprise components ## How - `unstract_migrations.py`: Uses try/except ImportError to load from `pluggable_apps.migrations_ext` - `api_hub_usage_utils.py`: Uses try/except ImportError to load from `plugins.verticals_usage` - `utils.py`: Uses try/except ImportError to load from `pluggable_apps.manual_review_v2` and `plugins.workflow_manager.workflow_v2.rule_engine` - `backend.Dockerfile`: Conditional install of `requirements.txt` if present ## Can this PR break any existing features. If yes, please list possible items. If no, please explain why. - No. The changes add optional plugin loading that gracefully falls back to default behavior when plugins are not present. Existing OSS functionality is preserved. ## Database Migrations - None ## Env Config - None ## Relevant Docs - None ## Related Issues or PRs - None ## Dependencies Versions - None ## Notes on Testing - OSS build: Verify app starts and works without enterprise plugins - Enterprise build: Verify plugins are loaded and function correctly 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughAdds an extensible migrations class, replaces OSS no-op API Hub usage functions with plugin-driven implementations, implements workflow utility functions that delegate to pluggable helpers, updates runtime service ignore list, and conditionally installs extra Python deps in the backend Docker production stage. (≤50 words) Changes
Sequence Diagram(s)sequenceDiagram
participant Caller
participant APIHubUtil as APIHubUsageUtil
participant Plugin as verticals_usage plugin
participant HeadersCache as headers_cache_class
participant UsageTracker as usage_tracker
Caller->>APIHubUtil: track_api_hub_usage(request, metadata)
APIHubUtil->>Plugin: obtain plugin and create headers_cache & usage_tracker
APIHubUtil->>Plugin: call extract_api_hub_headers(request)
Plugin-->>APIHubUtil: headers (or None / raises)
alt headers present
APIHubUtil->>HeadersCache: store_headers(headers)
HeadersCache-->>APIHubUtil: success/failure
APIHubUtil->>UsageTracker: store_usage(headers, metadata)
UsageTracker-->>APIHubUtil: success/failure
APIHubUtil-->>Caller: return True
else no headers or error
APIHubUtil-->>Caller: return False
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes 🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
for more information, see https://pre-commit.ci
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
backend/plugins/workflow_manager/workflow_v2/api_hub_usage_utils.py (1)
79-106: Remove unusedttl_secondsparameter or pass it tostore_headers.The
ttl_secondsparameter (line 83) is not passed tostore_headers(line 103). Either remove the parameter from the method signature or pass it to the underlying implementation. Additionally, uselogger.exceptioninstead oflogger.errorto capture stack traces.🔧 Proposed fix
try: - return api_hub_headers_cache.store_headers(execution_id, headers) + return api_hub_headers_cache.store_headers(execution_id, headers, ttl_seconds) except Exception as e: - logger.error(f"Error caching API hub headers: {e}") + logger.exception(f"Error caching API hub headers: {e}") return False
🤖 Fix all issues with AI agents
In @backend/plugins/workflow_manager/workflow_v2/utils.py:
- Around line 38-46: Remove the unused import by deleting
get_db_rules_by_workflow_id from the try block inside _mrq_files; keep the
random import intact and ensure the try/except still only catches ImportError
for missing dependencies, so the function uses random.sample as before without
importing the unused helper.
- Around line 93-95: The FileHash DTO's file_destination is typed as
tuple[str,str] | None but code assigns a plain string
(WorkflowEndpoint.ConnectionType.MANUALREVIEW) to file_hash.file_destination;
update the code to match the DTO by assigning a tuple (e.g.,
(WorkflowEndpoint.ConnectionType.MANUALREVIEW, "<optional-second>") or a
meaningful second element) wherever file_destination is set, or change the
FileHash type to str | None if the design intends a single string; ensure
consistency by updating all initializations and comparisons that currently use
empty strings "" and references to file_hash.file_destination and
WorkflowEndpoint.ConnectionType.MANUALREVIEW accordingly.
🧹 Nitpick comments (2)
backend/plugins/workflow_manager/workflow_v2/api_hub_usage_utils.py (2)
50-54: Uselogger.exceptionfor better error diagnostics.When catching exceptions during usage tracking,
logger.exceptionautomatically includes the stack trace, which aids debugging in production environments.♻️ Proposed fix
except Exception as e: - logger.error( + logger.exception( f"Failed to track API hub usage for execution {workflow_execution_id}: {e}" ) return False
75-77: Uselogger.exceptionfor better error diagnostics.♻️ Proposed fix
except Exception as e: - logger.error(f"Error extracting API hub headers: {e}") + logger.exception(f"Error extracting API hub headers: {e}") return None
📜 Review details
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Cache: Disabled due to Reviews > Disable Cache setting
Knowledge base: Disabled due to Reviews -> Disable Knowledge Base setting
📒 Files selected for processing (4)
backend/migrating/v2/unstract_migrations.pybackend/plugins/workflow_manager/workflow_v2/api_hub_usage_utils.pybackend/plugins/workflow_manager/workflow_v2/utils.pydocker/dockerfiles/backend.Dockerfile
🧰 Additional context used
🧬 Code graph analysis (3)
backend/plugins/workflow_manager/workflow_v2/api_hub_usage_utils.py (1)
backend/workflow_manager/workflow_v2/models/execution.py (1)
organization_id(266-274)
backend/migrating/v2/unstract_migrations.py (1)
backend/migrating/v2/query.py (3)
MigrationQuery(1-765)get_public_schema_migrations(9-252)get_organization_migrations(254-765)
backend/plugins/workflow_manager/workflow_v2/utils.py (4)
backend/workflow_manager/endpoint_v2/dto.py (1)
FileHash(11-54)workers/shared/clients/manual_review_stub.py (1)
get_q_no_list(52-72)workers/shared/utils/manual_review_factory.py (9)
get_q_no_list(121-123)get_q_no_list(313-314)get_q_no_list(402-403)add_file_destination_filehash(116-118)add_file_destination_filehash(309-310)add_file_destination_filehash(398-399)get_hitl_ttl_seconds(150-152)get_hitl_ttl_seconds(348-349)get_hitl_ttl_seconds(456-457)backend/workflow_manager/endpoint_v2/models.py (1)
WorkflowEndpoint(17-66)
🪛 Ruff (0.14.10)
backend/plugins/workflow_manager/workflow_v2/api_hub_usage_utils.py
48-48: Consider moving this statement to an else block
(TRY300)
50-50: Do not catch blind exception: Exception
(BLE001)
51-53: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
75-75: Do not catch blind exception: Exception
(BLE001)
76-76: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
104-104: Do not catch blind exception: Exception
(BLE001)
105-105: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
backend/plugins/workflow_manager/workflow_v2/utils.py
156-156: Consider moving this statement to an else block
(TRY300)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: build
🔇 Additional comments (8)
docker/dockerfiles/backend.Dockerfile (1)
76-79: LGTM!The conditional install pattern properly handles the optional
requirements.txtfor enterprise dependencies. The file existence check prevents build failures when the file is absent in the OSS build.backend/migrating/v2/unstract_migrations.py (2)
22-39: LGTM!The extension pattern correctly loads additional migrations when available while preserving core migration behavior. The fallback to core-only migrations ensures OSS compatibility.
41-66: LGTM!The organization migrations extension follows the same pattern and correctly passes all required parameters to the extended migrations function.
backend/plugins/workflow_manager/workflow_v2/utils.py (5)
48-67: LGTM!The method correctly retrieves DB rules and delegates to
_mrq_fileswhen a valid percentage is configured. The ImportError fallback ensures OSS compatibility.
97-126: LGTM!The rule engine validation correctly delegates to the plugin when available and returns
Falseas a safe fallback for OSS builds.
128-140: LGTM!The backward-compatible wrapper maintains existing API while deprecating in favor of
validate_rule_engine.
142-160: LGTM!The API rules check correctly queries the DB rules configuration. The naming suggests DB rules may contain API-specific configuration (rule_string).
162-181: LGTM!The HITL TTL retrieval follows the established plugin loading pattern with a safe
Nonefallback for unlimited TTL in OSS.
hari-kuriakose
requested review from
chandrasekharan-zipstack,
jaseemjaskp,
muhammad-ali-e and
vishnuszipstack
January 9, 2026 18:22
hari-kuriakose
marked this pull request as draft
January 9, 2026 20:27
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Removed redundant import of random and exception handling for manual_review_v2. Signed-off-by: Hari John Kuriakose <hari@zipstack.com>
hari-kuriakose
marked this pull request as ready for review
January 13, 2026 14:27
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Fix all issues with AI agents
In @backend/plugins/workflow_manager/workflow_v2/api_hub_usage_utils.py:
- Around line 98-103: The ttl_seconds parameter accepted by the function is
never forwarded to the cache call; update the call to
headers_cache.store_headers(execution_id, headers, ttl_seconds) so the
underlying headers_cache_class implementation receives the TTL (or, if TTL is
intentionally unused, remove ttl_seconds from the function signature and
docstring). Ensure you modify the invocation using headers_cache_class and the
store_headers method to include ttl_seconds (or remove parameter and update docs
accordingly).
In @backend/plugins/workflow_manager/workflow_v2/utils.py:
- Around line 39-40: The current logic can raise ValueError when n == 0 or
num_to_select > n; update the selection logic to first handle n == 0 by
returning an empty set, clamp percentage to the 0–100 range, compute
num_to_select = max(0, min(n, int(n * (percentage / 100)))), and then call
random.sample on range(1, n + 1) only when num_to_select > 0; reference the
variables num_to_select, n, percentage and the random.sample call in the return
expression to locate and modify the code.
- Around line 87-89: The FileHash.file_destination field is currently typed as
tuple[str, str] | None in the DTO but is assigned string values like
WorkflowEndpoint.ConnectionType.MANUALREVIEW elsewhere; update the type
annotation of FileHash.file_destination in the FileHash definition (in
endpoint_v2/dto.py) from tuple[str, str] | None to str | None, and run a quick
grep for FileHash.file_destination uses to ensure no code expects a tuple
(adjust any typed usages/imports accordingly) so that assignments like
WorkflowEndpoint.ConnectionType.MANUALREVIEW are type-safe.
🧹 Nitpick comments (4)
backend/plugins/workflow_manager/workflow_v2/utils.py (1)
146-154: Consider restructuring the try/except for clarity.The static analysis tool (TRY300) suggests moving the success return to an
elseblock. This is a minor style preference that separates the "happy path" from exception handling.Optional refactor
try: from pluggable_apps.manual_review_v2.helper import get_db_rules_by_workflow_id - - db_rule = get_db_rules_by_workflow_id(workflow=workflow) - return db_rule is not None and db_rule.rule_string is not None except ImportError: - pass - - return False + return False + else: + db_rule = get_db_rules_by_workflow_id(workflow=workflow) + return db_rule is not None and db_rule.rule_string is not Nonebackend/plugins/workflow_manager/workflow_v2/api_hub_usage_utils.py (3)
51-55: Uselogger.exceptionto preserve stack trace.When catching exceptions,
logger.exceptionautomatically includes the traceback, which aids debugging plugin issues.Proposed fix
except Exception as e: - logger.error( + logger.exception( f"Failed to track API hub usage for execution {workflow_execution_id}: {e}" ) return False
74-76: Uselogger.exceptionfor better debugging.Same recommendation as above—use
logger.exceptionto include the traceback.Proposed fix
except Exception as e: - logger.error(f"Error extracting API hub headers: {e}") + logger.exception(f"Error extracting API hub headers: {e}") return None
101-103: Uselogger.exceptionfor better debugging.Same recommendation—use
logger.exceptionto preserve the traceback.Proposed fix
except Exception as e: - logger.error(f"Error caching API hub headers: {e}") + logger.exception(f"Error caching API hub headers: {e}") return False
📜 Review details
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Cache: Disabled due to Reviews > Disable Cache setting
Knowledge base: Disabled due to Reviews -> Disable Knowledge Base setting
📒 Files selected for processing (2)
backend/plugins/workflow_manager/workflow_v2/api_hub_usage_utils.pybackend/plugins/workflow_manager/workflow_v2/utils.py
🧰 Additional context used
🧬 Code graph analysis (2)
backend/plugins/workflow_manager/workflow_v2/utils.py (4)
backend/workflow_manager/endpoint_v2/dto.py (1)
FileHash(11-54)workers/shared/clients/manual_review_stub.py (1)
get_q_no_list(52-72)workers/shared/utils/manual_review_factory.py (6)
get_q_no_list(121-123)get_q_no_list(313-314)get_q_no_list(402-403)get_hitl_ttl_seconds(150-152)get_hitl_ttl_seconds(348-349)get_hitl_ttl_seconds(456-457)backend/workflow_manager/endpoint_v2/models.py (1)
WorkflowEndpoint(17-66)
backend/plugins/workflow_manager/workflow_v2/api_hub_usage_utils.py (1)
backend/workflow_manager/workflow_v2/models/execution.py (1)
organization_id(266-274)
🪛 Ruff (0.14.10)
backend/plugins/workflow_manager/workflow_v2/utils.py
150-150: Consider moving this statement to an else block
(TRY300)
backend/plugins/workflow_manager/workflow_v2/api_hub_usage_utils.py
49-49: Consider moving this statement to an else block
(TRY300)
51-51: Do not catch blind exception: Exception
(BLE001)
52-54: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
74-74: Do not catch blind exception: Exception
(BLE001)
75-75: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
101-101: Do not catch blind exception: Exception
(BLE001)
102-102: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: build
🔇 Additional comments (5)
backend/plugins/workflow_manager/workflow_v2/utils.py (4)
1-16: LGTM!Module docstring clearly describes the plugin-based architecture, and imports are appropriate.
42-61: LGTM!Clean implementation of optional plugin loading with appropriate fallback to
None.
91-120: LGTM!Clean plugin loading with proper fallback to
Falsewhen the rule engine is unavailable.
156-175: LGTM!Consistent plugin loading pattern with appropriate return type.
backend/plugins/workflow_manager/workflow_v2/api_hub_usage_utils.py (1)
1-11: LGTM!Clean module structure with plugin-based architecture.
chandrasekharan-zipstack
approved these changes
Jan 14, 2026
jaseemjaskp
approved these changes
Jan 14, 2026
- Add explicit loadbalancer port labels for backend (8000) and frontend (3000) services in docker-compose to ensure proper Traefik routing - Rename spawned_services to ignored_services for clarity - Extend ignored_services list to include tool-classifier, tool-text_extractor, and worker-unified services that don't need environment setup 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
docker/docker-compose.yaml (1)
168-178: Verify the frontend Traefik port configuration.The frontend service maps
3000:80, meaning the container listens on port 80 internally. However, the Traefik label specifies port 3000, which is the host-side port. Traefik communicates with containers via the Docker network, so it should target the container's internal port.🐛 Proposed fix
- traefik.http.routers.frontend.rule=Host(`frontend.unstract.localhost`) && !PathPrefix(`/api/v1`) && !PathPrefix(`/deployment`) - - traefik.http.services.frontend.loadbalancer.server.port=3000 + - traefik.http.services.frontend.loadbalancer.server.port=80
🤖 Fix all issues with AI agents
In `@run-platform.sh`:
- Around line 198-203: The skip logic is broken by a variable name typo: the
loop references ignore_services but the array is declared as ignored_services;
update the usage to match the declared name (change ignore_services to
ignored_services in the for loop) or rename the declaration to ignore_services
so both match, ensuring the for loop (the for ignore_service in ...; do block)
iterates the intended array and the continue 2 path is triggered.
📜 Review details
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Cache: Disabled due to Reviews > Disable Cache setting
Knowledge base: Disabled due to Reviews -> Disable Knowledge Base setting
📒 Files selected for processing (2)
docker/docker-compose.yamlrun-platform.sh
🧰 Additional context used
🪛 Shellcheck (0.11.0)
run-platform.sh
[warning] 198-198: ignore_services is referenced but not assigned (did you mean 'ignore_service'?).
(SC2154)
[warning] 346-346: ignored_services appears unused. Verify use (or export if used externally).
(SC2034)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: build
🔇 Additional comments (1)
docker/docker-compose.yaml (1)
33-36: LGTM!The Traefik load balancer port label correctly targets the backend container's internal port 8000.
✏️ Tip: You can disable this entire section by setting review_details to false in your review settings.
Update Traefik port label to 80 to match nginx and fix Dockerfile to use BUILD_CONTEXT_PATH for the runtime config script in both dev and prod stages. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@docker/dockerfiles/frontend.Dockerfile`:
- Around line 60-61: The COPY uses ${BUILD_CONTEXT_PATH} which is unset in the
production stage causing an empty path; update the production stage to copy the
script from the builder stage instead (e.g., COPY --from=builder
<path-to>/generate-runtime-config.sh /docker-entrypoint.d/40-env.sh) or
re-declare the BUILD_CONTEXT_PATH ARG/ENV in the production stage so
generate-runtime-config.sh is deterministically available and then run chmod +x
/docker-entrypoint.d/40-env.sh; reference the BUILD_CONTEXT_PATH variable, the
generate-runtime-config.sh file and the destination
/docker-entrypoint.d/40-env.sh when making the change.
…file Convert BUILD_CONTEXT_PATH from environment variable to build argument for proper Docker multi-stage build support. ARGs must be declared globally and re-declared in each stage that needs them. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@docker/dockerfiles/frontend.Dockerfile`:
- Around line 22-24: The COPY of generate-runtime-config.sh into
/docker-entrypoint.d/40-env.sh is ineffective on the node:20-alpine base because
that image does not auto-run /docker-entrypoint.d/*; either add an explicit
entrypoint wrapper that sources /docker-entrypoint.d/40-env.sh or invoke
generate-runtime-config.sh from the container startup command (CMD) so the
script runs in development; locate the COPY of generate-runtime-config.sh and
update Dockerfile to call the script from the container startup path (or add a
custom entrypoint that runs /docker-entrypoint.d/40-env.sh) instead of relying
on auto-execution.
Signed-off-by: Hari John Kuriakose <hari@zipstack.com>
Contributor
Test ResultsSummary
Runner Tests - Full Report
SDK1 Tests - Full Report
|
|
hari-kuriakose
added a commit
that referenced
this pull request
Feb 24, 2026
* refactor: Add dynamic plugin loading for enterprise components - Add dynamic plugin loading support to OSS codebase - Enable enterprise components to be loaded at runtime without modifying tracked files - Enterprise code was overwriting git-tracked OSS files causing dirty git state - Need clean separation between OSS and enterprise codebases - OSS should work independently without enterprise components - `unstract_migrations.py`: Uses try/except ImportError to load from `pluggable_apps.migrations_ext` - `api_hub_usage_utils.py`: Uses try/except ImportError to load from `plugins.verticals_usage` - `utils.py`: Uses try/except ImportError to load from `pluggable_apps.manual_review_v2` and `plugins.workflow_manager.workflow_v2.rule_engine` - `backend.Dockerfile`: Conditional install of `requirements.txt` if present - No. The changes add optional plugin loading that gracefully falls back to default behavior when plugins are not present. Existing OSS functionality is preserved. - None - None - None - None - None - OSS build: Verify app starts and works without enterprise plugins - Enterprise build: Verify plugins are loaded and function correctly 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * refactor: Use get_plugin() for API Hub usage utilities 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Refactor random sampling logic in utils.py Removed redundant import of random and exception handling for manual_review_v2. Signed-off-by: Hari John Kuriakose <hari@zipstack.com> * fix: Add Traefik port labels and clean up service ignore list - Add explicit loadbalancer port labels for backend (8000) and frontend (3000) services in docker-compose to ensure proper Traefik routing - Rename spawned_services to ignored_services for clarity - Extend ignored_services list to include tool-classifier, tool-text_extractor, and worker-unified services that don't need environment setup 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: Update frontend Docker config for nginx serving Update Traefik port label to 80 to match nginx and fix Dockerfile to use BUILD_CONTEXT_PATH for the runtime config script in both dev and prod stages. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: Use ARG instead of ENV for BUILD_CONTEXT_PATH in frontend Dockerfile Convert BUILD_CONTEXT_PATH from environment variable to build argument for proper Docker multi-stage build support. ARGs must be declared globally and re-declared in each stage that needs them. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: typo in ignored services var name * fix: handle script execution via entrypoint --------- Signed-off-by: Hari John Kuriakose <hari@zipstack.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
gaya3-vijayakumar
added a commit
that referenced
this pull request
Mar 9, 2026
…1806) * refactor: Add dynamic plugin loading for enterprise components ## What - Add dynamic plugin loading support to OSS codebase - Enable enterprise components to be loaded at runtime without modifying tracked files ## Why - Enterprise code was overwriting git-tracked OSS files causing dirty git state - Need clean separation between OSS and enterprise codebases - OSS should work independently without enterprise components ## How - `unstract_migrations.py`: Uses try/except ImportError to load from `pluggable_apps.migrations_ext` - `api_hub_usage_utils.py`: Uses try/except ImportError to load from `plugins.verticals_usage` - `utils.py`: Uses try/except ImportError to load from `pluggable_apps.manual_review_v2` and `plugins.workflow_manager.workflow_v2.rule_engine` - `backend.Dockerfile`: Conditional install of `requirements.txt` if present ## Can this PR break any existing features. If yes, please list possible items. If no, please explain why. - No. The changes add optional plugin loading that gracefully falls back to default behavior when plugins are not present. Existing OSS functionality is preserved. ## Database Migrations - None ## Env Config - None ## Relevant Docs - None ## Related Issues or PRs - None ## Dependencies Versions - None ## Notes on Testing - OSS build: Verify app starts and works without enterprise plugins - Enterprise build: Verify plugins are loaded and function correctly 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * refactor: Use get_plugin() for API Hub usage utilities 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Refactor random sampling logic in utils.py Removed redundant import of random and exception handling for manual_review_v2. Signed-off-by: Hari John Kuriakose <hari@zipstack.com> * fix: Add Traefik port labels and clean up service ignore list - Add explicit loadbalancer port labels for backend (8000) and frontend (3000) services in docker-compose to ensure proper Traefik routing - Rename spawned_services to ignored_services for clarity - Extend ignored_services list to include tool-classifier, tool-text_extractor, and worker-unified services that don't need environment setup 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: Update frontend Docker config for nginx serving Update Traefik port label to 80 to match nginx and fix Dockerfile to use BUILD_CONTEXT_PATH for the runtime config script in both dev and prod stages. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: Use ARG instead of ENV for BUILD_CONTEXT_PATH in frontend Dockerfile Convert BUILD_CONTEXT_PATH from environment variable to build argument for proper Docker multi-stage build support. ARGs must be declared globally and re-declared in each stage that needs them. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * feat: Add HubSpot integration plugin for contact event tracking - Add new integrations plugin category under backend/plugins/integrations/ - Create HubSpot plugin with event-based contact updates - Track user milestone events: project creation, document upload, prompt run, tool export, and API deployment - Plugin validates is_first_for_org flag and first org member status - Remove unused hubspot_signup_api() stub from authentication_service - Update subscription_helper to use plugin pattern for form submissions 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * [FIX] Fix HITL review screen showing "Never expires" despite TTL being set (#1785) Fix two interacting bugs that prevented TTL from propagating to HITL queue records: 1. WorkflowUtil.get_hitl_ttl_seconds was an OSS stub that always returned None. Now delegates to get_hitl_ttl_seconds_by_workflow via try/except import, falling back to None in OSS environments. 2. _push_to_queue_for_api_deployment never fetched TTL. Now mirrors the connector path by calling WorkflowUtil.get_hitl_ttl_seconds and passing ttl_seconds through to _create_queue_result and _enqueue_to_packet_or_regular_queue. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * refactor: Add dynamic plugin loading for enterprise components (#1736) * refactor: Add dynamic plugin loading for enterprise components - Add dynamic plugin loading support to OSS codebase - Enable enterprise components to be loaded at runtime without modifying tracked files - Enterprise code was overwriting git-tracked OSS files causing dirty git state - Need clean separation between OSS and enterprise codebases - OSS should work independently without enterprise components - `unstract_migrations.py`: Uses try/except ImportError to load from `pluggable_apps.migrations_ext` - `api_hub_usage_utils.py`: Uses try/except ImportError to load from `plugins.verticals_usage` - `utils.py`: Uses try/except ImportError to load from `pluggable_apps.manual_review_v2` and `plugins.workflow_manager.workflow_v2.rule_engine` - `backend.Dockerfile`: Conditional install of `requirements.txt` if present - No. The changes add optional plugin loading that gracefully falls back to default behavior when plugins are not present. Existing OSS functionality is preserved. - None - None - None - None - None - OSS build: Verify app starts and works without enterprise plugins - Enterprise build: Verify plugins are loaded and function correctly 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * refactor: Use get_plugin() for API Hub usage utilities 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Refactor random sampling logic in utils.py Removed redundant import of random and exception handling for manual_review_v2. Signed-off-by: Hari John Kuriakose <hari@zipstack.com> * fix: Add Traefik port labels and clean up service ignore list - Add explicit loadbalancer port labels for backend (8000) and frontend (3000) services in docker-compose to ensure proper Traefik routing - Rename spawned_services to ignored_services for clarity - Extend ignored_services list to include tool-classifier, tool-text_extractor, and worker-unified services that don't need environment setup 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: Update frontend Docker config for nginx serving Update Traefik port label to 80 to match nginx and fix Dockerfile to use BUILD_CONTEXT_PATH for the runtime config script in both dev and prod stages. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: Use ARG instead of ENV for BUILD_CONTEXT_PATH in frontend Dockerfile Convert BUILD_CONTEXT_PATH from environment variable to build argument for proper Docker multi-stage build support. ARGs must be declared globally and re-declared in each stage that needs them. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: typo in ignored services var name * fix: handle script execution via entrypoint --------- Signed-off-by: Hari John Kuriakose <hari@zipstack.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * feat: Add auth error code for forbidden emails (#1789) * feat: add auth error code and frontend error display * fix: run frontend dev server on port 80 and add signup handler - Set PORT=80 env var in frontend Dockerfile development stage - Change EXPOSE from 3000 to 80 to match production nginx - Add handleSignup function and pass to LoginForm component 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: add ARG declaration to Dockerfile stages using BUILD_CONTEXT_PATH SonarQube flagged that ARG must be declared in each Docker build stage where it is used. Added the missing ARG BUILD_CONTEXT_PATH to both development and builder stages. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> * [MISC] Improve dev experience by adding a compose debug override (#1765) * [MISC] Improve Docker dev experience: separate debugpy, optimize memory, add V2 workers support - Move debugpy to optional compose.debug.yaml for cleaner default dev setup - Update compose.override.yaml with memory-optimized settings (1 worker, 2 threads) - Add V2 workers configuration with build definitions - Move V1 workers to optional workers-v1 profile - Use modern uv run python -Xfrozen_modules=off pattern for services - Updated README with compose.debug.yaml usage instructions Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * [MISC] Add Docker Compose version requirement and scheduler comment - Add Docker Compose 2.24.4+ requirement note for !override directive - Add comment explaining why worker-log-history-scheduler-v2 has no watch Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * [MISC] Fix debug ports table in README - Fix port order: runner=5679, platform=5680, prompt=5681 - Remove x2text-service (not in compose.debug.yaml) - Add V2 workers debug ports (5682-5688) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * [MISC] Simplify sample.compose.override.yaml Remove db image override and V1 worker command overrides from sample file. Users should reference compose.override.yaml for actual dev setup. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> * [FIX] Optimize queries made by worker and retry config of worker base client (#1798) * Fix production queryset performance and retry amplification Resolves 39.8s GET /internal/v1/file-execution/<uuid>/ latency by: - Removing 7 debug COUNT(*) full table scans from get_queryset() - Adding get_object() O(1) PK lookup override in ViewSet - Adding @with_cache decorators to pipeline fetch methods - Adding pipeline_data_key() to prevent cache key collisions - Fixing urllib3 retry amplification: clear status_forcelist, let app-level retries handle status codes - Use config defaults instead of hardcoded retry values Root cause: Count queries on every request + retry storm (urllib3 × app-level). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Address PR #1798 review comments - Chain exception context in get_object() (Ruff B904) - Fix cache key collision: get_workflow_definition() now uses CacheType.WORKFLOW_DEFINITION instead of CacheType.WORKFLOW to avoid type mismatch with get_workflow() sharing the same cache key - Include check_active in get_pipeline_data() cache key to prevent active-status bypass when check_active=False is cached first - Refactor status() and update_hash() to use self.get_object() instead of duplicating manual queryset + org filtering; add except APIException pass-through so NotFound propagates as 404 not 500 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * UN-2971 [FEAT] Pass selectedProduct to login/signup API for OAuth product scope (#1803) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Hari John Kuriakose <hari@zipstack.com> Co-authored-by: Claude <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: Scope HubSpot milestone count checks to current organization PromptStudioOutputManager and DocumentManager lack DefaultOrganizationManagerMixin, so .objects.count() was counting across ALL organizations. Filter through the tool FK to CustomTool (which is org-scoped) to get correct per-org counts. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [REFACTOR] Extract HubSpot notification logic into shared utility Move all _notify_hubspot_* methods from views into a shared utils/hubspot_notify.py module with a single notify_hubspot_event() function, reducing duplication across prompt_studio views and api_deployment_views. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Signed-off-by: Hari John Kuriakose <hari@zipstack.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: vishnuszipstack <117254672+vishnuszipstack@users.noreply.github.com> Co-authored-by: Chandrasekharan M <117059509+chandrasekharan-zipstack@users.noreply.github.com> Co-authored-by: vishnuszipstack <vishnu@zipstack.com> Co-authored-by: Gayathri <142381512+gaya3-zipstack@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Why
How
unstract_migrations.py: Uses try/except ImportError to load frompluggable_apps.migrations_extapi_hub_usage_utils.py: Uses try/except ImportError to load fromplugins.verticals_usageutils.py: Uses try/except ImportError to load frompluggable_apps.manual_review_v2andplugins.workflow_manager.workflow_v2.rule_enginebackend.Dockerfile: Conditional install ofrequirements.txtif presentdocker-compose.yaml: Addtraefik.http.services.*.loadbalancer.server.portlabels for backend (8000) and frontend (3000)run-platform.sh: Renamespawned_servicestoignored_servicesand extend list with tool-classifier, tool-text_extractor, worker-unifiedCan this PR break any existing features. If yes, please list possible items. If no, please explain why.
Reviewer Guide: Analyzing Refactoring Impact
When reviewing this PR, please pay special attention to the following areas:
Import path changes: Verify that the try/except ImportError patterns correctly fall back to the original behavior. Test both with and without enterprise plugins present.
Migration loading (
unstract_migrations.py): Confirm that database migrations run correctly in both OSS and enterprise configurations. Migration failures can be difficult to recover from.Usage utilities (
api_hub_usage_utils.py,utils.py): Check that the fallback implementations match the expected interfaces. Any mismatch could cause runtime errors in production.Dockerfile changes: Verify that the conditional
requirements.txtinstall doesn't break builds when the file is absent or empty.Traefik configuration: Test that services are accessible via the expected routes. Consider edge cases like container restarts and service discovery timing.
Service ignore list: Confirm that the renamed variable and extended list don't inadvertently exclude or include wrong services.
Database Migrations
Env Config
Relevant Docs
Related Issues or PRs
Dependencies Versions
Notes on Testing
Screenshots
Checklist
I have read and understood the Contribution Guidelines.
🤖 Generated with Claude Code