Problem / motivation
The current framework implements generation interfaces for comfyui_image, comfyui_video, and comfyui_music, which are intended to help users introduce custom generation workflows. However, in actual local generation scenarios, there are two significant bottlenecks in the user experience:
-
Incompatibility of Workflow JSON Formats
ComfyUI’s support for workflow JSONs is fragmented. The ComfyUI API requires JSONs in API Format, whereas the default JSONs saved via the ComfyUI Frontend are in Standard Format. Consequently, users must manually export the workflow as "JSON (API Format)" and provide the file path to the Agent. This manual step prevents the process from being easily automated. Furthermore, ComfyUI currently lacks an API to programmatically download API-formatted JSONs, and there is no immediate plan for a native backend/frontend extension to support this.
-
Rigidity of Built-in Comfy Workflows
The existing built-in Comfy workflows rely on a fixed set of providers, which are insufficient for complex, real-world generation needs. For example, I utilize ComfyScript to programmatically generate and run workflows via MCP tools (e.g., generate_image, video_t2i, i2v, ref2v). These scripts automatically expand prompts, select/apply LoRAs and weights, and refine prompts using triggers. Since this logic is independent and subject to frequent dynamic adjustments, it should be decoupled from the OpenMontage core.
Proposed solution
At a high level, this implementation introduces a standardized way to integrate external toolsets:
Support for MCP Tools: Auto-Discovery, Classification, and Registration.
Expected Benefits
- Increased Reliability: By integrating directly into the OpenMontage framework and pipeline, the tool-calling success rate is significantly improved.
- Architectural Decoupling: This approach separates the core system design of OpenMontage from the implementation details of third-party tools. This modularity allows the system to remain lightweight while enabling users to easily extend and customize their own toolsets.
User Story
- Configure: Edit mcp_servers.yaml to enable the server and add the desired MCP servers.
- Classify: Run the skills/meta/mcp-tool-classification.md skill once. This will update mcp_server.yaml with the classified tools, making them available to OpenMontage.
- Utilize: The MCP tools are now fully integrated and available for use within OpenMontage.
Alternatives considered
No response
Additional context
No response
Problem / motivation
The current framework implements generation interfaces for comfyui_image, comfyui_video, and comfyui_music, which are intended to help users introduce custom generation workflows. However, in actual local generation scenarios, there are two significant bottlenecks in the user experience:
Incompatibility of Workflow JSON Formats
ComfyUI’s support for workflow JSONs is fragmented. The ComfyUI API requires JSONs in API Format, whereas the default JSONs saved via the ComfyUI Frontend are in Standard Format. Consequently, users must manually export the workflow as "JSON (API Format)" and provide the file path to the Agent. This manual step prevents the process from being easily automated. Furthermore, ComfyUI currently lacks an API to programmatically download API-formatted JSONs, and there is no immediate plan for a native backend/frontend extension to support this.
Rigidity of Built-in Comfy Workflows
The existing built-in Comfy workflows rely on a fixed set of providers, which are insufficient for complex, real-world generation needs. For example, I utilize ComfyScript to programmatically generate and run workflows via MCP tools (e.g., generate_image, video_t2i, i2v, ref2v). These scripts automatically expand prompts, select/apply LoRAs and weights, and refine prompts using triggers. Since this logic is independent and subject to frequent dynamic adjustments, it should be decoupled from the OpenMontage core.
Proposed solution
At a high level, this implementation introduces a standardized way to integrate external toolsets:
Support for MCP Tools: Auto-Discovery, Classification, and Registration.
Expected Benefits
User Story
Alternatives considered
No response
Additional context
No response