Design and implement a distributed manufacturing pipeline simulation system that runs on Kubernetes, providing real-time control and monitoring capabilities through a web interface. The system follows modern microservices architecture principles, async processing, and cloud-native design patterns.

This project simulates a real-world manufacturing pipeline with multiple stages. It is designed to be scalable, resilient, and capable of real-time monitoring. Different microservices represent various production stages and are deployed using a Kubernetes-based cloud environment.

✅ Real-time control and monitoring ✅ Asynchronous processing for efficiency ✅ Microservices-based architecture ✅ Cloud-native deployment with Kubernetes ✅ Integration with RabbitMQ/NATS for messaging

The system follows a microservices architecture, where each component is modular and serves a specific purpose.

• Authentication & Authorization: User registration, login, and role-based access.
• Pipeline Management:
  • Create pipelines with configurable stages.
  • View all created pipelines with status updates.
  • Execute and cancel pipelines.
• Real-Time Monitoring:
  • WebSocket integration for live pipeline status updates.
  • Dashboard visualizations.

• User Authentication: Uses Supabase authentication with JWT-based session management.
• Pipeline Execution:
  • Execution model managed by ParallelPipelineOrchestrator.
  • Supports parallel and sequential execution.
  • Rollback support for failure handling.
• API Services:
  • REST API for frontend interactions.
  • gRPC API for CLI (democtl).
• Database Interaction: Uses Supabase (PostgreSQL) for pipeline, stage, and execution storage.
• Messaging System: Uses RabbitMQ/NATS for async processing.

• Maintains active client connections.
• Broadcasts events when a stage status changes.
• Eliminates polling for better performance.

• Frontend: React + Material UI + WebSockets
• Backend: Go + Gin (REST API) + gRPC (CLI)
• Database: Supabase (PostgreSQL)
• Messaging: RabbitMQ/NATS (async processing)
• Orchestration: Kubernetes + Docker

1. User submits a pipeline creation request.
2. Backend validates input and stores metadata in Supabase.
3. User receives a confirmation with pipeline details.

1. User selects a pipeline and starts execution.
2. Backend retrieves the pipeline configuration.
3. ParallelPipelineOrchestrator manages execution.
4. Each stage updates its status (Pending → Running → Completed).
5. WebSockets push real-time updates to the frontend.

• Kubernetes-Based Deployment
  • Backend & Frontend deployed as separate microservices.
  • Autoscaling backend instances based on load.
  • Dockerized containers for isolation.
• Messaging System (RabbitMQ/NATS) ensures event-driven execution.

Ensure you have the following installed:

• Golang v1.21+
• Node.js & npm
• Docker Desktop (with Kubernetes enabled)

git clone https://github.com/sarika-p9/my-pipeline-project.git
cd my-pipeline-project

Create a .env file in the project root with the following:

SUPABASE_URL=https://<Project_ID>.supabase.co
SUPABASE_KEY=xxxxxxxxxxxxxxxxxxxx
POSTGRES_DSN=postgresql://postgres.<Project_ID>:<password>@aws-0-ap-south-1.pooler.supabase.com:6543/postgres

Uncomment these lines in the following files to enable local execution:

// if err := godotenv.Load(); err != nil {
//     log.Println("Warning: No .env file found. Proceeding with existing environment variables.")
// }

Files to update:

• cmd/democtl/cmd/start.go
• cmd/grpc_server/main.go
• cmd/main_server/main.go
• internal/infrastructure/database.go
• internal/infrastructure/supabase_client.go

nats-server -DV
kubectl apply -f rabbitmq-deployment.yaml

# Start Backend
go run cmd/main_server/main.go

# Start Frontend
cd frontend
npm install  # First-time setup only
npm start

Open http://localhost:3000 in your browser.

Ensure Docker Desktop is installed and running with Kubernetes enabled.

docker run sarikapt9/my-backend
docker run sarikapt9/my-frontend

kubectl create secret generic supabase-postgres-secret \
  --from-literal=SUPABASE_URL="https://<Project_ID>.supabase.co" \
  --from-literal=SUPABASE_KEY="xxxxxxxxxxxxxxxxxxxx" \
  --from-literal=POSTGRES_DSN="postgresql://postgres.<Project_ID>:<password>@aws-0-ap-south-1.pooler.supabase.com:6543/postgres"

kubectl apply -f backend-deployment.yaml
kubectl apply -f frontend-deployment.yaml
kubectl apply -f rabbitmq-deployment.yaml
kubectl apply -f nats-deployment.yaml

Once deployed, access the frontend at http://localhost:30001.

# Register a new user
./democtl register --email="xyz@abc.com" --password="xxxxx"

# Login
./democtl login --email="xyz@abc.com" --password="xxxxx"

# Create a pipeline
./democtl pipeline create --user="xxxxx" --stages=3 --pipeline-name="TestPipeline" --stage-names="a,b,c"

# Start pipeline execution
./democtl pipeline start --pipeline-id="xxxxx" --user-id="xxxxx" --input="{}"

# Get pipeline status
./democtl pipeline status --pipeline-id="xxxxx"

The Distributed Manufacturing Pipeline Simulation System provides a scalable, real-time solution for managing manufacturing pipelines. It leverages modern cloud-native and microservices principles, ensuring high performance and efficiency.