🚀 Torrent Distributed Engine

Torrent is an enterprise-grade, distributed job execution engine designed to handle hundreds of thousands of asynchronous tasks with absolute fault tolerance. Built entirely on a Microservices Architecture, Torrent uses event-streaming and remote procedure calls to rapidly distribute background tasks across a cluster of resilient worker nodes.

This project comes equipped with a real-time React Developer Console and Simulation Dashboard, allowing you to visualize jobs flowing through the system's infrastructure dynamically, manage your cluster, and debug worker nodes in real-time.

Our architecture guarantees zero single points of failure. The system routes background tasks through two distinct pipelines based on priority:

1. Standard & Low Priority: Pushed into Apache Kafka event topics for guaranteed sequential delivery and high-throughput batching.
2. High Priority (Fast-Track): Bypasses Kafka entirely, using gRPC Streams for direct, low-latency execution by the worker nodes.

graph TD
    Client[React Simulation Dashboard] -->|REST HTTP POST| API(Spring Boot API Gateway)
    
    subgraph Routing Layer
        API -->|Standard Priority| Kafka[(Apache Kafka Topics)]
        API -->|High Priority| gRPC[gRPC Fast-Track]
    end

    subgraph Distributed Execution
        Kafka --> Worker1(Worker Node 1)
        gRPC --> Worker1
        Kafka --> Worker2(Worker Node 2)
        gRPC --> Worker2
    end
    
    subgraph Coordination & State
        Worker1 --> DB[(PostgreSQL)]
        Worker2 --> DB
        Worker1 -.->|ShedLock & Zookeeper| Worker2
    end

• Dynamic Fleet Tracking: Every worker node generates a unique cryptographic heartbeat signature (e.g., worker:6f3a...). The central database inherently tracks exactly which physical server executed which job for unmatched observability.
• Resilient Retry Policy: Configurable exponential backoff modifiers (maxAttempts, backoffMultiplier) for flaky tasks.
• Idempotency Guarantees: Strict checking on idempotencyKey ensures that network partitions do not result in duplicate job executions.
• Distributed Cron Scheduling: Capable of scheduling millions of recurring jobs without collision using Zookeeper-backed ShedLock.
• Secure API Keys: Full decoupling of master API keys managed securely via an external .env injection system.

• Docker Desktop (Make sure the Docker daemon is actively running!)
• Node.js (v18+) & NPM

Torrent ships with a powerful Global CLI for managing your local cluster development experience.

npm install -g @developer_4949/torrent-engine

Before starting the cluster, you must set a Master API Key. This key is securely saved to ~/.torrent/.env and automatically injected into the backend upon boot.

torrent key my_super_secret_key

The CLI handles the orchestration of the entire infrastructure (Zookeeper, Kafka, Redis, PostgreSQL) alongside the Spring Boot microservices and React Dashboards.

torrent start

Wait a minute for the Spring Boot JVMs to compile and boot up inside the Docker containers.

Once the system is up and running, open your browser:

• Developer Console: http://localhost:8081 (Login using the API key you set in Step 2!)
• Demo UI (Simulation): http://localhost:5173

torrent ps     # View the health of all running Torrent containers
torrent stop   # Gracefully shut down the entire cluster

Integrating Torrent into your daily Microservices, Web Apps, or Data Pipelines is incredibly simple. All you need is the Master API Key and an HTTP client!

Just set your .env and use the provided torrent_client.py SDK!

import os
from torrent_client import TorrentClient

# 1. Initialize the client with your secure cluster key
torrent = TorrentClient(
    api_key=os.getenv("TORRENT_API_KEY", "your_secret_key"),
    base_url="http://localhost:8080"
)

# 2. Offload a heavy background task instantly!
job = torrent.submit_job(
    job_type="PROCESS_S3_FILE",
    payload={"bucket": "user-uploads", "file": "video.mp4"},
    priority="HIGH", # Use gRPC Fast-Track!
    cron_expression="* * * * *" 
)

print(f"Successfully offloaded to Torrent! Job ID: {job['id']}")

Since Torrent is entirely language-agnostic via its REST Gateway, you can trigger jobs from any Java backend using standard HttpClient or RestTemplate!

HttpClient client = HttpClient.newHttpClient();

String jsonPayload = """
    {
        "jobType": "SEND_WELCOME_EMAIL",
        "priority": "STANDARD",
        "payload": { "userId": "12345", "email": "user@example.com" },
        "idempotencyKey": "email-signup-12345"
    }
    """;

HttpRequest request = HttpRequest.newBuilder()
    .uri(URI.create("http://localhost:8080/api/v1/jobs"))
    .header("Authorization", "Bearer your_secret_key")
    .header("Content-Type", "application/json")
    .POST(HttpRequest.BodyPublishers.ofString(jsonPayload))
    .build();

// Fire and forget! Let Torrent handle the retries and fault tolerance.
client.sendAsync(request, HttpResponse.BodyHandlers.ofString());

Ready to take Torrent to production? The entire system is built natively for container orchestrators like Kubernetes or cloud-native platform-as-a-service (PaaS) providers.

1. Managed Infrastructure: Spin up managed versions of PostgreSQL, Redis, and Apache Kafka (e.g., Confluent Cloud or Amazon MSK).
2. Environment Variables: Set the following environment variables in your cloud provider's dashboard:
   • SPRING_DATASOURCE_URL, SPRING_DATASOURCE_USERNAME, SPRING_DATASOURCE_PASSWORD
   • SPRING_KAFKA_BOOTSTRAP_SERVERS
   • SPRING_REDIS_HOST, SPRING_REDIS_PORT
   • TORRENT_API_KEY (Your master security key!)
3. Backend Deployment: Deploy the torrent-backend Dockerfile as a Web Service.
4. Console/UI Deployment: Deploy the torrent-console and torrent-ui React applications as Static Sites, pointing VITE_API_URL_BASE to your backend's public URL.

(Note: For the absolute highest performance, ensure your API Gateway and Worker Nodes are deployed in the same VPC/Region to minimize gRPC latency!)