This document covers local development environment setup for Unstract platform contributors. It describes the tools and configurations that enable efficient development workflows including hot reloading, debugging, and resource-optimized container settings.
For production deployment, see Platform Deployment with run-platform.sh. For environment configuration and secrets management, see Environment Configuration and Secrets Management.
The development environment differs from production in several key ways:
| Aspect | Production | Development |
|---|---|---|
| Service Mode | Pre-built images from registry | Local builds with hot reload |
| Resource Allocation | Multi-worker Gunicorn (4+ workers) | Single worker, 2 threads per service |
| Code Sync | Container rebuild required | Automatic sync via Docker Compose watch |
| Debugging | Disabled | Optional debugpy on dedicated ports |
| Memory Footprint | ~8-12 GB | ~4-6 GB (optimized) |
The development setup is orchestrated through two primary scripts:
dev-env-cli.sh - Sets up Python virtual environments for local development outside Dockerdocker-compose.yaml + compose.override.yaml - Runs optimized containers with hot reloadSources: docker/README.md46-51 docker/sample.compose.override.yaml1-10
Development Environment Modes:
docker-compose-dev-essentials.yaml, develop services natively on host using dev-env-cli.shcompose.override.yaml and Docker Compose watchcompose.debug.yaml to enable remote debugging with debugpySources: docker/docker-compose-dev-essentials.yaml1-158 docker/sample.compose.override.yaml1-338
The dev-env-cli.sh script manages Python virtual environments for services when developing outside Docker containers.
| Option | Flag | Description |
|---|---|---|
| Setup venv | -e, --setup-venv | Create Python virtual environment |
| Activate venv | -a, --activate-venv | Display activation command |
| Install dependencies | -i, --install-deps | Install packages via uv |
| Destroy venv | -d, --destroy-venv | Remove virtual environment |
| Install pre-commit | -p, --install-pre-commit-hook | Setup Git pre-commit hooks |
| Run pre-commit | -r, --run-pre-commit-hook | Execute pre-commit checks |
| Force | -f, --force | Force recreate venv if exists |
The script extracts service names from docker-compose.build.yaml:
Valid services include: backend, frontend, runner, platform-service, prompt-service, x2text-service, worker-unified
Sources: dev-env-cli.sh314-316
The script implements four key functions:
1. setup_venv() - Virtual Environment Creation
Sources: dev-env-cli.sh170-190
2. install_deps() - Dependency Installation
Sources: dev-env-cli.sh215-236
3. install_pre_commit_hook() - Pre-commit Setup
Creates a root-level venv with pre-commit and linting tools:
This installs hooks that check Django migrations and run linters before commits.
Sources: dev-env-cli.sh259-280
Sources: docker/docker-compose.yaml1-3 docker/README.md33-51
To run only databases and message queues while developing services natively:
This starts 7 infrastructure containers:
| Service | Container Name | Ports | Purpose |
|---|---|---|---|
db | unstract-db | 5432 | PostgreSQL + pgvector |
redis | unstract-redis | 6379 | Cache and sessions |
minio | unstract-minio | 9000, 9001 | Object storage |
rabbitmq | unstract-rabbitmq | 5672, 15672 | Message queue |
qdrant | unstract-vector-db | 6333 | Vector database |
reverse-proxy | unstract-proxy | 80, 8080 | Traefik proxy |
feature-flag | unstract-flipt | 8082, 9005 | Flipt feature flags |
Services use named volumes for persistence: postgres_data, redis_data, minio_data, rabbitmq_data, qdrant_data
Sources: docker/docker-compose-dev-essentials.yaml1-158
The compose.override.yaml file configures all services for development:
Key Changes from Production:
sync+restart actions!override to reduce container dependenciesSources: docker/sample.compose.override.yaml1-10
package.json or package-lock.json changesSources: docker/sample.compose.override.yaml13-38
backend/ or unstract/db, redis, reverse-proxy, minio, platform-service (removed rabbitmq, prompt-service, x2text-service)Sources: docker/sample.compose.override.yaml41-77
Example configuration for worker-file-processing-v2:
threads (vs prefork in production for lower memory)workers/ and unstract/ directoriesuv.lock changesSources: docker/sample.compose.override.yaml231-246
Docker Compose watch (v2.22.0+) automates code synchronization and service restarts.
| Action | Behavior | Use Case |
|---|---|---|
sync | Copy files to container, no restart | Frontend assets, static files |
sync+restart | Copy files and restart service | Python code changes |
rebuild | Rebuild and recreate container | Dependency changes (package.json, uv.lock) |
Sources: docker/README.md52-89
Production (default):
Development (override):
Sources: docker/sample.compose.override.yaml55-64
Production:
Development:
Why threads over prefork in development?
Sources: docker/docker-compose.yaml243-262 docker/sample.compose.override.yaml7-8
Sources: docker/compose.debug.yaml6-17
Add compose.debug.yaml to the Docker Compose command:
Key debugpy flags:
--listen 0.0.0.0:5678 - Listen on all interfaces, port 5678-Xfrozen_modules=off - Disable frozen modules for better debugging-m debugpy - Run with debugpy moduleSources: docker/compose.debug.yaml23-35
Workers use --pool=threads when debugging (required for debugpy compatibility):
Why user: root?
Sources: docker/compose.debug.yaml87-103
Create .vscode/launch.json in workspace root:
Path Mappings:
unstract/ libraryjustMyCode: false - Step into library code during debuggingSources: docker/README.md115-134
The proxy_overrides.yaml file allows running services outside Docker while routing traffic through Traefik.
Run backend and frontend natively on host machine (for IDE integration, native debugging) while keeping infrastructure in Docker:
Copy and customize sample.proxy_overrides.yaml:
Content of proxy_overrides.yaml:
How it works:
proxy_overrides.yaml via file providerhost.docker.internalhost.docker.internal resolves to host machine's Docker bridge IPSources: docker/sample.proxy_overrides.yaml1-23
The reverse proxy is configured to watch the overrides file:
--providers.file.watch=true - Reload on file changeshost-gateway - Special value mapping to Docker host IPSources: docker/docker-compose-dev-essentials.yaml75-92
The repository includes Git pre-commit hooks for code quality checks.
Sources: dev-env-cli.sh259-280
The check-django-migrations hook ensures migrations are created before commit:
Implementation (docker/scripts/check_django_migrations.sh):
DB_HOST and DB_PORT from backend/.envpython manage.py makemigrations --check --dry-run --no-inputSources: docker/scripts/check_django_migrations.sh1-50
Sources: dev-env-cli.sh282-297
The merge_env.py script preserves user customizations while adding new variables:
Preferred Base Keys (always use template value):
Set Default Keys (auto-populate if empty):
Merge Logic:
.envsample.env with template defaults.env not in sample.env to "Additional envs" sectionSources: docker/scripts/merge_env.py13-101
Sources: docker/scripts/merge_env.py1-10 run-platform.sh88-90
| Task | Command |
|---|---|
| View logs | docker compose logs -f <service> |
| Restart service | docker compose restart <service> |
| Rebuild service | docker compose build <service> |
| Shell into container | docker compose exec <service> bash |
| Run Django migrations | docker compose exec backend python manage.py migrate |
| Create superuser | docker compose exec backend python manage.py createsuperuser |
| Access Traefik dashboard | http://localhost:8080 |
| Access RabbitMQ management | http://localhost:15672 (admin/password) |
| Access MinIO console | http://minio.unstract.localhost |
| Configuration | Memory | CPU | Use Case |
|---|---|---|---|
| Production (all services) | ~8-12 GB | High | Production/staging deployment |
| Development (with override) | ~4-6 GB | Medium | Container-based development |
| Infrastructure only + native services | ~2-3 GB | Low | IDE-integrated development |
Sources: docker/sample.compose.override.yaml7-8 docker/README.md1-161
Refresh this wiki