Agentic Observability Platform

A Dockerized monitoring and automation stack using Grafana, Prometheus, Alertmanager, and AI-driven agentic workflows for real-time system and KPI observability.

• Monitors real systems, not mock data — The CXP exporter authenticates via OAuth2, handles paginated responses, auto-refreshes tokens, and scrapes 9 live API endpoints from a running FastAPI platform with 224 endpoints.
• End-to-end alert pipeline — Not just dashboards. Alerts fire in Prometheus, route through Alertmanager, and trigger an AI-powered analyzer that generates root cause analysis, severity explanation, and suggested actions.
• AI is integrated, not bolted on — Swappable analyzer backends (mock/Claude/OpenAI), AI agent monitoring with token usage and tool call tracking, and an MCP server exposing observability tools to LLM agents.
• Everything is swappable — Collectors (mock → database → API), analyzers (mock → Claude → OpenAI), storage (JSON → SQLite), KPI modules (ai_agents → operations → sales). Factory patterns and ABC base classes throughout.
• Self-validating — A metric validation script checks every PromQL expression in alert rules and dashboards against live Prometheus to catch silent failures before they reach production.
• Tested and CI-ready — 30 pytest tests with HTTP mocking, async FastAPI testing, and a 3-job GitHub Actions pipeline (test + lint + docker build). Rare for infrastructure projects.

• Prometheus — Metrics collection, alert rule evaluation, file-based service discovery
• Grafana — Auto-provisioned dashboards (AI agents, infrastructure, CXP business, CXP platform)
• Alertmanager — Alert routing with Slack/email/webhook support
• Node Exporter — Host-level CPU, memory, disk, network metrics
• Custom Python Exporters — KPI metrics, AI agent health, and CXP platform metrics
• CXP Integration — Real-time metrics from a live FastAPI platform (users, contracts, tickets, assets)
• AI Alert Analyzer — AI-powered root cause analysis with Claude/OpenAI backends
• MCP Server — Expose observability tools for Claude and AI agents (optional)
• Dockerized — Single docker-compose up -d runs the full stack
• Health checks — All services have Docker health checks
• Pinned versions — Production-ready image tags, not :latest

15 panels tracking agent runs (6,442 total), 82.6% success rate, failure breakdown by error type, token usage (input vs output), tool calls by tool, latency heatmap, and per-agent scorecard.

Live data from a real FastAPI platform — CXP API health status (UP), 4 active sessions, users/customers by status, vendors, contracts, scrape duration (444ms), and error tracking.

Real host metrics from Node Exporter — memory usage (25.6%), memory breakdown (Used/Available/Buffers/Cached), filesystem usage per mount, disk I/O by device, and network traffic (rx/tx).

7 scrape targets across 6 jobs — all reporting UP with sub-50ms scrape durations. Agent exporter, CXP exporter, KPI exporter, Node exporter, file-based service discovery, and Prometheus self-monitoring.

15 alert rules across 5 groups. Screenshot shows ServiceDown alert actively firing (2 PENDING instances) after stopping the agent-exporter service — proving the alert pipeline works end-to-end.

AgentExporterDown (1 alert) and ServiceDown (2 alerts) routed by severity. Alertmanager groups, deduplicates, and forwards to the AI Alert Analyzer webhook.

FastAPI service (OAS 3.1) with 4 endpoints: webhook receiver for Alertmanager, analysis listing, per-alert lookup, and health check. Pydantic schemas for Alert, AlertManagerPayload, and validation.

# 1. Clone
git clone https://github.com/your-username/Agentic-Observability-Platform.git
cd Agentic-Observability-Platform

# 2. Setup environment
cp .env.example .env   # Edit with your values

# 3. Start the stack
docker-compose up -d

# 4. Verify all services
docker-compose ps

# 5. Optional: enable MCP layer
docker-compose --profile mcp up -d

                              ┌─────────────────┐
                              │    Grafana       │
                              │    :3005         │
                              │  4 dashboards    │
                              └────────▲─────────┘
                                       │
  ┌──────────────┐              ┌──────┴──────┐
  │Node Exporter │─────────────>│             │
  │   :9100      │              │ Prometheus  │
  └──────────────┘              │   :9090     │
  ┌──────────────┐              │             │
  │ KPI Exporter │─────────────>│  15 alert   │
  │   :8005      │              │  rules      │
  └──────────────┘              │             │
  ┌──────────────┐              └──────┬──────┘
  │Agent Exporter│─────────────>       │
  │   :8002      │              ┌──────▼──────┐       ┌────────────────┐
  └──────────────┘              │Alertmanager │──────>│ Alert Analyzer │
  ┌──────────────┐              │   :9093     │       │ :5100 (AI)     │
  │CXP Exporter  │─────────────>└─────────────┘       └────────────────┘
  │   :8003      │
  └──────┬───────┘              ┌─────────────┐
         │                      │  MCP Server  │
    ┌────▼────┐                 │  :5200       │
    │ CXP API │                 │  (optional)  │
    │ :8000   │                 └──────────────┘
    └─────────┘

├── docker-compose.yml          # Service orchestration (8 services)
├── .env.example                # Environment variable template
├── Makefile                    # Convenience commands
├── prometheus/
│   ├── prometheus.yml          # 6 scrape jobs
│   ├── rules/                  # 15 alert rules across 5 groups
│   └── targets/                # File-based service discovery
├── grafana/
│   ├── provisioning/           # Datasources + dashboard loader
│   └── dashboards/
│       ├── ai-agents/          #   Agent monitoring (15 panels)
│       ├── operations/         #   Infrastructure dashboard (CPU, memory, disk, network)
│       ├── sales/              #   CXP Business dashboard (customers, contracts, vendors)
│       └── cxp/                #   CXP platform dashboard (15 panels)
├── alertmanager/               # Routing config + templates
├── exporters/
│   ├── kpi_exporter/           # AI agent metrics (mock data)
│   ├── agent_exporter/         # Agent health metrics
│   └── cxp_exporter/           # CXP platform metrics (real API data)
├── services/
│   └── alert_analyzer/         # AI-powered alert analysis (FastAPI)
├── agents/
│   ├── workflows/              # Incident responder, anomaly detector, report gen
│   └── tools/                  # Prometheus query, Grafana snapshot, Slack notify
├── kpi_modules/                # Modular KPI plugin system
│   ├── ai_agents/              #   Agent runs, failures, latency, tokens
│   ├── operations/             #   Uptime, requests, errors, CPU, memory
│   └── sales/                  #   Revenue, conversion, pipeline, churn
├── mcp/                        # MCP server + tools (optional)
├── scripts/                    # Setup, validation, seed data
├── tests/                      # Unit tests
└── docs/                       # Architecture docs + screenshots

The CXP Exporter scrapes a live FastAPI platform (224 endpoints, 24 modules) and exposes real business metrics:

Switch between monitoring domains by changing one config value:

# kpi_modules/config.yml
active_module: "ai_agents"   # or "operations" or "sales"

15 alert rules across 5 groups:

30 tests covering the CXP exporter, alert analyzer, and KPI modules:

make test    # Run all tests
make lint    # Run ruff linter

GitHub Actions pipeline runs on every push and PR:

• test — pytest on Python 3.12
• lint — ruff linter on all Python code
• docker-build — builds all custom Docker images + validates compose config

AI-powered alert analysis service with swappable backends:

# Set in .env
ANALYZER_BACKEND=mock    # mock, claude, openai
STORAGE_BACKEND=json_file  # json_file, sqlite

See docs/security.md for production security recommendations including:

• Authentication on exporters and API endpoints
• Secrets management (Docker secrets, Vault)
• Network isolation between services
• TLS/HTTPS with reverse proxy
• Container hardening and resource limits

MIT — see LICENSE for details.

Maria Naseem