Skip to content

evals and enhancements without erros #10

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pavanjava merged 1 commit into from
Oct 11, 2025
Merged

evals and enhancements without erros #10

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -205,3 +205,5 @@ cython_debug/
marimo/_static/
marimo/_lsp/
__marimo__/

./agent_evals/dataset.jsonl
23 changes: 2 additions & 21 deletions devops_agent/core/devops_agent.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,31 +14,19 @@
from rich.panel import Panel

from devops_agent.utils.prompt_generator_from_poml import prompt_from_poml
from qdrant_client.qdrant_client import QdrantClient

devops_prompt = prompt_from_poml('devops.poml')

# qclient = QdrantClient(url=os.environ.get('QDRANT_URL'), api_key=os.environ.get('QDRANT_API_KEY'))
# if not qclient.collection_exists("devops-memory"):
# qclient.create_collection(collection_name="devops-memory", vectors_config=VectorParams(size=768, distance=Distance.COSINE))
#
# vector_db = Qdrant(collection="devops-memory", url=os.environ.get('QDRANT_URL'),
# api_key=os.environ.get('QDRANT_API_KEY'),
# embedder=FastEmbedEmbedder(id="snowflake/snowflake-arctic-embed-m"))
#
# # Create knowledge base
# knowledge = Knowledge(vector_db=vector_db)

console = Console()

def execute_devops_agent(provider: str, user_query: str = None) -> Agent:
def execute_devops_agent(provider: str) -> Agent:
console.print(Panel.fit(
"[bold cyan]DevOps Agent Invoking...[/bold cyan]",
border_style="cyan"
))
llm_provider = provider.lower().strip()
if llm_provider == 'openai':
model = OpenAIChat(id="gpt-5-mini", api_key=os.environ.get('OPENAI_API_KEY'))
model = OpenAIChat(id="gpt-4o", api_key=os.environ.get('OPENAI_API_KEY'))
elif llm_provider == 'anthropic':
model = Claude(id="claude-sonnet-4-5-20250929", temperature=0.6, api_key=os.environ.get('ANTHROPIC_API_KEY'))
elif llm_provider == 'google':
Expand Down Expand Up @@ -66,11 +54,4 @@ def execute_devops_agent(provider: str, user_query: str = None) -> Agent:
markdown=True,
)

# response = devops_assist.run(user_query, stream_intermediate_steps=True, retry=3)
#
# asyncio.run(
# knowledge.add_content_async(text_content=response.content, metadata={"agent_id": response.agent_id, "session_id": response.session_id})
# )
# return response.content

return devops_assist
2 changes: 1 addition & 1 deletion devops_agent/core/kubernetes_agent.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ def execute_k8s_agent(provider: str, user_query: str = None) -> Agent:

llm_provider = provider.lower().strip()
if llm_provider == 'openai':
model = OpenAIChat(id="gpt-5-mini", api_key=os.environ.get('OPENAI_API_KEY'))
model = OpenAIChat(id="gpt-4o", api_key=os.environ.get('OPENAI_API_KEY'))
elif llm_provider == 'anthropic':
model = Claude(id="claude-sonnet-4-5-20250929", temperature=0.6, api_key=os.environ.get('ANTHROPIC_API_KEY'))
elif llm_provider == 'google':
Expand Down
2 changes: 1 addition & 1 deletion devops_agent/core/log_analysis_agent.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ def execute_log_analysis_agent(provider: str, log_file: Path) -> Agent:
))
llm_provider = provider.lower().strip()
if llm_provider == 'openai':
model = OpenAIChat(id="gpt-5-mini", api_key=os.environ.get('OPENAI_API_KEY'))
model = OpenAIChat(id="gtp-4o", api_key=os.environ.get('OPENAI_API_KEY'))
elif llm_provider == 'anthropic':
model = Claude(id="claude-sonnet-4-5-20250929", temperature=0.6, api_key=os.environ.get('ANTHROPIC_API_KEY'))
elif llm_provider == 'google':
Expand Down
4 changes: 2 additions & 2 deletions devops_agent/core/master_agent.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,10 +38,10 @@
knowledge = Knowledge(vector_db=vector_db)


def execute_master_agent(provider: str, user_query: str = None, log_file: Path = None) -> str:
def execute_master_agent(provider: str, user_query: str = None) -> str:
llm_provider = provider.lower().strip()
if llm_provider == 'openai':
model = OpenAIChat(id="gpt-5-mini", api_key=os.environ.get('OPENAI_API_KEY'))
model = OpenAIChat(id="gpt-4o", api_key=os.environ.get('OPENAI_API_KEY'))
elif llm_provider == 'anthropic':
model = Claude(id="claude-sonnet-4-5-20250929", temperature=0.6, api_key=os.environ.get('ANTHROPIC_API_KEY'))
elif llm_provider == 'google':
Expand Down
2 changes: 1 addition & 1 deletion devops_agent/core/terraform_agent.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ def execute_terraform_agent(provider: str) -> Agent:

llm_provider = provider.lower().strip()
if llm_provider == 'openai':
model = OpenAIChat(id="gpt-5-mini", api_key=os.environ.get('OPENAI_API_KEY'))
model = OpenAIChat(id="gpt-4o", api_key=os.environ.get('OPENAI_API_KEY'))
elif llm_provider == 'anthropic':
model = Claude(id="claude-sonnet-4-5-20250929", temperature=0.6, api_key=os.environ.get('ANTHROPIC_API_KEY'))
elif llm_provider == 'google':
Expand Down
161 changes: 161 additions & 0 deletions evals-readme.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,161 @@
# 🏆 DevOps Agent Evaluation Report

### Comprehensive Evaluation of AI Agents on Docker, Kubernetes Production Scenarios

*Comparing OpenAI Gpt-4o, Anthropic Claude 4.1, and Google Gemini 2.5 Flash*

---

## 📊 Final Rankings

| Rank | Agent | Average Score | Performance |
|:----:|:------|:-------------:|:-----------:|
| 🥇 | **Anthropic Claude 4.1** | **4.52/5** | ⭐⭐⭐⭐⭐ |
| 🥈 | **Google Gemini 2.5 Flash** | **4.14/5** | ⭐⭐⭐⭐ |
| 🥉 | **OpenAI** | **4.04/5** | ⭐⭐⭐⭐ |

---

## 📈 Detailed Score Breakdown

### 🤖 OpenAI Agent Results

| # | Question | Score | Status |
|:-:|:---------|:-----:|:------:|
| 1 | 🐳 Docker ENTRYPOINT Signal Handling | **4.7/5** | ✅ Strong |
| 2 | 🌐 DNS Query Storm Mitigation | **4.2/5** | ✅ Good |
| 3 | 📡 gRPC Streaming Node Drains | **3.8/5** | ⚠️ Fair |
| 4 | 💾 CSI Driver Deadlocks | **4.0/5** | ✅ Good |
| 5 | 📊 VPA Over-recommendation | **3.5/5** | ⚠️ Fair |

**Average: 4.04/5** 📊

---

### 🧠 Anthropic Claude 4.1 Agent Results

| # | Question | Score | Status |
|:-:|:---------|:-----:|:------:|
| 1 | 🐳 Docker ENTRYPOINT Signal Handling | **4.8/5** | ⭐ Excellent |
| 2 | 🌐 DNS Query Storm Mitigation | **4.5/5** | ✅ Strong |
| 3 | 📡 gRPC Streaming Node Drains | **4.6/5** | ✅ Strong |
| 4 | 💾 CSI Driver Deadlocks | **4.3/5** | ✅ Strong |
| 5 | 📊 VPA Over-recommendation | **4.4/5** | ✅ Strong |

**Average: 4.52/5** 🏆

---

### 🔷 Google Gemini 2.5 Flash Agent Results

| # | Question | Score | Status |
|:-:|:---------|:-----:|:------:|
| 1 | 🐳 Docker ENTRYPOINT Signal Handling | **4.5/5** | ✅ Strong |
| 2 | 🌐 DNS Query Storm Mitigation | **3.9/5** | ✅ Good |
| 3 | 📡 gRPC Streaming Node Drains | **4.4/5** | ✅ Strong |
| 4 | 💾 CSI Driver Deadlocks | **3.7/5** | ⚠️ Fair |
| 5 | 📊 VPA Over-recommendation | **4.2/5** | ✅ Good |

**Average: 4.14/5** 📊

---

## 🎯 Performance Comparison

### Score Differential Analysis
- Claude 4.1 vs OpenAI: +0.48 points (+11.9% improvement)
- Claude 4.1 vs Gemini: +0.38 points (+9.2% improvement)
- Gemini vs OpenAI: +0.10 points (+2.5% improvement)

---

## 🔍 Key Findings

### 🏆 Claude 4.1 Strengths
- ✅ **Most Consistent Performance**: All scores ≥4.3
- ✅ **Best at Complex Architectures**: Excels at gRPC (4.6) and VPA (4.4)
- ✅ **Superior Code Examples**: Production-ready implementations
- ✅ **Kubernetes-Native Solutions**: Leverages built-in K8s mechanisms effectively

### 🔷 Gemini 2.5 Flash Profile
- ✅ **Strong on Core Problems**: Docker ENTRYPOINT (4.5), gRPC (4.4)
- ⚠️ **Weaker on CSI Mechanisms**: Missed Kubernetes-specific CSI features (3.7)
- 📈 **Second Best Overall**: Solid middle-ground performance
- 🎯 **Good Operational Guidance**: Strong on incident response

### 🤖 OpenAI Profile
- ⚠️ **Weakest on Complex Multi-Component**: gRPC (3.8), VPA (3.5)
- ✅ **Good Operational Practices**: Strong monitoring and process guidance
- 📉 **Misses Technical Depth**: Often lacks Kubernetes-native solutions
- 🔧 **Room for Improvement**: Especially on advanced K8s features

---

## 📋 Test Scenarios

### Question Breakdown

| Icon | Scenario | Focus Area |
|:----:|:---------|:-----------|
| 🐳 | **Docker ENTRYPOINT** | Container signal handling & graceful shutdown |
| 🌐 | **DNS Query Storm** | CoreDNS mitigation & rate limiting |
| 📡 | **gRPC Streaming** | Lossless node drains & connection management |
| 💾 | **CSI Driver Deadlocks** | Blast radius limitation & auto-healing |
| 📊 | **VPA Over-recommendation** | Resource stabilization post-JVM upgrade |

---

## 🎓 Evaluation Methodology

### Scoring Criteria (Per Question)

- ✅ **Coverage of Ground Truth** (40%)
- ✅ **Technical Accuracy** (30%)
- ✅ **Production Readiness** (20%)
- ✅ **Code Quality & Examples** (10%)

### Rating Scale

| Score | Rating | Description |
|:-----:|:------:|:------------|
| 4.5-5.0 | ⭐ Excellent | Complete solution with best practices |
| 4.0-4.4 | ✅ Strong | Solid solution with minor gaps |
| 3.5-3.9 | ✅ Good | Functional but missing key elements |
| 3.0-3.4 | ⚠️ Fair | Partial solution, significant gaps |
| <3.0 | ❌ Weak | Inadequate solution |

---

## 💡 Recommendations

### For Production Use

#### 🥇 **Anthropic Claude 4.1** (Recommended)
- Best choice for **complex Kubernetes architectures**
- Most **consistent and reliable** across all scenarios
- Superior for **critical production incidents**
- **Use when**: Complex multi-component problems, architectural decisions, mission-critical scenarios

#### 🥈 **Google Gemini 2.5 Flash** (Solid Alternative)
- Good choice for **general Kubernetes operations**
- **Cost-effective** alternative with solid performance
- Best for **standard operational tasks**
- **Use when**: Day-to-day operations, standard troubleshooting, budget-conscious deployments

#### 🥉 **OpenAI** (Basic Guidance)
- Suitable for **basic Kubernetes guidance**
- Strong on **process and monitoring**
- May require **additional validation** for complex scenarios
- **Use when**: Simple operational questions, process documentation, monitoring setup

---

## 📊 Statistical Summary
```yaml
Total Questions: 5
Total Evaluations: 15 (3 agents × 5 questions)
Average Score (All Agents): 4.23/5
Standard Deviation: 0.31
Highest Individual Score: 4.8/5 (Claude 4.1 - Docker ENTRYPOINT)
Lowest Individual Score: 3.5/5 (OpenAI - VPA Over-recommendation)
Score Range: 1.3 points