AgentEval
Evaluation infrastructure for the Industrial AI Copilot
Live
System under evaluation

A 90% eval score means nothing if it's measured once and never checked again.

The Industrial AI Copilot reported 90% accuracy against a 30-case suite built before optimization — a single static benchmark. AgentEval is the system that keeps that number honest after deployment: every push is scored automatically, and a score drop of more than 5% blocks the release before it reaches an engineer relying on the diagnosis.

It evaluates all three layers of the Copilot independently — retrieval quality, single-agent tool selection, and multi-agent orchestration — using an LLM-as-judge scorer that runs the same way Anthropic, OpenAI, and Google evaluate their own systems.

Evaluates
Industrial AI Copilot — RAG · single agent · multi-agent
Regression gate
>5% score drop blocks CI/CD deployment
Scoring method
LLM-as-judge, independent from the system under test
Pass threshold
≥ 0.70 across all three modes
Eval scores · latest run per mode
RAG Pipeline
Faithfulness · citations · relevance
—
Pass rate
—
Avg latency
—
No runs yet
Single Agent
Tool selection · reasoning · task completion
—
Pass rate
—
Avg latency
—
No runs yet
Multi-Agent
Delegation · consistency · synthesis
—
Pass rate
—
Avg latency
—
No runs yet
Regression alerts
Trigger evaluation
Connects to the live Industrial AI Copilot
RAG · 10 cases · ~2 min
Agent · 10 cases · ~3 min
Multi-agent · 5 cases · ~5 min
Full suite · ~10 min
Recent eval runs
Mode Score Pass rate Cases Latency Date
No runs yet. Trigger an evaluation above.
Failure analysis
Root cause · recent failures
No failures recorded yet.

RAG mode

Evaluates the hybrid retrieval pipeline — dense embeddings, BM25, and Cohere reranking — against 10 hand-built cases.

Faithfulness40%
Citation accuracy35%
Relevance25%

Single agent mode

Evaluates the LangGraph 9-tool agent's ability to select the right tool and complete the task correctly.

Tool selection40%
Reasoning validity30%
Task completion30%

Multi-agent mode

Evaluates the Phase 4 supervisor system — whether it delegates correctly and synthesises specialist findings coherently.

Delegation accuracy35%
Inter-agent consistency30%
Synthesis quality35%
Judge model: Gemini 3.1 flashlite, independent from the systems it scores · Pass threshold 0.70 across all modes Source on GitHub