A lightweight but extensible platform for evaluating Large Language Models as autonomous tool-using agents across structured reasoning tasks, benchmark datasets, execution traces, and evaluation pipelines.
Most current LLM evaluation systems only measure:
- Final answer accuracy
- Benchmark score
- Token usage
But modern LLM agents are significantly more complex.
They:
- reason step-by-step
- call external tools
- generate intermediate traces
- fail in multiple ways
- behave differently under execution pressure
Traditional benchmarks treat these systems like black boxes.
AgentBench Lite was built to solve that problem.
Current AI benchmarks fail to answer questions like:
| Problem | Traditional Benchmarks |
|---|---|
| How did the model reason? | ❌ |
| Which tools were used? | ❌ |
| Why did execution fail? | ❌ |
| Where was latency introduced? | ❌ |
| How efficient was the agent? | ❌ |
| Can execution be replayed? | ❌ |
This makes modern LLM-agent evaluation:
- opaque
- difficult to debug
- hard to reproduce
- difficult to analyze scientifically
AgentBench Lite introduces a structured execution pipeline for LLM agents.
Instead of evaluating only outputs:
Prompt → Final Answer
The platform evaluates:
Prompt
↓
Reasoning
↓
Tool Selection
↓
Tool Execution
↓
Intermediate State
↓
Evaluation
↓
Execution Trace
↓
Analytics
| Feature | Description |
|---|---|
| Benchmark Datasets | Structured reasoning/task datasets |
| Tool Execution Engine | Real-time tool calling system |
| Agent Execution Loop | Multi-step autonomous execution |
| Replay Viewer | Full execution trace visualization |
| Evaluation Engine | Task scoring & grading |
| Analytics Dashboard | Charts & benchmark insights |
| Export System | CSV/JSON benchmark exports |
| Leaderboards | Model comparison system |
| Execution Traces | Reasoning + tool replay |
| Research Dashboard | Research-focused UX |
┌────────────────────────────────────────────┐
│ Frontend │
│ Next.js + TypeScript UI │
└────────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────┐
│ API Layer │
│ FastAPI Backend │
└────────────────────────────────────────────┘
│
┌────────────┼────────────┐
▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Benchmark │ │ Execution │ │ Analytics │
│ Engine │ │ Engine │ │ System │
└──────────────┘ └──────────────┘ └──────────────┘
│ │ │
▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Tool Calling │ │ Replay Trace │ │ Evaluation │
│ System │ │ System │ │ Engine │
└──────────────┘ └──────────────┘ └──────────────┘
| Technology | Purpose |
|---|---|
| FastAPI | Async API backend |
| Pydantic | Validation & schemas |
| Uvicorn | ASGI server |
| aiofiles | Async dataset loading |
| Groq API | LLM inference |
| OpenRouter | Multi-provider inference |
| Technology | Purpose |
|---|---|
| Next.js 14 | Frontend framework |
| TypeScript | Type-safe frontend |
| TailwindCSS | UI styling |
| shadcn/ui | Component system |
| Recharts | Analytics visualizations |
User selects benchmark
↓
Frontend sends execution request
↓
Backend creates execution session
↓
Model receives task
↓
Agent reasoning loop starts
↓
Tool calls executed
↓
Intermediate traces stored
↓
Evaluation engine scores execution
↓
Analytics updated
↓
Replay trace generated
↓
Frontend renders results
| Stage | Purpose |
|---|---|
| Dataset Loading | Load benchmark tasks |
| Agent Execution | Run reasoning loop |
| Tool Invocation | Execute tools |
| Trace Recording | Save intermediate states |
| Evaluation | Score execution |
| Analytics | Aggregate metrics |
| Replay | Visualize execution |
One of the core research features.
The Replay Viewer visualizes:
- reasoning traces
- tool usage
- execution timing
- intermediate outputs
- failure states
- execution progression
Example:
Step 1:
Reasoning → "Need arithmetic tool"
Step 2:
Tool Call → calculator
Step 3:
Tool Output → 350
Step 4:
Reasoning → "Final answer generated"
The analytics engine tracks:
| Metric | Description |
|---|---|
| Success Rate | Completed tasks |
| Average Latency | Execution timing |
| Tool Frequency | Tool usage patterns |
| Model Performance | Benchmark scoring |
| Failure Types | Error categorization |
| Evaluation Scores | Overall grading |
| Module | Description |
|---|---|
| Dashboard | Research overview |
| Run Benchmark | Benchmark launcher |
| Results | Execution summaries |
| Leaderboard | Model rankings |
| Analytics | Research insights |
| Replay Viewer | Trace inspection |
{
"task_id": "reasoning-001",
"question": "If a train travels 60km in 1 hour, how far in 5 hours?",
"expected_answer": "300"
}{
"tool_name": "calculator",
"input": {
"expression": "60 * 5"
}
}{
"execution_id": "exec_001",
"score": 0.95,
"latency": 1.2,
"tool_calls": 1,
"status": "success"
}AgentBench Lite was designed for:
- LLM evaluation research
- agent systems analysis
- execution observability
- benchmarking workflows
- reasoning inspection
- reproducible experimentation
Modern AI systems are moving toward:
- autonomous agents
- tool usage
- long reasoning chains
- multi-step execution
But evaluation systems have not evolved fast enough.
This project attempts to bridge that gap by turning:
- hidden execution into
- measurable research data
backend/
│
├── app/
│ ├── agents/
│ ├── api/
│ ├── evaluators/
│ ├── llm/
│ ├── schemas/
│ ├── services/
│ ├── tools/
│ └── main.py
│
├── benchmarks/
├── datasets/
├── models/
├── results/
└── .env
frontend/
│
├── src/
│ ├── app/
│ ├── components/
│ ├── services/
│ ├── hooks/
│ └── lib/
│
└── public/
cd backend
venv\Scripts\activate
uvicorn app.main:app --reloadBackend runs at:
http://127.0.0.1:8000
Swagger Docs:
http://127.0.0.1:8000/docs
cd frontend
npm install
npm run devFrontend runs at:
http://localhost:3000
Planned future directions:
- multi-agent benchmarks
- distributed execution
- prompt strategy comparison
- hallucination analysis
- long-context benchmarking
- memory-based agents
- autonomous web agents
- benchmarking reproducibility suites
Independent builder focused on:
- AI systems
- benchmarking infrastructure
- LLM evaluation
- agent tooling
- research engineering
MIT License