agent-evaluation

Evaluate LLM agents using behavioral regression tests, capability assessments, and reliability metrics. This skill helps identify issues before deployment, addressing the challenges of testing LLM agents where outputs can vary and correctness isn't always definitive. It focuses on building robust evaluation frameworks to improve agent reliability.

The skill includes methods like statistical test evaluation, behavioral contract testing, and adversarial testing. It also highlights anti-patterns such as single-run testing, only happy path tests, and output string matching. The goal is to bridge the gap between benchmark performance and real-world application.

Addresses sharp edges like agents failing in production despite benchmark success by preventing data leakage and providing multi-dimensional evaluation to prevent gaming the metrics.

What It Does

Provides tools and methodologies for testing and benchmarking LLM agents, including behavioral testing, capability assessment, reliability metrics, and production monitoring.

When To Use

Use when you need to test agent performance, evaluate agent capabilities, benchmark agents against each other, assess agent reliability, or perform regression testing on agents.

Installation

Copy SKILL.md to your skills directory