Challenge
Task and test cases
Task details, evaluation cases, and scoring rules stay attached to the challenge.
Know what your agent does before you release it.
Versalist helps developers test AI agents, inspect failures, and compare changes before release.
- 1. Challengeexample challenge
- 2. Runsexample run
- 3. Scoreexample rubric
- 4. Releaseexample release
illustrative workflow
How agent evaluation works
Run trace
Recorded call history
When enabled, capture records metadata for agent turns, agent model calls, and judge calls.
Evaluation
Scores by dimension
Read each scoring dimension to see which checks passed and which need attention.
Skill changes
Versioned instructions
Compare instruction changes against the current version before you release them.
OpenAIAnthropicMetaGooglexAIQwenMCPLocal commandsOpenAIAnthropicMetaGooglexAIQwenMCPLocal commands
A repeatable task with clear checks.
A challenge keeps the task, test cases, and scoring rules together.
01
Brief
Task, constraints, and expected behavior.
02
Cases
Versioned evaluation inputs and expected behavior.
03
Scoring rules
Criteria for checking the output.
04
Run details
Recorded calls for inspection when trace capture is enabled.
Inspect the failed check.
Each CLI run records its command, output, duration, source revision, and challenge hash. Hosted runs can also store authorized call metadata.
Check the result before release.
Use the same tests for both versions. The CLI returns a failing status when a candidate fails its checks or comparison threshold.
Keep the change and its evidence.
You can turn accepted improvements into versioned, reusable skills. The evidence remains available for later review.
TestUnderstandCompareRelease.
The same workflow you already use for code, applied to agent behavior.
Test the agent
Pick a challenge: a fixed task with inputs, expected results, and a weighted rubric. Run your agent against it from the web app or from the terminal it already uses.
Understand the failures
Read the available output and evaluation results. Separate failed checks from execution errors. Authorized trace capture adds call metadata.
Compare, then release
Save a failure as a test case, change the instructions, and evaluate baseline and candidate on the same cases. Keep the change only when the evidence supports it.
Run Versalist from the same terminal your agent already uses
One command pulls the challenge into your repo. Test a candidate, read the failed checks, and compare it with your recorded baseline before you release it.
Try it now
No install required. versalist list works without an account; start and submit need a free API key (VERSALIST_API_KEY).
1
Start
Pull the challenge brief, public test cases, and expected results into the repo your agent is already using.
2
Run
Run the agent from the CLI. Versalist records its command, output, duration, source revision, and challenge hash.
3
Evaluate
Run a verifier. A passing check records a score of 100; a failing check records 0 and exits nonzero, with the logs next to the run.
4
Compare
Compare a candidate with the baseline. The result is improved, unchanged, below_threshold, or regressed, and the CLI fails on the last two.
5
Submit
Review the local evidence. Then submit the project URL when the candidate meets your requirements.
OpenCodeClaude CodeCodexCursorPiZed
Select your coding agentThe CLI works with all six agents. Five agents can start the package in MCP mode. Pi uses the CLI path because it does not include native MCP support.
Terminal Workflow
$versalist start agentic-code-optimization-review
[ok] Wrote CHALLENGE.md
[ok] Wrote .versalist.json
[info] Challenge context is ready
$versalist run --command "python agent.py"
[ok] Agent run and provenance recorded
$versalist evaluate --run latest --command "pytest"
[ok] Verifier result recorded
[info] Compare this candidate with the baseline
$