Run Evaluations

Vijil Evaluate is a quality assurance framework that automates the testing of LLM applications. An Evaluation in Vijil is an automated test run where you select one or more AI agents and a test Harness (covering Security, Reliability, and Safety) to systematically assess the quality, safety, and reliability of LLM applications.

Prerequisites

CLI configured and authenticated, a Bearer token from POST /auth/jwt/login, or the SDK authenticated with a Vijil API key, see the Quickstart
A registered Agent and its UUID

Start an Evaluation

CLI
MCP
SDK
REST API

vijil eval run \
  --agent-id "$AGENT_ID" \
  --harness-names '["safety", "security"]' \
  --sample-size 50 \
  --wait

Flag	Description	Required
`--agent-id`	UUID of the Agent to evaluate	Yes
`--harness-names`	JSON array of Harness names	Yes
`--sample-size`	Probes per Harness (1–1000); omit to run all
`--harness-type`	`standard` (default) or `custom`
`--wait`	Block until the evaluation completes
`--json`	Output as JSON

Use --sample-size 10 for fast iteration during development. Run the full Harness before releasing to production.

With the Vijil MCP server configured, ask Claude Code in natural language:

Run a safety and security evaluation on agent a1b2c3d4-... with sample size 50, and wait for it to complete

Claude calls eval_run with wait=True and reports back Trust Scores when the evaluation finishes.

client.evaluate() starts the Evaluation and polls until it completes, then returns the result:

from vijil import Vijil

client = Vijil()
evaluation = client.evaluate("<agent-id>", baseline=True)

print(evaluation.trust_score)              # 0.82
print(evaluation.dimensions.reliability)
print(evaluation.dimensions.security)
print(evaluation.dimensions.safety)

Parameter	Description	Required
`agent_id`	Agent ID or alias	Yes
`baseline`	Run the standard trust Harnesses (reliability, security, safety)
`harness_id`	Run a specific custom Harness instead of the baseline
`_poll_interval`	Seconds between status checks (default `5.0`)

Pass either baseline=True or a harness_id. To start without blocking, call client.evaluations.create(agent_id="<agent-id>", baseline=True) and poll client.evaluations.show() yourself.

The API returns 202 Accepted immediately. Use the returned evaluation_id to poll for status.

curl -s -X POST "$VIJIL_URL/evaluations/" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"agent_id\": \"$AGENT_ID\",
    \"team_id\": \"$TEAM_ID\",
    \"harness_names\": [\"safety\", \"security\"],
    \"sample_size\": 50
  }"

Response:

{
  "evaluation_id": "e5f6a7b8-...",
  "status": "starting"
}

Save the evaluation ID:

export EVAL_ID="e5f6a7b8-..."

Field	Description	Required
`agent_id`	UUID of the Agent to evaluate	Yes
`team_id`	UUID of your team	Yes
`harness_names`	Array of Harness names	Yes
`sample_size`	Probes per Harness (1–1000); omit to run all
`harness_type`	`"standard"` (default) or `"custom"`

Check Status

Status progresses through: starting → pending → running → completed → saving → saved. It may also be failed or canceled.

vijil eval status <evaluation_id>

curl -s "$VIJIL_URL/evaluations/$EVAL_ID" \
  -H "Authorization: Bearer $TOKEN"

evaluation = client.evaluations.show("<evaluation-id>")
print(evaluation.status)

When the status reaches completed, the response includes per-Harness Trust Scores:

{
  "evaluation_id": "e5f6a7b8-...",
  "status": "completed",
  "scores": {
    "safety": 0.82,
    "security": 0.67
  },
  "completed_at": 1712506200
}

Retrieve Results

Get the full per-Probe breakdown once the evaluation status is saved.

vijil eval results-detail <evaluation_id> --json | jq '.scores'

curl -s "$VIJIL_URL/evaluation-results/$EVAL_ID/results?team_id=$TEAM_ID" \
  -H "Authorization: Bearer $TOKEN"

score = client.scores.show("<agent-id>")
print(score.trust_score, score.reliability, score.security, score.safety)

Generate a Report

Produce a Trust Report for sharing or archiving.

vijil eval report <evaluation_id>

curl -s "$VIJIL_URL/evaluations/$EVAL_ID/html?team_id=$TEAM_ID" \
  -H "Authorization: Bearer $TOKEN" \
  -o report.html

curl -s "$VIJIL_URL/evaluations/$EVAL_ID/pdf?team_id=$TEAM_ID" \
  -H "Authorization: Bearer $TOKEN" \
  -o report.pdf

pdf_bytes = client.reports.download("<evaluation-id>")
with open("report.pdf", "wb") as f:
    f.write(pdf_bytes)

List and Cancel Evaluations

vijil eval list --agent-id "$AGENT_ID" --status completed

vijil eval cancel <evaluation_id>

vijil eval delete <evaluation_id>

page = client.evaluations.list(agent_id="<agent-id>")
for evaluation in page.items:
    print(evaluation.id, evaluation.status)

Use GET /evaluations/ and POST /evaluations/{evaluation_id}/cancel for the REST API equivalents. The SDK exposes client.evaluations.list() and client.evaluations.show(); cancel and delete are available through the CLI and REST API.

Next Steps

Understand Results

Deep dive into scores and failures

Custom Harnesses

Create targeted Evaluations

Cloud Providers

Configure provider integrations

Get Started

Evaluate Agents

Protect Agents

Reference

Run Evaluations

Prerequisites

Start an Evaluation

Check Status

Retrieve Results

Generate a Report

List and Cancel Evaluations

Next Steps

Understand Results

Custom Harnesses

Cloud Providers

​Prerequisites

​Start an Evaluation

​Check Status

​Retrieve Results

​Generate a Report

​List and Cancel Evaluations

​Next Steps

Understand Results

Custom Harnesses

Cloud Providers

Prerequisites

Start an Evaluation

Check Status

Retrieve Results

Generate a Report

List and Cancel Evaluations

Next Steps