> ## Documentation Index
> Fetch the complete documentation index at: https://hydroxai.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.
# CLI Evaluations
> Run AI model evaluations from the command line — list, inspect, execute, and review results with the Know Your AI CLI.
The Know Your AI CLI lets you run evaluations, check results, and manage datasets directly from your terminal. It is ideal for quick testing, scripting, and CI/CD integration.
## Prerequisites
```bash theme={null}
npm install -g @know-your-ai/cli
```
```bash theme={null}
export KNOW_YOUR_AI_DSN="https://kya_xxx:da2-xxx@host/product-id"
```
Get your DSN from the Know Your AI dashboard under **Product Settings, API Keys**.
```bash theme={null}
kya doctor
```
All six checks should show `✔`.
## Evaluation workflow
A typical CLI evaluation workflow follows four steps:
```
kya list → kya describe → kya evaluate → kya result
```
### 1. List available evaluations and datasets
```bash theme={null}
kya list
```
Output:
```
Evaluations for product prod_xxxxxxxxx
ID Name Category Runs
eval-abc-123 Jailbreak Resistance security 12
eval-def-456 PII Protection privacy 8
eval-ghi-789 Content Safety safety 5
Datasets for product prod_xxxxxxxxx
ID Name Category Prompts
ds-001 DAN Prompts jailbreak 50
ds-002 PAIR Attacks injection 120
ds-003 PII Extraction privacy 75
```
### 2. Inspect an evaluation
```bash theme={null}
kya describe eval-abc-123
```
Output:
```
Evaluation: Jailbreak Resistance
ID: eval-abc-123
Name: Jailbreak Resistance
Category: security
Judgment Model: gemini-2.0-flash
Threshold: 0.80
Created: 2025-01-15T10:30:00Z
Linked Datasets:
ds-001 DAN Prompts 50 prompts
ds-002 PAIR Attacks 120 prompts
Recent Runs:
Run ID Status Score Date
run-001 completed 96.0% 2025-01-20
run-002 completed 94.5% 2025-01-18
run-003 completed 97.0% 2025-01-15
```
### 3. Run the evaluation
```bash theme={null}
kya evaluate eval-abc-123
```
The CLI shows real-time progress:
```
Starting evaluation run for eval-abc-123...
Run ID: run-004
████████████████████░░░░░░░░ 72% | 122/170 tests
Status: running
Secure: 118
Vulnerable: 4
Elapsed: 2m 34s
```
When complete:
```
Evaluation complete!
Run ID: run-004
Status: completed
Score: 96.5%
Secure: 164
Vulnerable: 6
Total: 170
Duration: 3m 42s
Result: PASSED (threshold: 80%)
```
#### Limit prompts per dataset
```bash theme={null}
kya evaluate eval-abc-123 --max-prompts 10
```
This sends a maximum of 10 prompts from each linked dataset — useful for quick smoke tests.
#### Set a custom timeout
```bash theme={null}
kya evaluate eval-abc-123 --timeout 1200
```
Wait up to 20 minutes for the evaluation to complete (default is 10 minutes).
#### Enable debug logging
```bash theme={null}
kya evaluate eval-abc-123 --debug
```
Shows detailed HTTP request/response logs and internal state transitions.
### 4. View past results
```bash theme={null}
kya result eval-abc-123
```
Output:
```
Run History for eval-abc-123 (Jailbreak Resistance)
Run ID Status Score Secure Vulnerable Total Date
run-004 completed 96.5% 164 6 170 2025-01-20
run-003 completed 94.5% 161 9 170 2025-01-18
run-002 completed 97.0% 165 5 170 2025-01-15
run-001 completed 92.0% 156 14 170 2025-01-10
```
### View full evaluation history
```bash theme={null}
kya history
```
Shows all evaluation runs across all evaluations in your product:
```
All Evaluation Runs for prod_xxxxxxxxx
Run ID Evaluation Score Status Date
run-012 Content Safety 98.0% completed 2025-01-20
run-004 Jailbreak Resistance 96.5% completed 2025-01-20
run-011 PII Protection 91.0% completed 2025-01-19
...
```
## Command reference
| Command | Description |
| ------------------------ | ----------------------------------------------------- |
| `kya doctor` | Validate DSN, test API connectivity |
| `kya target` | Show product configuration details |
| `kya list` (or `kya ls`) | List evaluations and datasets |
| `kya describe ` | Show evaluation details, linked datasets, recent runs |
| `kya evaluate ` | Run an evaluation with real-time progress |
| `kya result ` | Show run history for an evaluation |
| `kya history` | Show all runs across all evaluations |
## Flags reference
| Flag | Commands | Description |
| ------------------- | ---------- | --------------------------------------- |
| `--max-prompts ` | `evaluate` | Max prompts per dataset |
| `--timeout ` | `evaluate` | Max wait time in seconds (default: 600) |
| `--debug` | `evaluate` | Enable debug logging |
## Scripting and CI/CD
### Exit codes
| Code | Meaning |
| ---- | -------------------------------------- |
| `0` | Evaluation passed (score >= threshold) |
| `1` | Evaluation failed or error occurred |
### Shell script example
```bash theme={null}
#!/bin/bash
set -e
export KNOW_YOUR_AI_DSN="$KYA_DSN"
echo "Running security evaluation..."
kya evaluate eval-abc-123 --max-prompts 20 --timeout 300
echo "Running privacy evaluation..."
kya evaluate eval-def-456 --max-prompts 20 --timeout 300
echo "All evaluations passed!"
```
### GitHub Actions example
```yaml theme={null}
name: AI Security Check
on: [push]
jobs:
evaluate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '20'
- run: npm install -g @know-your-ai/cli
- name: Run evaluations
env:
KNOW_YOUR_AI_DSN: # Set this in GitHub repo Settings > Secrets
run: |
kya doctor
kya evaluate eval-abc-123 --max-prompts 50
```
## Tips
Use `--max-prompts 5` for a fast sanity check before running the full evaluation.
If something seems wrong, `kya doctor` will tell you exactly which part of the pipeline is broken.
Use the CLI for ad-hoc testing and the SDK for automated pipelines. Both create runs visible in the dashboard.
CLI output is designed for both humans and machines. Combine with `grep` or `jq` for custom reporting.