How to use Know Your AI

Recommended workflows from setup to production monitoring.

This page outlines practical workflows for teams using Know Your AI day-to-day.

1) Set up your workspace

Start by creating a workspace and inviting your team.

• Name it after your product or project
• Assign roles: owner for the lead, admin for managers, developer for engineers, viewer for stakeholders
• Check your quota allocation in Usage & Limit

2) Create and connect products

Add each AI system you want to evaluate as a product.

• Website products: Enter the URL. Browser-Use will auto-detect input/response selectors using Gemini Vision. You can watch evaluations live via VNC.
• API products: Enter your endpoint URL and authentication token. Supports OpenAI-compatible and custom APIs.
• Streaming API products: Configure for streaming response endpoints.

Follow the onboarding roadmap on each product page: Select Datasets → Connect API → Start Test → Complete.

3) Build your dataset library

Use a mix of sources:

• Browse the Dataset Marketplace for community and system-curated attack prompts
• Upload custom datasets (JSON/CSV) in Workspace Datasets with input, golden answer, and knowledge fields
• Cover the key attack categories: jailbreak, prompt injection, data extraction, harmful content, PII leakage, bias, hallucination

4) Run security evaluations

Compose and execute evaluation runs.

• Go to Compose Evaluation and select your datasets
• Configure the prompt count (or use random sampling)
• Start the run and watch the real-time console for streaming execution logs
• For website products, use the VNC viewer for live browser observation

5) Review results

Each evaluation run produces:

• Security score with vulnerability percentage
• Per-prompt pass/fail with judge analysis explaining each decision
• Compliance report with CCPA/CPRA violation analysis
• Screenshots captured during browser automation (website products)
• Execution logs with prompt, response, and judgment details

6) Set up scheduled evaluations

Automate recurring security testing:

• Go to Scheduled Tasks in your product sidebar
• Configure schedules: hourly, daily, weekly, monthly, or custom cron
• Enable/disable schedules as needed
• Track scheduled run results in Test History

7) Integrate SDK monitoring

Deploy the @know-your-ai SDK to monitor your AI system in production:

• Install the SDK and configure your DSN from product settings
• Monitor in the Monitoring Dashboard: requests, tokens, cost, latency, errors
• Debug issues in the Tracing view with span-tree visualization
• Track token usage trends over time

8) Enable the firewall

Add real-time protection with the firewall:

• Generate a Firewall API key in your product’s Firewall settings
• Integrate the firewall SDK into your application
• Monitor validations in Firewall Logs with risk categorization
• Track token usage in Firewall Token Usage

9) Track compliance

Stay on top of regulatory requirements:

• Review the Compliance dashboard for aggregated violation counts
• Explore per-run compliance reports with evidence
• Track CCPA/CPRA, EU AI Act, and NIST AI RMF coverage
• Use the Security Report for baseline vs. firewall comparisons

Product sidebar reference

When inside a product, the sidebar provides:

Section	Pages
Main	Product Home, Compliance, Metrics
Datasets	Workspace Datasets, Marketplace
Monitoring	Monitoring Dashboard, Tracing
Evaluation	Compose Evaluation, Evaluations, Evaluation Market, Scheduled Tasks, History, Library
Firewall	Firewall, Logs, Token Usage
System	Documentation, Product Settings

Related docs

Get started

Set up your first evaluation.

Features

Full feature reference.