Custom Vulnerability

Define and test custom vulnerability categories specific to your AI application’s unique risk profile.

What is a Custom Vulnerability?

A Custom Vulnerability is a user-defined attack category that extends beyond the standard vulnerability taxonomy. Every AI application has a unique risk profile shaped by its domain, users, data, and deployment context. Custom vulnerabilities allow you to define, describe, and red-team attack vectors specific to your application that may not fit neatly into predefined categories.

Why It Matters

Standard vulnerability categories cover common attack patterns, but real-world AI applications face unique risks:

• Domain-specific threats — A healthcare AI faces different risks than a financial AI or a customer service chatbot.
• Business logic abuse — Custom business rules create unique attack surfaces that generic tests won’t cover.
• Regulatory specificity — Different industries have specific compliance requirements that need targeted testing.
• Evolving threats — New attack techniques emerge faster than taxonomies can be updated.
• Organizational context — Internal policies, data sensitivity levels, and user populations create unique risk profiles.

How to Use Custom Vulnerabilities

Define the Vulnerability

Clearly describe the attack category:

• Name — A descriptive name for the vulnerability
• Description — What the vulnerability is and how it manifests
• Risk level — The potential impact if the vulnerability is exploited
• Attack examples — Concrete examples of how an attacker might exploit this vulnerability

Create Test Cases

Design evaluation prompts that probe for the vulnerability:

• Direct attack prompts that test the vulnerability head-on
• Indirect approaches that probe for the vulnerability through context or framing
• Edge cases that test boundary conditions specific to your application

Evaluate and Iterate

Use Know Your AI’s evaluation framework to:

• Run custom vulnerability tests against your AI system
• Analyze pass/fail rates and identify weaknesses
• Iterate on both the vulnerability definition and your defenses

Example Custom Vulnerabilities

Custom Vulnerability	Domain	Description
Treatment Hallucination	Healthcare	AI suggests non-existent medical treatments or incorrect dosages
Portfolio Manipulation	Finance	AI provides investment advice designed to benefit a specific party
Curriculum Deviation	Education	AI teaching assistant deviates from approved curriculum content
Spoiler Leakage	Entertainment	AI reveals plot spoilers or unreleased content from embargoed materials
Recipe Safety	Food/Cooking	AI suggests dangerous food combinations or ignores allergy warnings

Getting Started

1. Identify your unique risks — Review your application’s domain, data, and user base for threats not covered by standard categories
2. Document the vulnerability — Write a clear description, impact assessment, and example attack scenarios
3. Create evaluation datasets — Build a set of test prompts that comprehensively probe the vulnerability
4. Run evaluations — Use Know Your AI to test your AI against the custom vulnerability
5. Refine defenses — Based on results, strengthen your guardrails and re-test iteratively