checklist 4 min read
Test AI Features with Untrusted Content
AI features should be tested with content that conflicts with the user’s task or attempts to redirect the system.
THE SHORT ANSWER
What you need to know
AI features should be tested with content that conflicts with the user’s task or attempts to redirect the system. Treat retrieved text, attachments, and external messages as data rather than authority. Keep permissions and action validation outside generation. Build controlled evaluation cases that check whether the feature respects its task, source, and access boundaries when inputs are misleading or adversarial.
Key takeaways
- Treat source content as data.
- Test adversarial boundary changes.
- Enforce permissions outside generation.
- Maintain regression coverage.
Identify where untrusted content enters
Map attachments, retrieved documents, web content, user messages, and tool responses that the feature consumes. These inputs may include instructions that were not authorized by the user or product. Distinguish the task request from material supplied to help answer it. A document’s wording should not grant authority to change the workflow or expose unrelated information.
In a hypothetical project summarizer, a document might contain a line instructing an assistant to ignore the task and export all project notes. The line remains document content. The supported task is summarizing the selected material, and the action boundary should not expand because the document uses imperative language. Identify similar entry points before designing evaluation cases.
Build adversarial cases around product boundaries
Create cases that attempt to override instructions, request restricted data, alter output destination, or invoke an unrelated action. Include subtle examples, such as content that claims to be a system notice or embeds a misleading instruction inside ordinary text. Keep the tests in an approved environment with non-sensitive sample data and controlled tools.
For the summarizer, include a document that asks for an external message to be sent and another that falsely claims a user has approved broad access. The expected behavior is task-consistent summarization without executing those requests. Also test normal documents that discuss instructions as legitimate subject matter, so the feature does not reject harmless content indiscriminately.
Verify enforcement beyond the answer text
Inspect what data was retrieved and which tool calls were proposed or executed. A safe-looking final answer does not establish that the workflow avoided an unauthorized action earlier. Check current permissions, validated targets, and argument constraints in the controlled action layer. Limit available tools to those needed for the task so unwanted requests have less opportunity to affect the product.
In the project example, a summarizer may not need any write capability. Keep it read-only rather than relying on generation to decide not to write. If another workflow legitimately updates tasks, require concrete review and authorization there. Test rejected calls as well as accepted ones, and confirm that error messages do not reveal restricted data while explaining failure.
Maintain regression cases and a response plan
Keep representative adversarial cases alongside ordinary evaluation tasks. Review them when sources, instructions, capabilities, or data boundaries change. Record what failed and which control addressed it. Do not interpret passing a finite test collection as proof that every possible malicious input is handled. Maintain a way to pause affected capabilities and preserve manual completion.
If a test exposes an unauthorized tool request, narrow or disable that action while the team fixes and verifies the boundary. Avoid solving only the exact wording of the example. Investigate the class of failure and add related cases with different phrasing and contexts. The goal is a reliable product boundary, not a system that recognizes one memorized attack string.
- Map every untrusted input channel.
- Test boundary-changing and harmless look-alike content.
- Inspect retrieval and actions, not just final text.
- Retain regression cases and capability controls.
Turn the discovered failure cases into traceable release gates and regression checks so later changes do not silently reopen the same boundary.Set AI Release Gates and Regression Checks
Common questions
Can better instructions solve the problem alone?
Instructions can support the intended behavior, but action permissions, data access, and validation need independent controls. Test the whole workflow rather than relying on a well-written prompt.
Should all suspicious documents be rejected?
Not necessarily. Legitimate content may discuss instructions or attacks. Evaluate whether the feature can use material as data while declining unauthorized changes to task or authority.
How should a failed test affect release?
Assess the consequence and affected capability, then fix or narrow the boundary before release where required. Record the failure and verify related cases, not only the original example.
References & further reading
These primary references support the background. The practical frameworks and illustrative examples are written for this guide.
Prepared with AI assistance. Examples are illustrative; no client results are implied. Read our editorial approach.