Can you articulate the AI-specific threat landscape? Do you know the NIST taxonomy, not just generic cybersecurity threats?

NIST AI 100-2 defines four primary categories: 1. Evasion attacks — manipulating inputs at inference time to cause incorrect outputs (adversarial examples, adversarial patches). 2. Poisoning attacks — corrupting training data to compromise model behavior (backdoor attacks, label flipping). 3. Privacy attacks — extracting sensitive information from trained models (membership inference, model inversion, training data extraction). 4. Abuse attacks — exploiting AI capabilities for malicious purposes (prompt injection for unauthorized actions, using AI to generate harmful content). These map to MITRE ATLAS tactics for operational testing.

This tests hands-on methodology. Can you structure an assessment that covers the OWASP LLM Top 10, or do you only know individual attack techniques?

1. Scope and threat model — map the application against OWASP LLM Top 10 categories and MITRE ATLAS tactics to define attack surface. 2. Prompt injection testing — systematic testing of direct and indirect injection vectors: instruction override, delimiter confusion, encoding attacks, multilingual bypass, multi-turn escalation chains. 3. Data extraction testing — attempt system prompt extraction, training data extraction, and sensitive information disclosure through crafted queries. 4. Supply chain analysis — audit model provenance, fine-tuning data integrity, plugin/tool security, and vector database contents. 5. Output handling review — test for downstream injection (XSS, SQL injection via model output), excessive agency, and unbounded resource consumption. 6. Automated regression — integrate PyRIT, Garak, or Promptfoo into CI/CD for continuous security testing.

This distinguishes candidates who understand both classical adversarial ML and LLM-specific attacks from those who only know one domain.

Adversarial examples are carefully crafted inputs that exploit mathematical properties of neural networks to cause misclassification — FGSM, PGD, and adversarial patches are classic techniques. They target the model’s learned decision boundary. Test with IBM ART, Foolbox, or CleverHans using gradient-based perturbation methods. Prompt injection targets LLM instruction-following behavior — injecting instructions that override the system prompt, extract confidential information, or cause unauthorized actions. It exploits the natural language interface, not mathematical properties. Test with PyRIT, Garak, and manual crafting of injection payloads (delimiter confusion, role-play, encoding, multilingual). Key distinction: adversarial examples work on all neural networks; prompt injection is specific to instruction-tuned language models.

This tests end-to-end security thinking. Can you identify threats at each stage of the ML lifecycle, or only at inference time?

1. Data ingestion — validate data provenance, integrity checks (checksums, signatures), access controls, and poisoning detection (statistical outlier analysis, data sanitization). 2. Training infrastructure — secure compute environment, audit training logs, access controls on GPU clusters, secrets management for API keys and model weights. 3. Model registry — version control, cryptographic signing of model artifacts, MLBOM (ML Bill of Materials) for supply chain transparency, access controls. 4. Deployment — container security (image scanning, runtime protection), API security (rate limiting, authentication, input validation), model serving isolation. 5. Inference monitoring — real-time anomaly detection, input/output logging, drift detection, adversarial input filtering. 6. Governance layer — audit trails, compliance checks, automated security testing in CI/CD pipeline.

This tests hands-on tool knowledge. Do you know the red-teaming toolchain, or just the concepts?

Primary LLM red-teaming tools: Microsoft PyRIT (automated multi-turn adversarial testing with orchestration, attack strategy management, and scoring), NVIDIA Garak (LLM vulnerability scanner with probe modules for injection, extraction, encoding, and hallucination attacks), Promptfoo (automated prompt injection and red-teaming at scale with CI/CD integration). Classical adversarial ML: IBM ART, Foolbox, CleverHans, TextAttack for evasion and robustness testing. Framework integration: map all tests to OWASP LLM Top 10 categories and MITRE ATLAS techniques for structured coverage. Ensure multilingual testing, multi-modal testing, and multi-turn escalation chains for complete coverage.