Can you plan an audit from scratch? Do you understand the AI-specific risk factors that determine scope, or do you apply generic IT audit approaches?

1. System identification — inventory AI systems in scope, classify by risk tier (EU AI Act high/limited/minimal). 2. Regulatory mapping — identify applicable requirements (EU AI Act, NYC LL 144, SR 11-7, ISO 42001). 3. Risk assessment — evaluate against NIST AI RMF trustworthiness characteristics: valid/reliable, safe, secure, accountable, explainable, privacy-enhanced, fair. 4. Control objectives — define what controls should exist: data governance, model versioning, monitoring, human oversight, bias testing. 5. Testing approach — determine sample-based vs. population testing, automated vs. manual procedures, and technical depth required.

Do you know the specific legal requirements, or just the general concept of bias testing? This is a concrete, enforceable mandate with specific calculation requirements.

NYC LL 144 requires annual independent bias audits for automated employment decision tools (AEDTs). Key steps: 1. Define scope — identify the AEDT, its purpose (screening or scoring), and the employment decisions it informs. 2. Calculate impact ratios — compute selection rates or scoring rates for each race/ethnicity and sex category, including intersectional analysis. 3. Apply four-fifths rule — compare each group’s selection rate against the most-selected group. 4. Document and publish — results must be publicly posted on the employer’s website. Non-compliance: $500–$1,500 per violation per day.

Do you have the technical depth to evaluate AI systems, or do you rely on model developers to explain their own work? Auditors need independent assessment capability.

Interpretability is the degree to which a human can understand the model’s decision logic natively (e.g., linear regression, decision trees). Explainability uses post-hoc techniques (SHAP, LIME) to approximate why a complex model made a decision. For auditing, this matters because: 1. High-risk systems under EU AI Act require explainability for affected individuals. 2. Audit evidence — you need to independently verify model behavior, not rely on developer assertions. 3. Bias detection — SHAP values reveal which features drive decisions across demographic groups, exposing proxy discrimination.

Can you evaluate governance beyond just checking boxes? Do you understand what good AI governance looks like in practice, or just on paper?

Assess controls across five dimensions: 1. Data governance — data provenance, labeling quality, representativeness, consent, and bias testing at the data layer. 2. Model lifecycle — version control, testing protocols, approval workflows, rollback procedures, and model cards. 3. Monitoring — drift detection, performance degradation alerts, fairness metric tracking, and human oversight triggers. 4. Access controls — who can modify models, retrain systems, override decisions, and access sensitive data. 5. Documentation — ISO 42001 requires documented evidence of governance processes. Test for completeness, accuracy, and timeliness.

This tests technical hands-on capability. Do you know the tools, or do you just know they exist? Can you interpret the output and identify limitations?

Primary toolkits: IBM AI Fairness 360 (70+ fairness metrics, 11 mitigation algorithms) and Microsoft Fairlearn (scikit-learn integration, dashboard). Key validation steps: 1. Metric selection — choose appropriate fairness metrics for the context (demographic parity, equalized odds, calibration). No single metric captures all fairness dimensions. 2. Baseline comparison — test against established thresholds (four-fifths rule for employment, regulatory benchmarks). 3. Cross-validation — run multiple tools on the same data to identify tool-specific artifacts. 4. Limitation awareness — automated tools can’t detect all forms of bias; intersectional analysis and contextual judgment remain essential.