Can you articulate the foundational regulatory framework? Do you understand conceptual soundness, outcome analysis, and ongoing monitoring as distinct validation activities?

1. Conceptual soundness — evaluate the model’s theoretical framework, methodology, mathematical assumptions, and limitations. Assess whether the design is appropriate for its intended purpose. 2. Outcome analysis — compare model outputs against actual observed outcomes (backtesting). Statistical tests: Kolmogorov–Smirnov, binomial tests, traffic light approach for VaR models. 3. Ongoing monitoring — continuous performance tracking, trigger-based reviews, periodic revalidation schedule. SR 26-2 now supersedes SR 11-7 but explicitly excludes Gen AI, creating a dual governance requirement for organizations deploying AI models.

This tests your core technical capability. Can you independently build an alternative model and use it to benchmark the model under review?

Start by understanding the model’s intended use case and regulatory context (e.g., fair lending, ECOA/Reg B). Build a challenger using a different methodology — if the model uses gradient boosting, build a logistic regression or simpler tree-based alternative. Use the same training data and feature set. Compare performance: AUC-ROC, Gini coefficient, KS statistic. Critically, test fairness metrics: disparate impact ratio across protected classes. Document where the challenger diverges from the production model and assess whether the additional complexity is justified by measurable performance gains.

Can you apply fairness metrics methodically? Do you understand the tradeoffs between different fairness definitions and the regulatory implications?

Define protected attributes (race, gender, age, disability) per EEOC guidelines. Calculate disparate impact ratio — selection rate of protected group divided by selection rate of favored group; 80% (4/5ths) rule as screening threshold. Apply multiple fairness metrics: demographic parity (equal selection rates), equalized odds (equal TPR and FPR across groups), calibration (same score = same probability regardless of group). Use SHAP to identify which features drive disparate outcomes. Note: perfect fairness across all definitions is mathematically impossible (Chouldechova 2017) — document the chosen definition and justify it for the use case.

Do you understand what regulators look for? Can you ensure your validation documentation, model inventory, and governance processes withstand scrutiny?

Regulators evaluate four areas: 1. Model inventory completeness — every model cataloged with risk tier, validation date, owner, and next review date. 2. Validation report quality — findings documented with evidence, risk ratings (high/medium/low), remediation timelines, and management response. 3. Independence — validation function organizationally separate from model development (second line of defense). 4. Issue tracking — open findings tracked to closure, overdue items escalated. Ensure your model risk policy is current with SR 26-2 requirements, and that your AI/ML models have a separate governance layer since SR 26-2 explicitly excludes Gen AI from scope.

Can you monitor models after deployment? This is the third pillar of SR 11-7 — ongoing monitoring — and it’s where most organizations are weakest.

Monitor three types of drift: 1. Data drift — input feature distributions shift (Population Stability Index, Jensen–Shannon divergence). 2. Concept drift — relationship between features and target changes (track prediction accuracy over time windows, ADWIN algorithm). 3. Model performance drift — accuracy/AUC/precision degrade below threshold (define trigger levels, e.g., AUC drops >5% from baseline). Build automated monitoring pipelines in Python using Evidently AI or custom scripts. Establish trigger-based revalidation thresholds: performance degradation, material data changes, regulatory changes, or significant model modifications all trigger full revalidation.