What is new in model risk management?

One of the most exciting areas where we can see technological developments in risk management are our models. It’s time to improve risk managers’ ability to respond to changes as quick as possible. In this article, Jing Zou, Managing Director, Enterprise Model Risk Management at the Royal Bank of Canada, explores the latest trends in model risk management in order to become as efficient as possible.

Model risk is “the potential for adverse consequences from decisions based on incorrect or misused model outputs and reports”^[1]. The uncontrolled model risk can potentially cause poor business decisions, trigger financial losses, or even damage a firm’s reputation. In the past ten years, the financial industry has seen the prosperity of model risk management, with a quick expansion in model inventory as well as personnel in the model risk area, mostly driven by the enhanced regulatory scrutiny and the realisation of the importance of model risk.

So what is new in model risk management today? Some of the hottest topics are efficiency improvement, model risk reporting, and artificial intelligence/ machine learning (AI/ML) models.

Model risk management efficiency

In recent surveys ^[2,3], model risk heads from various banks “identified the incomplete or poor quality of model submissions as the largest barrier for their validation timelines”^[2]. To overcome this barrier, they agreed that the most effective method is to educate the first line of model risk, namely, model developers and model business owners, in terms of model risk policy, roles, and responsibilities, and model documentation/testing expectation.

This education can be useful in the following aspects. First, once the first line better understands the importance of model risk, they are more likely to execute the corresponding requirement, such as seeking for model approval before a model update and self-identifying models in use but unvalidated. Moreover, it helps enhance the model documentation and testing quality. Sometimes, we observe a lengthy process of going back and forth between modelers and validators because of the lack of clarity/details and insufficient testing in a model submission. This process can be expedited by providing more guidance on the documentation and testing, and sometimes an excellent white paper example (of a similar model) as a reference.

Other approaches to enhance the efficiency include prioritising the validation work commensurate with its model tiering/ risk rating, building a code library for standard tests, automating some part of a validation report with model-related information. That said, one still needs to maintain the effective challenge, such as actively designing validation test scope (not just limited to the existing automation library) and reviewing the testing results.

Model risk reporting

A financial institution may have hundreds or even thousands of models. How can we aggregate, summarise, and report the model risk to the board of directors and senior executives? There are three potential areas.

• Compliance reporting
  There are a few compliance metrics, such as the number of models in use but unvalidated, the overdue validation/ annual assessment submission, and the number of overdue issues. One can even show a trend in the past year to see if the model compliance has improved.
• Model performance monitoring
  The model performance monitoring is critical to ensure a model performs reasonably well as the market environment changes. Typically, there are performance metrics and thresholds associated with a model. A breach in model performance threshold should be escalated and reviewed by model stakeholders, and if material, be reported to the board.
• Model risk buffer
  Another challenging component of model risk reporting is to quantify and aggregate model risk. One potential approach is the bottom-up model risk buffer methodology. For a model, its distribution of model uncertainty (not just one single number) can be determined by backtesting, benchmarking, and/ or validation findings. Then these uncertainties can be aggregated to calculate model risk buffers at the firm level, which is reported to the board.

Artificial intelligence/ machine learning (AI/ML) models

The emergence of AI/ML models provides challenges and opportunities to model risk management. Although the current model risk management framework is still applicable, it needs to be enhanced to mitigate specific risks associated to AI/ML models, including model interpretability, model bias, and recalibration ^[4,5,6,7].

For example, model results are generated by these black-box approaches and are often difficult to interpret, especially for neural network method. On the other hand, the business/ model validators need to understand the model results to be comfortable enough to use/ approve the models. The good news is that more and more model diagnostic and distillation techniques are being developed to help interpret the model results. Moreover, the AI/ML model results need to be reviewed through benchmarking to other models or expert opinions to make sure there is neither algorithm bias nor bias against groups or classes of people. Finally, when more data are available, the AI/ML model can be recalibrated frequently and dynamically to capture the latest trend. To set up a proper control, the model risk management team needs to enhance its current policy, approve the recalibration methodology, and monitor model performance with some thresholds.

The usage of AI/ML models also provides opportunities for model risk management. They can be used for building benchmarking models, conducting feature selection, monitoring performance, and challenging theoretical framework.

Beside the three mentioned above, there are several other exciting topics in model risk management that are becoming more and more relevant today, such as model vs. non-model, benchmarking, and model inventory. If you want to know more about these topics, I’d be happy to welcome you to my session during RiskMinds and QuantMinds Americas.

References

1. SR11-7 Supervisory Guidance on Model Risk Management, Board of Governors of the Federal Reserve System, Office of the Comptroller of the Currency, 2011.
2. The Evolution of Model Risk Management, I. Crespo, P. Kumar, P. Noteboom, M. Taymans, McKinsey & Company, 2017.
3. Model Risk Management Round Table, organized by Ernst & Young, 2019.
4. Machine Learning and Model Risk Course, A. Sudjianto, J. Chen, B. Hientzsch, Risk USA, 2018.
5. AI/ML Model Risk Management: Challengers and Opportunities, H. Sharma, Risk.Net Model Risk Management Training, 2018.
6. Derisking Machine Learning and Artificial Intelligence, B. Babel, K. Buehler, A. Pivonka, B. Richardson, D. Waldron, McKinsey & Company, 2019
7. Validation of Machine Learning Models: Challenges and Alternatives, S. Dil, S. Baral, Protviti Compliance Insights, 2019.