|
Model Inventory Effective model governance takes careful planning. At first, a business may be apprehensive about the resources and time required to create a central model repository. However, best practice and regulatory guidance show that developing a model inventory, including the risk assessment and personnel responsible for each model and supporting documentation, can result in better model quality and risk management. With this in mind, a business must carefully consider these questions:
A well-defined Model Inventory will act as a framework for categorising and tracking models, model use, and model changes or updates. This model repository may include information such as:
The Model Inventory process has 3 key stages: Figure 01 - Key Model Inventory Stages  Identification First and foremost, define and identify all models. There are regulatory guidelines to help identify models, but a business must settle on its own accepted definition. This will help create uniform standards across the business and provide clear guidelines for all stakeholders. Some processes will categorise clearly as models, others more as complex calculation exercises. The key variable is some form of repeatable calculation process that results in an output in the form of a forecast or estimate that drives a business decision. Once a process meets these 3 criteria, then it can be defined as a model. Model Risk Assessment Each model should be risk rated based on the complexity and financial impact (materiality) of the calculation process. This risk rating will determine the rigour and prioritisation of all governance and risk management activities, including performance monitoring and periodic validation. Elements that should be considered when determining the risk classification may include the models’:
It is important to have a consistent model risk assessment process that is revisited periodically, to ensure consistency, and monitor for changes to risk. Governance – a business should develop and maintain detailed governance guidelines and technical governance work instructions to help establish, implement and maintain a robust model governance framework. This documentation should be stored or linked to the model inventory, to make it easier to identify the model owners who are responsible for all governance related activities. The model parameters, controls and criteria for the use of the output of each model should also be identified, governed and evidenced. In Summary Models are now critical, if not central, to business. Model Governance therefore is key to maintaining control in an ever evolving regulatory landscape. However, to get maximum value from this type of investment companies may need to rethink and fine-tune their strategies including changing culture and processes to deliver the control and governance needed by both Regulation and good business management. As models have to adapt, so will model validation and governance. Therefore, make sure that you review your models and catalogue your model inventory. Then proceed to prioritising your models by importance and materiality. The good news is that you don’t necessarily have to build all this from scratch. There are tools and technologies that you could leverage such as Azure ML or AWS SageMaker, which can be used to automate the end to end model lifecycle, including monitoring and capture of governance data i.e. model audit trail. If you wish to discuss any of the points raised or are interested in finding out how we can help you review or implement your own MRM and Model Governance framework, please contact us here to book your free Discovery consultation. Brendan JayagopalFounder and Managing Director. Blue Label Consulting
1 Comment
Model Monitoring – Ongoing Performance and Validation  Maintaining models after deployment is one of the key stages of the overall model lifecycle. Once models are deployed, it may be tempting to view the modelling process as complete. In many ways, this is just the beginning. Model monitoring is a key operational stage in the model life cycle, coming after model deployment. After a model is deployed within the business, the next priority is to ensure the most effective use of the model. It is therefore essential to properly monitor the model for as long as it is in use to gain maximum benefit, to stay in control and to maintain an audit trail. The purpose of model monitoring is to ensure that the model is maintaining a predetermined required level of performance but also entails aspects such as prediction errors, process errors and latency. The main reason model monitoring is so important is Model Drift over time. In other words, the overall predictive power of a model gradually degrades over time. This can be caused by several factors, such as: Figure 01 – Model Drift  There is also the issue of Concept Drift – whereby the expectations of what constitutes a correct prediction change over time even though the distribution of the input data has not changed. In other words, the relationship between model predictor(s) and the outcome being predicted has changed. This Concept Drift could occur because of changes in:
As can be seen, there are a number of factors that could impact what is defined as a successful prediction from the model. Given this and changing regulatory considerations ongoing model monitoring is not only important for maintaining the validity of the models but also future business success. There is high potential of significant negative impact from model degradation, so an effective model monitoring process will aim to identify and mitigate potential sources of model drift as soon as possible. Figure 02 – Model Monitoring Framework  Periodic Review A successful model governance framework will always include a full Periodic Review (PR) and validation of models, regardless of whether or not the model has undergone any change during its operational history. The PR differs from the usual ongoing performance monitoring of the models, in that it is a more holistic total review of the model concept including:
Frequency of performing these PRs should be dependent on the complexity and criticality (material impact) of the model. The higher a model scores on these factors, the more often and more in-depth the PR. In my next article, I will outline the importance of Model Inventory. If you wish to discuss any of the points raised or are interested in finding out how we can help you review or implement your own MRM and Model Governance framework, please contact us here to book your free Discovery consultation. Brendan JayagopalFounder and Managing Director. Blue Label Consulting  Although many institutions in the UK are progressing towards attaining a certain standard in MRM practices, the rate of progress is uneven and unfortunately so are the ambition levels. It is critical therefore to start with the correct understanding of the essential building blocks for an effective MRM framework as identified below:
Figure 01 – Buildings blocks of an effective MRM Framework  Model Tiering and Risk Classification In order for any institution to have a successful MRM Framework, it will need to have a standardised model tiering procedure. The assigned model Tier will dictate all model related processes:
All models should have an assigned tier / risk classification, based on:
Figure 02 Model Tiers  Note(s): *Pillar-1 Risks are Credit, Market & Operational Risk. Pillar-2 risks are residual risks after accounting for Pillar-1 risks- refer BCBS128 for details on Pillar-1 and Pillar-2 risks. **Assess the model materiality, quantitatively and qualitatively. Quantitative Materiality of the models is calculated as the percentage impact on the institutions relevant financial portfolio. Qualitative Materiality is subjective and dependant on both regulatory and customer expectations. The Tiering methodology should be segmented between a ‘Preliminary’ and ‘Final’ decision, and the process will ideally include both 1LOD and 2LOD stakeholders: Figure 03 Model Tiering Methodology  Once you have a clear understanding of the framework and have deployed models, it is essential to properly monitor the model for as long as it is in use to gain maximum benefit and stay in control. I will talk about this in my next article. If you wish to discuss any of the points raised or are interested in finding out how we can help you review or implement your own MRM and Model Governance framework, please contact us here to book your free Discovery consultation. Sources: Risk.net MRM report (Risk_MRM_0719.pdf) – slide 9. Deloitte (MRM_Risk_Ldn_01_MRM_Landscape_Sjoerd_K_Deloitte.pdf) – Model risk management building blocks and industry insights, slide 10. Brendan JayagopalFounder and Managing Director. Blue Label Consulting The need for Model Risk Management (MRM) Organisations are making major investments today to harness their massive and rapidly growing quantities of information. They are putting existing data to work and building better models with greater predictive power by applying advanced tools and techniques like Machine Learning (ML) and Artificial Intelligence (AI). Models are now critical, if not central, to business. Particularly in financial services which finds itself under ever increasing scrutiny. Model Governance therefore is key to maintaining control in an ever evolving regulatory landscape. However, it’s clear that for organisations to get maximum value from this type of investment they need to rethink and fine-tune their strategies. In many instances, this will also mean changing their culture and processes to deliver the control and governance needed by both Regulation and good business management. To help accomplish these goals, there needs to be a combination of advanced analytics along with a deep understanding of how to apply the correct Model Risk Management (MRM) framework that suits the business and the industry sector it’s in. Where can we use Model Risk Management? There are many processes across an organisation that fit the definition of a Model. A widely accepted identification of a model is: ’Any process which consists of Input, Calculation Throughput and an Output that drives key decision making process for the business.’ Here are some specific areas where advanced analytics and MRM are commonly used:
Analytics can be used at each stage of a model’s lifecycle as outlined below. Advanced Analytics and Model Risk Management Models have long been part of the toolkit used by the financial community to assess, price, and manage the various risks they face. Yet, they are also an emerging risk factor in themselves as model failure, or misuse, can seriously damage the finances, reputations, and even the solvency of firms. In addition to individual model failure, many models use the output of other models as input (e.g., a portfolio model could use modelled probability of default as input). As a result, small errors in one model might be compounded or amplified when their erroneous results are fed into other models. To mitigate this model risk, an organisation must develop and implement a suitable MRM and Model Governance framework. These governance control mechanisms are separated and outlined below to show each stage of the model lifecycle:
Figure 1 – Key Model Lifecycle Stages  Model Testing and Validation – a critical area for Advanced Analytics Testing and validation are critical parts for both the development and ultimate acceptance of a model by both the business lines using the model and the external regulators that will have to review the models. Model testing and validation teams should be independent from model development teams. Three Lines of Defence Typically the Three Line of Defence (3LOD) technique is used to structure the teams, with Model Developers usually being 1LOD and Model Testers & Validations being 2LOD. Key stakeholders and model sponsors occupy the 3LOD slot. Using advanced analytics skills, model testers and validators should be in a position to challenge model developers on a regular basis. In particular, the analysis should look at all aspects of a model including inputs, analytics, reporting, and performance. Where and when needed, they should have the power to be able to prevent a model that doesn’t pass their criteria from being utilised by the institution. Testing and validation should focus on three key areas:
As already identified, sophisticated models are critically important to the success of financial institutions and fair treatment of their customers. But improperly managed and monitored, they can be dangerous to any business. Accurate control of the process and ongoing governance is key to ensuring regulatory compliance and continued business success. In my next article, I will outline the correct overview of the essential building blocks for an effective MRM framework including the importance of having a standardised model tiering procedure. If you wish to discuss any of the points raised or are interested in finding out how we can help you review or implement your own MRM and Model Governance framework, please contact us here to book your free Discovery consultation. Sources: McKinsey - advanced analytics and how it helps business functions Moody’s – the need for MRM PWC - MRM Brendan JayagopalFounder and Managing Director. Blue Label Consulting |
brendan jayagopalBrendan launched Blue Label Consulting in 2011. With innovative use of Data through AI, ML and other quantitative methods, he delivers robust analytics and actionable insights to solve business problems. Archives
February 2021
Categories
All
|
RSS Feed
|
Our Services |
Our Clients |
|