Blue Label Consulting
  • Our Services
  • Our Clients
  • Contact
  • Blog
  • First Class Analytics Framework

Blue Label Consulting blog

Model Inventory

28/1/2021

1 Comment

 
Model Inventory
 
Effective model governance takes careful planning.
 
At first, a business may be apprehensive about the resources and time required to create a central model repository. However, best practice and regulatory guidance show that developing a model inventory, including the risk assessment and personnel responsible for each model and supporting documentation, can result in better model quality and risk management.
 
With this in mind, a business must carefully consider these questions:

  1. How many models does the institution have?
  2. What governance has been completed and evidenced?
  3. What levels of risk do those models pose?
  4. Who is responsible for ensuring that the models align to business and regulatory requirements?
 
A well-defined Model Inventory will act as a framework for categorising and tracking models, model use, and model changes or updates.
 
This model repository may include information such as:
  • Key stakeholders – e.g. model owner, developer, and user
  • Model type – regulatory or non-regulatory
  • Model area – e.g. credit risk, asset risk, market risk, Accounting & IFRS etc
  • Model risk classification – based on complexity and material impact to the business
  • Risk and materiality considerations – such as frequency of monitoring and review
  • Model status – is the current governance in place and evidenced, date of last validation, next scheduled review and / or validation
 
The Model Inventory process has 3 key stages:

Figure 01 - Key Model Inventory Stages
Picture
Identification
First and foremost, define and identify all models. There are regulatory guidelines to help identify models, but a business must settle on its own accepted definition.
 
This will help create uniform standards across the business and provide clear guidelines for all stakeholders. Some processes will categorise clearly as models, others more as complex calculation exercises. The key variable is some form of repeatable calculation process that results in an output in the form of a forecast or estimate that drives a business decision.
 
Once a process meets these 3 criteria, then it can be defined as a model.
 
Model Risk Assessment
Each model should be risk rated based on the complexity and financial impact (materiality) of the calculation process. This risk rating will determine the rigour and prioritisation of all governance and risk management activities, including performance monitoring and periodic validation.
 
Elements that should be considered when determining the risk classification may include the models’:
  • Financial impact
  • Regulatory requirements
  • Data quality
  • Calculation complexity
  • Assumptions
  • Business decisions based on model output
  • Potential for reputational risk

It is important to have a consistent model risk assessment process that is revisited periodically, to ensure consistency, and monitor for changes to risk.
 
Governance – a business should develop and maintain detailed governance guidelines and technical governance work instructions to help establish, implement and maintain a robust model governance framework.
 
This documentation should be stored or linked to the model inventory, to make it easier to identify the model owners who are responsible for all governance related activities. The model parameters, controls and criteria for the use of the output of each model should also be identified, governed and evidenced.
 
In Summary
Models are now critical, if not central, to business. Model Governance therefore is key to maintaining control in an ever evolving regulatory landscape.
 
However, to get maximum value from this type of investment companies may need to rethink and fine-tune their strategies including changing culture and processes to deliver the control and governance needed by both Regulation and good business management.
 
As models have to adapt, so will model validation and governance. Therefore, make sure that you review your models and catalogue your model inventory. Then proceed to prioritising your models by importance and materiality.
 
The good news is that you don’t necessarily have to build all this from scratch. There are tools and technologies that you could leverage such as Azure ML or AWS SageMaker, which can be used to automate the end to end model lifecycle, including monitoring and capture of governance data i.e. model audit trail.
 
If you wish to discuss any of the points raised or are interested in finding out how we can help you review or implement your own MRM and Model Governance framework, please contact us here to book your free Discovery consultation.

Brendan Jayagopal

Founder and Managing Director. Blue Label Consulting

1 Comment

Model Monitoring and Validation

21/1/2021

0 Comments

 
Model Monitoring – Ongoing Performance and Validation
Picture
Maintaining models after deployment is one of the key stages of the overall model lifecycle. Once models are deployed, it may be tempting to view the modelling process as complete. In many ways, this is just the beginning.
 
Model monitoring is a key operational stage in the model life cycle, coming after model deployment.
​
After a model is deployed within the business, the next priority is to ensure the most effective use of the model. It is therefore essential to properly monitor the model for as long as it is in use to gain maximum benefit, to stay in control and to maintain an audit trail.
 
The purpose of model monitoring is to ensure that the model is maintaining a predetermined required level of performance but also entails aspects such as prediction errors, process errors and latency.
 
The main reason model monitoring is so important is Model Drift over time. In other words, the overall predictive power of a model gradually degrades over time. This can be caused by several factors, such as:
 
Figure 01 – Model Drift
Picture
There is also the issue of Concept Drift – whereby the expectations of what constitutes a correct prediction change over time even though the distribution of the input data has not changed. In other words, the relationship between model predictor(s) and the outcome being predicted has changed.
 
This Concept Drift could occur because of changes in:

  • The business strategy and objectives
  • The business view on macroeconomic conditions
  • Changes in consumer needs and/or expectations (e.g. of what constitutes a desirable product)
  • Other market factors such as macro-economic conditions affecting supply and demand
  • Regulations and legislation
 
As can be seen, there are a number of factors that could impact what is defined as a successful prediction from the model. Given this and changing regulatory considerations ongoing model monitoring is not only important for maintaining the validity of the models but also future business success.
 
There is high potential of significant negative impact from model degradation, so an effective model monitoring process will aim to identify and mitigate potential sources of model drift as soon as possible.
 
Figure 02 – Model Monitoring Framework
Picture

​Periodic Review 
A successful model governance framework will always include a full Periodic Review (PR) and validation of models, regardless of whether or not the model has undergone any change during its operational history.
 
The PR differs from the usual ongoing performance monitoring of the models, in that it is a more holistic total review of the model concept including:
  • Statistical approach – ensuring the overall modelling method is still the most suited and robust approach
  • Data management – processes to sanitize, process and store data meet optimal model requirements
  • Operational setup – facilities, equipment, staff, systems and processes are correctly and efficiently setup to maintain model pipelines and deliver output
  • Regulatory standards – Recent (or even upcoming) changes in legislation are (or will be) met in due course
 
Frequency of performing these PRs should be dependent on the complexity and criticality (material impact) of the model. The higher a model scores on these factors, the more often and more in-depth the PR.
 
In my next article, I will outline the importance of Model Inventory.
 
If you wish to discuss any of the points raised or are interested in finding out how we can help you review or implement your own MRM and Model Governance framework, please contact us here to book your free Discovery consultation.

Brendan Jayagopal

Founder and Managing Director. Blue Label Consulting

0 Comments

MRM framework and Model Tiers

14/1/2021

1 Comment

 
Picture
​Although many institutions in the UK are progressing towards attaining a certain standard in MRM practices, the rate of progress is uneven and unfortunately so are the ambition levels.
 
It is critical therefore to start with the correct understanding of the essential building blocks for an effective MRM framework as identified below:
 
  • Policies and Documents
  • Stakeholder Management
  • Model Inventory and Tiering
  • MRM Processes and Lifecycle Management
  • Model Monitoring and Reporting
 
Figure 01 – Buildings blocks of an effective MRM Framework
Picture
Model Tiering and Risk Classification
In order for any institution to have a successful MRM Framework, it will need to have a standardised model tiering procedure.
 
The assigned model Tier will dictate all model related processes:
  • Rigour of the pre-implementation validation
  • Frequency and rigour of ongoing performance monitoring
  • Frequency and rigour of periodic validations
  • Model approval process (for both new model and model iterations)
 
All models should have an assigned tier / risk classification, based on:
  • Model type (i.e. regulatory or non-regulatory)
  • Complexity
  • Financial Impact (from model error or misuse)
  • Reputational Impact (from model error or misuse)
 
Figure 02 Model Tiers                     ​
Picture
Note(s): 
*Pillar-1 Risks are Credit, Market & Operational Risk. Pillar-2 risks are residual risks after accounting for Pillar-1 risks- refer BCBS128 for details on Pillar-1 and Pillar-2 risks.
**Assess the model materiality, quantitatively and qualitatively. Quantitative Materiality of the models is calculated as the percentage impact on the institutions relevant financial portfolio. Qualitative Materiality is subjective and dependant on both regulatory and customer expectations.


The Tiering methodology should be segmented between a ‘Preliminary’ and ‘Final’ decision, and the process will ideally include both 1LOD and 2LOD stakeholders:
 
Figure 03 Model Tiering Methodology                      ​
Picture
​Once you have a clear understanding of the framework and have deployed models, it is essential to properly monitor the model for as long as it is in use to gain maximum benefit and stay in control. I will talk about this in my next article.

If you wish to discuss any of the points raised or are interested in finding out how we can help you review or implement your own MRM and Model Governance framework, please contact us here to book your free Discovery consultation.

​
Sources:
Risk.net MRM report (Risk_MRM_0719.pdf) – slide 9.
Deloitte (MRM_Risk_Ldn_01_MRM_Landscape_Sjoerd_K_Deloitte.pdf) – Model risk management building blocks and industry insights, slide 10.

Brendan Jayagopal

Founder and Managing Director. Blue Label Consulting

1 Comment

The need for MRM

7/1/2021

0 Comments

 
The need for Model Risk Management (MRM)
 
Organisations are making major investments today to harness their massive and rapidly growing quantities of information. They are putting existing data to work and building better models with greater predictive power by applying advanced tools and techniques like Machine Learning (ML) and Artificial Intelligence (AI).
 
Models are now critical, if not central, to business. Particularly in financial services which finds itself under ever increasing scrutiny. Model Governance therefore is key to maintaining control in an ever evolving regulatory landscape.
 
However, it’s clear that for organisations to get maximum value from this type of investment they need to rethink and fine-tune their strategies. In many instances, this will also mean changing their culture and processes to deliver the control and governance needed by both Regulation and good business management.
 
To help accomplish these goals, there needs to be a combination of advanced analytics along with a deep understanding of how to apply the correct Model Risk Management (MRM) framework that suits the business and the industry sector it’s in.
 
Where can we use Model Risk Management?
 
There are many processes across an organisation that fit the definition of a Model.
 
A widely accepted identification of a model is:
 
’Any process which consists of Input, Calculation Throughput and an Output that drives key decision making process for the business.’
 
Here are some specific areas where advanced analytics and MRM are commonly used:

  • Credit Risk – decision support systems, probability of default, impact of default
  • Asset or Market Risk – forecasting techniques to predict future asset values, maintenance spend
  • Operational Risk and Fraud Analytics – debt reporting, debt forecasting, recoveries and repossessions
  • Insurance analytics – probability and financial impact of insurance claims
  • Pricing Models – both from a Finance and Commercial perspective
  • Machine learning & Artificial Intelligence – can be applied to a wide range of Business Areas
 
Analytics can be used at each stage of a model’s lifecycle as outlined below.
 
Advanced Analytics and Model Risk Management
Models have long been part of the toolkit used by the financial community to assess, price, and manage the various risks they face.
 
Yet, they are also an emerging risk factor in themselves as model failure, or misuse, can seriously damage the finances, reputations, and even the solvency of firms.
 
In addition to individual model failure, many models use the output of other models as input (e.g., a portfolio model could use modelled probability of default as input). As a result, small errors in one model might be compounded or amplified when their erroneous results are fed into other models.
 
To mitigate this model risk, an organisation must develop and implement a suitable MRM and Model Governance framework. These governance control mechanisms are separated and outlined below to show each stage of the model lifecycle:

  • Model Creation (Initial Development)
  • Model Implementation
  • Model Ongoing Use & Periodic Validation
  • Model Governance and Audit

​Figure 1 – Key Model Lifecycle Stages
Picture
​Model Testing and Validation – a critical area for Advanced Analytics
Testing and validation are critical parts for both the development and ultimate acceptance of a model by both the business lines using the model and the external regulators that will have to review the models.

Model testing and validation teams should be independent from model development teams. 
 
Three Lines of Defence
Typically the Three Line of Defence (3LOD) technique is used to structure the teams, with Model Developers usually being 1LOD and Model Testers & Validations being 2LOD. Key stakeholders and model sponsors occupy the 3LOD slot.
​
Using advanced analytics skills, model testers and validators should be in a position to challenge model developers on a regular basis. In particular, the analysis should look at all aspects of a model including inputs, analytics, reporting, and performance. Where and when needed, they should have the power to be able to prevent a model that doesn’t pass their criteria from being utilised by the institution.

Testing and validation should focus on three key areas:
  1. The conceptual soundness of the model
  2. Identifying potential limitations in the model and in its range of applicability
  3. Evaluating model effectiveness, both through back testing and periodic reviews of model results
 
As already identified, sophisticated models are critically important to the success of financial institutions and fair treatment of their customers. But improperly managed and monitored, they can be dangerous to any business. Accurate control of the process and ongoing governance is key to ensuring regulatory compliance and continued business success.

In my next article, I will outline the correct overview of the essential building blocks for an effective MRM framework including the importance of having a standardised model tiering procedure.
 
If you wish to discuss any of the points raised or are interested in finding out how we can help you review or implement your own MRM and Model Governance framework, please contact us here to book your free Discovery consultation.
​
​
​Sources: 
McKinsey - advanced analytics and how it helps business functions
Moody’s – the need for MRM
PWC - MRM

Brendan Jayagopal

Founder and Managing Director. Blue Label Consulting

0 Comments

    brendan jayagopal

    Brendan launched Blue Label Consulting in 2011. With innovative use of Data through AI,  ML and other quantitative methods, he delivers robust analytics and actionable insights to solve business problems.
    ​

    Underpinning this is his core belief that the key to any successful data project is bringing the client team along the journey and embedding his knowledge and skills to ensure long term gains are achieved.

    Archives

    February 2021
    January 2021
    December 2020
    November 2020
    October 2020
    September 2020
    August 2020
    November 2018
    July 2018
    June 2018
    May 2018

    Categories

    All
    Analytics And Leasing
    Data Projects
    First Class Analytics
    Leadership
    Model Risk Management
    Model Risk Monitoring
    Risk
    Sentiment Analysis

    RSS Feed

Picture

Our Services

Our Clients

Contact

© COPYRIGHT 2017. ALL RIGHTS RESERVED.
Privacy Policy
  • Our Services
  • Our Clients
  • Contact
  • Blog
  • First Class Analytics Framework