Blue Label Consulting
  • Our Services
  • Our Clients
  • Contact
  • Blog
  • First Class Analytics Framework

Blue Label Consulting blog

Model Monitoring and Validation

21/1/2021

0 Comments

 
Model Monitoring – Ongoing Performance and Validation
Picture
Maintaining models after deployment is one of the key stages of the overall model lifecycle. Once models are deployed, it may be tempting to view the modelling process as complete. In many ways, this is just the beginning.
 
Model monitoring is a key operational stage in the model life cycle, coming after model deployment.
​
After a model is deployed within the business, the next priority is to ensure the most effective use of the model. It is therefore essential to properly monitor the model for as long as it is in use to gain maximum benefit, to stay in control and to maintain an audit trail.
 
The purpose of model monitoring is to ensure that the model is maintaining a predetermined required level of performance but also entails aspects such as prediction errors, process errors and latency.
 
The main reason model monitoring is so important is Model Drift over time. In other words, the overall predictive power of a model gradually degrades over time. This can be caused by several factors, such as:
 
Figure 01 – Model Drift
Picture
There is also the issue of Concept Drift – whereby the expectations of what constitutes a correct prediction change over time even though the distribution of the input data has not changed. In other words, the relationship between model predictor(s) and the outcome being predicted has changed.
 
This Concept Drift could occur because of changes in:

  • The business strategy and objectives
  • The business view on macroeconomic conditions
  • Changes in consumer needs and/or expectations (e.g. of what constitutes a desirable product)
  • Other market factors such as macro-economic conditions affecting supply and demand
  • Regulations and legislation
 
As can be seen, there are a number of factors that could impact what is defined as a successful prediction from the model. Given this and changing regulatory considerations ongoing model monitoring is not only important for maintaining the validity of the models but also future business success.
 
There is high potential of significant negative impact from model degradation, so an effective model monitoring process will aim to identify and mitigate potential sources of model drift as soon as possible.
 
Figure 02 – Model Monitoring Framework
Picture

​Periodic Review 
A successful model governance framework will always include a full Periodic Review (PR) and validation of models, regardless of whether or not the model has undergone any change during its operational history.
 
The PR differs from the usual ongoing performance monitoring of the models, in that it is a more holistic total review of the model concept including:
  • Statistical approach – ensuring the overall modelling method is still the most suited and robust approach
  • Data management – processes to sanitize, process and store data meet optimal model requirements
  • Operational setup – facilities, equipment, staff, systems and processes are correctly and efficiently setup to maintain model pipelines and deliver output
  • Regulatory standards – Recent (or even upcoming) changes in legislation are (or will be) met in due course
 
Frequency of performing these PRs should be dependent on the complexity and criticality (material impact) of the model. The higher a model scores on these factors, the more often and more in-depth the PR.
 
In my next article, I will outline the importance of Model Inventory.
 
If you wish to discuss any of the points raised or are interested in finding out how we can help you review or implement your own MRM and Model Governance framework, please contact us here to book your free Discovery consultation.

Brendan Jayagopal

Founder and Managing Director. Blue Label Consulting

0 Comments



Leave a Reply.

    brendan jayagopal

    Brendan launched Blue Label Consulting in 2011. With innovative use of Data through AI,  ML and other quantitative methods, he delivers robust analytics and actionable insights to solve business problems.
    ​

    Underpinning this is his core belief that the key to any successful data project is bringing the client team along the journey and embedding his knowledge and skills to ensure long term gains are achieved.

    Archives

    February 2021
    January 2021
    December 2020
    November 2020
    October 2020
    September 2020
    August 2020
    November 2018
    July 2018
    June 2018
    May 2018

    Categories

    All
    Analytics And Leasing
    Data Projects
    First Class Analytics
    Leadership
    Model Risk Management
    Model Risk Monitoring
    Risk
    Sentiment Analysis

    RSS Feed

Picture

Our Services

Our Clients

Contact

© COPYRIGHT 2017. ALL RIGHTS RESERVED.
Privacy Policy
  • Our Services
  • Our Clients
  • Contact
  • Blog
  • First Class Analytics Framework