CRISIL analytics head speaks about challenges of insurance model risk management

What we're seeing in insurance contrasts with what we see in the work we do in other industries, specifically banking. In the U.S., banks are under SR 11-7 guidelines for model risk management. There's a lot of maturity in banks' processes for model risk management. There's a 'three lines of defense model,' which means that in the first line of defense, you have people who document the model, the ones that build the model, document it well and test it well. They hand it over to the second line of defense and that performs a number of activities. It provides challenge and review to what's been done in the first line. That's an independent line of business, which reports separate from the first line. The third line is basically an audit function, which makes sure that the first two lines operate very, very well. 

Now, in contrast, when we work with insurance firms there are insurance firms that have established lines of defense, which means that they have those controls in place. They are quite sophisticated … but it's not uniform. There is a lot of disparity. Some of them do not yet have a very robust second line, which means that whatever is done by the developers and the documentation is not effectively challenged by the second line function. In a number of cases, the third line or the internal logic does not have the capabilities to assess these models or provide guidance. Certain insurance firms are very, very sophisticated and have adopted the best practices, but a number of firms are still lagging or are on their journey to adopt that.

Model risk is becoming more and more important because AI and machine learning models – their usage is going up. There are risks that get associated when you use these sorts of models. There is concern amongst the insurance regulators, as well as senior people in insurance as well.

One component of model risk is monitoring. Under the three lines of defense model, how do I challenge what someone has built? The second line is monitoring on an ongoing basis. One of the main things is just to figure out what's your model inventory? How many models do you actually have in your inventory? Is it a hundred? Is it a thousand? Is it 4,000? What's the range of that? Insurance firms have models with thousands of ways to track that inventory. That's where technology plays a big role. 

We have built something called model infinity. It provides a full platform to look into any point of time, how many models you have, whether they've been checked, validated or not, and how's the ongoing monitoring of these models. That's the kind of technology that you need. It's quite important for any insurance firm to have something like that to mitigate model risk. Our technology solution gives a lot of comfort to senior management that their model risk is under control and there is ongoing monitoring, effective review and challenge of the models. Any regulator can come in and there's a full audit trail available. 

Machine learning models continually evolve. They change over time. As more data comes in, the characteristics of the model undergo changes. So you need more real-time monitoring. There are various players providing that technology. A lot of this is cloud based. You need to implement that and get it going. Those are the two things that are useful from a technology perspective for model risk.

For example, auto insurance underwriting models. We are seeing quite a lot of innovation that's happening there. Machine learning models creep into auto insurance claims processing. You take a picture of the damage to your car and upload that to your insurer. The insurer has a machine learning model in the background that looks at the claim and assesses the severity of the damage based on comparing your photo to others. There's direct customer involvement. New challenges are coming up related to ethics, fairness and bias. These are challenges that impact models of this sort where there's a huge consumer interface that's required.

When you're building models that have consumer impact, you need to keep in mind these considerations. When selecting variables that could impact decision making, steer clear from using variables that could be gender-biased. When making models, that's something the insurance industry is quite concerned about – ensuring that management of model risk takes care of that at the design stage and that someone has reviewed it and challenged it.