Transcription:
Tracy Waller: (
Good afternoon everybody and thank you for joining us for the applied AA AI track. This is Scott and Tracy, and we're very excited to be hosting you today. We're going to start by talking about applied AI and underwriting, and we were really excited to have this topic today because we are at a convergence as respects AI and digital transformation in commercial lines. What we've seen is that something real is beginning to happen. The availability of data right now is absolutely astounding. It's everywhere. It's not all created equal, but it is very much available. We've seen high technology like AI and machine learning. Finally evolve to the point where it's benefits can be practically realized in an insurance context. And what I think is probably the biggest catalyst of them all is that there is of late an unprecedented willingness in the industry, particularly in commercial lines to take these big ideas and these big advancements out of the R and D labs and putting them out into the real world.
Tracy Waller: (
So it's a very exciting time for data analytics and AI and digital transformation, all the good buzzwords. When that happens, when you have a convergence with all of this advantage and opportunity, there are a couple of different paths that you can take in order to realize practical benefits. So I'd like to take you on a little bit of a journey, if you will picture it. LA 1994, Sega has just overtaken Nintendo as the top video game company in the world. And they're about to launch their newest system, the Sega Saturn. Now Sega has arrival that they're not too worried about in the industry and Sony, who is also about to launch their newest system, the PlayStation. So Sega knows exactly what they have to do. They're looking back and do taking all the lessons that they've learned that made them what they are today.
Tracy Waller: (
They're pushing a massive marketing campaign on their strongest segment, boys, eight to 14, they are bulking up their hardware, so it can have fast gameplay. It's a really powerful and heavy machine. They are also launching with a suite of games that have been their bread and butter from day one, arcade games, very simple. This is a recipe for success for Sega, and they're really excited to get started. Now, Sony took a completely different path, and I think we all know how the story ends. Sega made a nearly fatal mistake in their launch in that they only looked backward. Sony looked forward. They had the foresight to understand that the greatest growth market for video games, wasn't young boys. It was actually college age adults, particularly women. They understood that the pace of software evolution was just increasing and they made their machine light as air and software focused so that they could do a lot of system updates on the back end.
Tracy Waller: (
Lastly, they saw that the types of games that were getting increasing amounts of attention in the market were actually very graphics and design heavy. So they launched with a suite of role playing games, which were a hit. And here we are today. Raise your hand if you have, or had a Sega Saturn, we had somebody who had one and it's almost impossible of the last couple of years to get your hands on, a PS-5. So some people might think of this as a cautionary tale, but I prefer to think of it as something a little bit more inspiring because we, as the people in this room today have the influence and the insight and the opportunity to be more like Sony. We can take this spot in the market where we are and listen to the underwriters, listen to the agents and the insureds and hear what they're saying about what they want to see when it comes to their experience and their data and analytics.
Tracy Waller: (
But because this is insurance, whenever you talk to anybody or a couple of people in the industry, we're all at different points in the realm of digital transformation. There are some who are really just beginning to dip a toe into things like prefill, even just starting with PPC code, they are proving out the concept step by step, taking the learnings, moving it forward, but they're, there, there are others who have automated entire geographies or full segments of their books. And there are still more who are actually true digital shops today. And that's great, but then there are some others who are on a longer term roadmap and they think that, you know, everything will be fine if they just have a five year plan to get to some basic prefill. From my perspective, I think that's a little bit risky. I think within the next couple of years, particularly in the small commercial market, the race to zero applications will have been one.
Tracy Waller: (
We're going to be looking at carriers and MGAs who have embraced a digital transformation and maybe that's it as we start looking for and it sounds a little bit hyperbolic, but it truly is a reflection of what we're seeing in the industry today, because what AI is doing for us is creating a foundation to be able to see the speed and scale that are gonna bring us forward in the market. And if you have a little bit of foresight and can look ahead and listen to the market, there's a lot that's actually possible even today using applied AI. So what you're seeing on the screen here are some actual real world examples of models being used in the market today that are built on a large set of structured and unstructured data, text and images. But the goal is not to have a lot of models, right?
Tracy Waller: (
It's to be able to have models that are practically useful in insurance and have results like these, where you are automatically detecting the exposures that drive premium, the exposures that drive claims and the exposures that drive the coverage needs for your insureds. Because when you have a more sophisticated AI and digital transformation strategy put together, you could take all of that effort out no more 10 minutes, figuring out whether the insured is a condo complex that has a pool or a community grill, or, whether the contractor is using cranes or if they're really, truly just a landscaper. Like it looks like one of the biggest value props that AI can give in an underwriting perspective is understanding the difference in the level of risks that a given insured presents. So there's a huge difference between a family restaurant and a bar, and a big difference between a car wash that uses brushes and creates a potential third party, property, damage exposure, and one that doesn't.
Tracy Waller: (
But when we're putting this all together and we wanna run forward and get all these results in place that we can come out in the market and say, "Hey, we've got this fantastic digital solution." There are a couple of pitfalls that you may encounter. So bear with me, if you will, can anybody answer what these three things have in common? We have a zombie bank, a network password, and a celebrity home, any takers I will spare you. All of these represent examples of a perverse incentive. Oh, it's known colloquially as the Cobra effect. So a perverse incentive is created when somebody like all of us here sets out with one goal and, maybe it's a really, an even noble goal. And in order to get to that goal, they encourage specific sets or discouraged specific sets of behaviors, and most likely unintentionally end up with what is the exact opposite of the result that they were going for.
Tracy Waller: (
So in the case of the zombie banks, the term was coined in 1987, after the savings and loan crisis to describe banks that were failing, but were being held up artificially through other means funding. The point of course was to stave off, financial collapse and panic and a run on the healthy banks in Japan. This went a little bit further. The Japanese government was concerned that if certain banks or long established corporations were allowed to fail, that the economy would tank because, the populace would lose faith in the government and in these corporations. So they continued to loan money over and over again with no plans in place to banks and corporations and at one point it was even where they were lending money so that these banks and corporations could just pay off the interest on other loans. Subsequently what happened was that the capital that was available for healthy and growing businesses dried up to nothing.
Tracy Waller: (
In trying to stave off slowing economic growth, they actually flattened their economic growth for about a decade. This may seem familiar to people in the room with network passwords in the name of cyber security. Most companies and websites are getting in increasingly stringent about what they're requiring for passwords and how often you change them and how complex they have to be. And many times the end result is that people just end up writing them on a post-it note and sticking them under the keyboard, which is not a best practice for cyber security. Lastly, we've got the celebrity home. A few years ago, a celebrity bought a compound and a small gossip website caught wind of it, took pictures and posted them on their site where their readership of a few thousand people were able to see it. Celebrity was not happy thought this was an invasion of their privacy, and they didn't want random people being able to see their home.
Tracy Waller: (
So they sued. The lawsuit got national attention and the major publications found these pictures and posted them on their own websites where many millions of people were now able to see pictures of the home. So exact opposite of what was intended. So why do I tell you all this? these are fun anecdotes, but the reason that I tell you them is because when it comes to implementing a more sophisticated and fast to market AI and digitization strategy, we are all potentially at risk of these same types of unintended consequences and perverse incentives. That risk can generally come to us in the form of data that is deemed good enough. It's a trap, and I say, it's a trap because sometimes good enough really is good enough, but it is so rare, you know, if you have an agent portal that isn't the most beautiful, but it works and people use it and you're getting good feedback.
Tracy Waller: (
You can pretty up later. Good enough, but I've heard some definitions in the market where good enough essentially means bare minimum or exists. Maybe not good enough if your agent portal is out there and it's technically live, but it's glitchy and it crashes and people are complaining not good enough. You've actually probably annoyed your agent force more so than you've helped them, but we're all susceptible to the allure of good enough data. And it makes sense. You know, you're trying to get them maximum of ROI on something that can mean incredibly complex initiative and lower quality data can certainly be cheaper than better quality data. You want to understand where you start to see that point of diminishing return on your investment. And really you just wanna get out there with some sort of solid MVP where you can say we're playing in the digital realm.
Tracy Waller: (
Again it all makes sense. We have the best of intentions, but the problem with the good enough, when it comes to a sophisticated AI digital transformation strategy, is that it rarely ever is bad. Data will out. And it generally is caused by what we describe as noise in your data. And when we say noise, we mean data that is inaccurate. That is incomplete, that is out of date, or that is simply irrelevant from the perspective of, an insurance context, noise will out. And that's true, whether it's from structured or unstructured data, and it's generally the result of an indiscriminate sourcing. You know when your goal is, give me as much data as possible. So I can build these models rather than give me the most of the most relevant data possible. So I can build these models because when you're feeding your AI algorithms data, that is just good enough.
Tracy Waller: (
You could end up with the exact opposite of what you intended. You may start to see things like misclassification because the data was wrong. And it didn't pick up that this landscaper is actually doing a lot of roofing work. You might start to see things like claims where you didn't expect them or where you were under reserved because the AI was out of date, the data feeding the AI was out of date, and it didn't see that this business had expanded and is now offering tanning beds and drive through Botox. But the worst point of all that can happen is that you haven't insured who submits for a claim that is denied because they're doing an exposure that you generally exclude. Maybe they use drones, but you would've covered it. Had you known that the exposure existed and you didn't have the opportunity because the data that was being fed into your system was only really good enough.
Tracy Waller: (
So these are some anecdotal examples, but we do know it happens in the real world. The US spends 3.1 trillion a year, addressing bad data. Most of that is from knowledge workers, like all of us finding and fixing errors. We've done studies where we understand that in the property space, you're looking at over four years, four and a half billion dollars in premium leakage. And that's just on construction and PPC code, similar story for auto radius for risks that say, or seem local, but they're really not. You can see a leakage of nearly six and a half billion dollars over four years and in the Bot market, we know that over half of the classifications on policies are wrong at the two digit level. And that translates to about 22 billion over four years in premium leakage, but all is not lost of course, because if you put together a sound AI strategy, working with solid data and understand that you need to look forward to implement it, you can have really solid results.
Tracy Waller: (
Again, this is a real profile, that was created, it's anonymized so that we can share it here, but putting together structured traditional pre-filled data with unstructured text and image analytics that are powered by high quality and vetted data can net you results on the AI front that are over 90% accurate, and that are addressing the exposures that you need to understand in order to truly have a solution that, supports a digital transformation and that your underwriters will trust. So I'm actually gonna hand it over here to Dr. Z. Who's gonna take us through some of the best practices to use for applied AI. Scott.
Scott Zrebiec: (
Thank you, Tracy. It's a very exciting time to be an insurance data scientist. We're seeing techniques that were developed in the nineties, really get traction in the industry and bring new techniques to bear and solve new problems. Not only that we're seeing these new data elements that are useful and pertinent to solving insurance underwriting questions. So just might be the case that AI and ML, is the medicine that you need to solve your business problems, but it's also like medicine in another way. You shouldn't snack on it just coz you're bored one day. It's a tool with very profound and revolutionary impact, but it's just an expansion of the toolbox, not a panacea. Now I'm gonna be talking about AI and ML a lot. So I thought I should probably define my terms. And there's one person who defines AI and ML better than others.
Scott Zrebiec: (
That's this tweet by Mat Velloso. If it's written in PowerPoint, it's AI, if it's written in Python it's ML, which is very problematic, coz I had to fight with autocorrect so much on this presentation. I actually have a personal and practical definition. ML is any technique where the user only weekly specifies the solution. They're choosing some parameters, but they're really, letting the machine find what's right, basing it on empirical observations and data. AI to me, refines refers to any technique that's designed to mimic human intelligence and in for practical, applications. This is neural networks and deep learning.
Scott Zrebiec: (
I wanted to start by talking about machine learning and really go back into the sixties and go to my second favorite technique that was developed in the sixties, which would be trees, decision trees. My first favorite was actually fuzzy logic, which was invented in a way of thinking that was invented in Berkeley in the sixties. Trees have a very natural place in insurance underwriting, especially commercial insurance underwriting. Commercial insurance is fundamentally heterogeneous. When an underwriter looks at a restaurant, they're not gonna be asking the exact same questions as they look. If they're looking at underwriting, a large construction company, it's the questions that you would ask for a restaurant are irrelevant and that's where this kind of pertinent and nuanced data is very important coz it answers these questions and this fits into a tree. The key part of a tree is that with each question that you ask, once you get the answer to that, you ask a different question that's pertinent and this brings up the bias variance trade off, which is one of two practical techniques that occur and have to be dealt with as a data scientist on every single problem.
Scott Zrebiec: (
Every single model we built and the bias variance trade off is illustrated in trees. If you have a very deep tree, you can capture the nuances in the data, but you have a lot of variance in the prediction. Your model might not generalize very well. Flip side of that is if you ask too few questions, if you leave a lot of signal on the table, you only ask one question, is it a restaurant or is it a construction company? That's not how you wanna underwrite a business. That's a high bias solution, low variance, very simple model, not a very useful model. In both cases, they're problematic now. In particular, if you have deep trees, you have high variance machine learning kind of fixes this. It's a very empirical way and really two techniques that are, I see all of the time in insurance data science are random forest and GBMs and both seek to kind of fix the issues with trees while capturing the strengths.
Scott Zrebiec: (
So GBM, you kind of start with, you typically start with a medium complexity solution and you take a partial step towards that. So you just take a one 10th of that step, it improves and fixes some of the interactions and then you turn it around and say, is this the best we can do? Or can we improve this in another way, by moving it in another step, in a different direction. By doing that, you have, lower variance than a deep tree, but you still have these nuances where you're capturing these interactions, but not overfitting and it generalizes. That's very important because it produces very strong results because well, I'm very happy building exciting machine learning models. Tracy is more interested in the products that I build. The other side of this would be random forests.
Scott Zrebiec: (
Again, the goal is to improve trees, capture the strengths of the tree and being able to capture these interactions. And it does it by taking these typically very deep trees, very high variance and averaging them. And now when you average them, a lot of average results tends to be more average, tends to have lower variance tends to generalize better, but it still captures a lot of the underlying nuances and these techniques become more important as we get better data because we have data elements that are, restaurant related. We have data, have data elements that are very specific and that these interactions are so important. When I look at insurance applications and compare it to a linear model, which is a high biased technique, and doesn't have those interactions, I tend to see a 15 to 20% better, application, better results, which matters and doesn't matter in the academic sense, it matters in the real sense.
Scott Zrebiec: (
Let's look at what a 20% better model looks like on a gains chart. So I've got two models here and let's just focus on the dark blue. This is a good model. Y axis has lost ratio relativity. So the first decile for the lowest least risky 10% of premium is about 45% better average, better than average and then it consistently increases in terms of risk until the last decile is about 45% worse than average. This is actually a line. So it looks a little bit smooth and in practice, really the only difference with real data would be is it would be a little bit steeper on the edges and a little bit flatter in the middle. More risks are roughly average.
Scott Zrebiec: (
Now under realistic assumptions, we can lower the price on the low risk policies. We can give them a credit, schedule credit, and we can put debits on the high risk policies and we'll lose some of the high risk business, but it was losing money in the first place. If we do this with a variety of assumptions, like price elasticity equals two and variety of use cases on this hypothetical example, we would see about a 3% improvement in loss ratio. It's significant. Actually you'll know that right? 20 years ago when people started using predictive analytics, the early adopters of say credit in personal auto did very, very well. They expanded profitably, right? That's really the difference between using predictive analytics and not using predictive analytics critical. But more than that, if we use machine learning over predictive analytics, over linear models, which can't capture the nuances, we get that Sine curve that Sine curve is 20% better then the dark blue curve.
Scott Zrebiec: (
If I do the same analysis, it operates at nearly a full percentage point better in terms of loss ratio. So that's kind of the advantage and that's why 20% better matters. Also if your competitors are more accurately pricing, you're then exposed to adverse selection. Now I've been talking about ML. So let's start talking about human intelligence, which I find truly fascinating.as an outsider, you look at this and you see a picture of a cat in a dog. I think that's truly amazing, right? I mean is very obvious. Everyone knows it, everyone sees it. But technically that's how your brain processes it. That's not how your eye sees it. Your eye sees it the same way a computer sees it. And that's in terms of red, green, and blue, which is the only colors you really can see your brain just puts them together for the rest.
Scott Zrebiec: (
AI sees a solution as, something like 1000 x 1000 pixel values for red, green, and blue, which is absurd. Especially if you look at the blue picture, turns out, cores don't have too much blue in them. It's also very hard to use. And historically a lot of techniques have had trouble using this data and has trouble using this data because of the course of dimensionality when none of the pixel values, matter, or when you flood your data set with useless data, useless elements, you just collect everything. That's not relevant. It becomes harder to get good solutions.
Scott Zrebiec: (
Neural networks are awesome. And again, this is a technique again from the nineties, which where it finds patterns in data and it finds patterns where there's a very high structure in the data. Whereas the red pixels and blue pixels in the green pixels do not matter like no one says it's a cat or a dog because pixels 1012 is red. Like that's absurd. The overall structure and overall features do matter. And neural networks find this in very structured data. So we're seeing tremendous applications, especially with the, advancements in technology with GPUs and what have you and also lots of nuanced versions of this, where it's able to find a pattern and summarize it. That's really what neural networks do. It summarizes these extremely hard, high cardinality data and creates, data that is then ingestible in other systems.
Scott Zrebiec: (
Certainly there's a lot of applications with text images, speech language. I actually wanna bring med bills up there and ICD codes, coz that's a good example of something that's kind of skirting the boundaries. There's time data, there's high cardinality data, it's high dimensional and there is structure to it. So this is kind of a area that's kind of spreading halfway between tabular and you know, sequence data. Another really cool part is the reusable. I would love to do this cause I would love to learn karate just by plugging in some neural network component. But you know, you can take a Google model, you can take something like Yolo or you can take Bert and you can immediately use it for an insurance application and fine tune it, to get a insurance specific solution without doing some of the kind of, heavy lifting up front.
Scott Zrebiec: (
Other thing I kind of love about this is that, this tech, you know, nothing's a panacea, there's no free lunch here, the techniques actually don't work well on tabular data. I say that and just to make my life difficult exactly a week ago, a team, from Austria showed that on very small data sets of a thousand observations, neural network, I type of neural network can outperform a GBM. So if they did that, just to mess me up, which is actually kind of a funny example because I don't think I would use a neural network or a GBM for a very small data set, but that's another thing
Scott Zrebiec: (
I kind of wanted to end with, talking about kind of the future, cuz this is an ongoing, subject, the techniques we use, most of them are done in the nineties, but they're still being developed constantly. And you know, just 10 years ago, explainability was really, really challenging. Now in many ways, things like shop values, actually get most of the solution, but there's other areas of concern, including fairness, accountability, and transparency in machine learning. We know this in insurance insurance always has done sort of a review of the inputs. But more than that, what we really should be doing is additionally looking at metrics that evaluate the performance for fairness of the results. Then, also, and there's also of course, techniques that directly address this. If a fairness issue becomes apparent, likewise monitoring deployment for fairness applications is an ongoing issue and should always be done. Also wanted to say my favorite thing and what makes me happy every day is, when I was a, when I was a kid 10 year or when I was a young data scientist and deploying models used to be quite challenging. But things like Kubernetes, Docker containers makes this much easier to deploy. You can take a complicated solution from a data science workstation and deploy it in production. And then, lastly the cutting edge is always changing. And with that, I'll hand it over to Tracy.
Tracy Waller: (
Sure. Thanks Scott.
Tracy Waller: (
So as we wrap up today, one of the things that we've seen is that we are really truly in a golden age when it comes to data and analytics and using AI in an applied manner to gain real measurable results in our businesses and for our insureds as well. So, we suspect that the race to zero in small commercial will be over in just a couple of years. And we think it's gonna be a pretty exciting journey along the way. And hopefully some of these best practices that we've outlined will be helpful to you. So thank you all and happy to open it up for questions.
Scott Zrebiec: (
Don't make me a Fall on.
Tracy Waller: (
Thank you everybody. Cheers. Thanks so much.