Aviva, an insurance giant with a rich history dating back to 1696, operates in 16 countries and serves over 33 million customers. The company has always placed a high value on helping people safeguard what is most important to them, whether that be health, home, family, or financial future. The need for innovation led them to exploit machine learning (ML) across more than 70 use cases. However, the previous approach, which involved using a graphical UI-driven tool and manual deployment, demanded over half of data scientists’ time for operational tasks, significantly stifling innovation and complicating the monitoring of model performance. Recognizing these inefficiencies, Aviva embarked on a transformation journey with Amazon Web Services (AWS) to build a scalable, secure, and reliable MLOps platform using Amazon SageMaker.
Incident Occurrence
The transformation journey was a response to a significant challenge: deploying and operating ML models efficiently at scale. According to Gartner, approximately 47% of ML projects never reach production, emphasizing the need for consistent processes, effective monitoring, and robust technical and cultural foundations. Aviva, handling around 400,000 insurance claims annually and approximately £3 billion in settlements, felt the pressure to offer a seamless digital experience to its customers. Increasing automation via AI technologies became crucial as the company faced rising claim volumes.
To prove the viability of the newly developed platform, Aviva chose their Remedy use case as the pilot project. This use case focuses on a claim management system that uses data to decide whether submitted automobile insurance claims qualify as total loss or repair cases. The workflow, which starts when a customer experiences a car accident, involved several intricate steps demonstrating the platform’s capabilities.
Customer Report
The next step in the workflow requires the customer to report the incident to Aviva, detailing the specifics of the accident and the damage incurred. This step is pivotal as it forms the basis for the ML model to estimate repair costs. The data collected is processed using 14 ML models in conjunction with a set of business rules to derive a preliminary repair cost estimate. This integration of business rules with advanced ML models ensures that the predictions are not only data-driven but also align with Aviva’s established policies, making the entire process both efficient and reliable.
Repair Cost Estimation
Once the customer has provided the necessary incident details, the ML models kick into action to estimate the repair costs. This stage is complex, involving 14 ML models and a set of business rules. These models process the incident information to generate an estimated cost of repair. The rigorous training and fine-tuning of these models have equipped them to handle a variety of input data, ensuring that the repair cost estimation is as accurate as possible. This multi-layered approach of using both ML models and business rules ensures that the system is both flexible and robust, capable of handling various types of claims efficiently.
Cost Comparison
After the preliminary repair cost is estimated, this figure is then compared with the vehicle’s current market value acquired from external sources. This step is crucial, as it helps to determine whether repairing the car is economically viable. This comparison involves merging data from various sources to get a comprehensive view of the vehicle’s current worth. The market value acts as a benchmark against which the repair cost is judged, ensuring that the decisions are grounded in current market realities. The models used for this comparison are continuously improved based on feedback and new data, ensuring that they stay relevant and accurate over time.
Market Analysis
The subsequent step is market analysis, which augments the cost comparison by incorporating data about similar cars for sale in the local area. This additional layer of information helps refine the decision-making process. By analyzing similar vehicles in the market, the models can provide a more nuanced recommendation, taking into account not just the repair costs and market value of the car, but also how these figures compare to other vehicles that are currently available for sale. This helps in making a more informed and transparent decision, benefiting both the company and the customer.
Recommendation Generation
Finally, based on all the processed data, a recommendation is generated on whether to repair or total the car. This recommendation, along with supporting data, is then passed to the claims handler, concluding the workflow. The recommendation is derived using a combination of ML model predictions, external data, and business logic, ensuring that it is both accurate and actionable. This entire workflow serves as a template for future use cases, enabling Aviva to replicate similar processes efficiently and consistently across different scenarios.
Reusable and Scalable ML Pipelines
Building the MLOps platform involved developing reusable ML pipelines that streamlined the entire model lifecycle from experimentation to deployment. The processing, training, and inference code for the Remedy use case was developed in SageMaker Studio. Once the experimentation phase concluded, the refined seed code was pushed to an AWS CodeCommit repository. This push triggered a CI/CD pipeline that automated the construction of a SageMaker pipeline encompassing data processing, model training, parameter tuning, and model evaluation.
The platform also employed Amazon SageMaker Automatic Model Tuning, allowing advanced tuning strategies that overcame complexities related to parallelism and distributed computing. The initial model tuning involved training nearly 100 model variations using Bayesian optimization, helping to achieve optimal performance. Only models that met specific accuracy thresholds were registered in the SageMaker Model Registry. A custom approval step ensured that only Aviva’s lead data scientist could deploy a model through the CI/CD pipeline to SageMaker real-time inference endpoint, further validating the models before they went into production.
Serverless Workflow for ML Model Inference
To harness the business value of ML model predictions, a serverless workflow was integrated with Aviva’s internal systems for real-time inference. The workflow begins with a request to an API endpoint hosted on Amazon API Gateway. This request invokes an AWS Step Functions workflow utilizing AWS Lambda for various tasks, including feature encoding, model inference, business logic processing, and response generation. The real-time inference endpoints addressed the challenge of providing consistent predictions at low latency, crucial for Aviva’s daily claims processing.
Monitoring Model Performance
To ensure the reliability of the ML models and build trust among users, real-time monitoring of model decisions was essential. Detailed insights into each state machine run and task were crucial for claim handlers to understand the rationale behind each decision. Snowflake, Aviva’s data platform of choice, was integrated with Amazon CloudWatch logs and Amazon Kinesis Data Firehose to aggregate data for business analytics. This setup enabled the creation of comprehensive dashboards and reports, providing insights into various metrics like total losses by region and average repair costs by vehicle type.
Emphasizing Security
Handling personally identifiable information (PII), the platform incorporated robust security measures to protect customer data. This included network restrictions, data encryption in transit and at rest using AWS Key Management Service (AWS KMS), and restricted access to production data. Additionally, safeguarding Aviva’s intellectual property was a focus, with secure storage of code, training data, and model artifacts in S3 buckets, protected by IAM policies. The platform also ensured auditability by logging all steps of inference and decision-making, with logs encrypted and stored with lifecycle policies for regulatory compliance.
Conclusion
The next step in the workflow requires the customer to report the incident to Aviva, providing detailed information about the accident and the resulting damage. This crucial step sets the foundation for the machine learning (ML) model to estimate the cost of repairs. The detailed data collected from the customer is then processed through 14 ML models along with a set of predefined business rules. These rules are crafted to ensure that the preliminary repair cost estimate is not only driven by data but also compliant with Aviva’s established policies. This mix of business rules and advanced ML models ensures both the efficiency and reliability of the cost predictions.
Aviva places significant emphasis on the accuracy and reliability of the repair cost estimation process. By employing multiple ML models, the system can cross-verify predictions, thereby minimizing errors. Furthermore, the business rules integrated into the models reflect Aviva’s long-standing expertise in handling such claims, ensuring the outcomes are not only technologically advanced but also practically sound.
This thorough approach guarantees that customers receive a well-rounded and fair estimate of repair costs. The combination of data-driven ML models and business rules means that every estimate aligns with both modern technology and traditional industry standards. As a result, the workflow remains robust, efficient, and trustworthy, enhancing overall customer satisfaction and operational effectiveness for Aviva.
