Introduction

Validation of actuarial results is crucial for insurers to ensure accuracy and reliability of results for various purposes, such as financial reporting, pricing, risk, regulatory compliance and audit requirements. There has also been an increased focus by the Central Bank of Ireland on the validation of key inputs to results such as manual adjustments and expert judgements, further stressing the importance of results validation.

Our earlier briefing notes, “Structuring an Effective Model Risk Management and Validation Framework”¹ and “Harmonising Data: The Art of Validation and Management,”² provide a comprehensive overview of model and data validation frameworks, respectively, going into detail on inputs and calculations validation and touching on results output validation.

This briefing note delves deeper into results validation. We begin with some examples of results validation checks that can be carried out, before getting into the importance of results validation and the current challenges faced within insurance companies. We then cover how companies can implement a results validation framework and examples of tools and technologies to consider for addressing concerns regarding output quality and its governance.

Output validation checks

Companies can perform different types of checks on their results depending on the nature of the calculations performed by the actuarial models for different use cases and varying regulatory requirements.

We have developed a comprehensive in-house checklist for Solvency II results of various output validation checks and controls that can be used by clients as part of their own processes to ensure reliability and accuracy of results or for independent reviews of results.

Broadly, the checks to address output issues can be split into the following categories.

Sense checks on a single run output

These are reasonability checks that can be performed on various output fields in the set of results produced from a single actuarial model run. The run itself can be for varying purposes, for example valuation, pricing, acquisition or financial reporting. It may also vary by reporting bases, for example Solvency II or International Financial Reporting Standard (IFRS) 17.

Some examples of single run checks that can be performed by life insurers include checks on:

• Premiums: Checks that premiums are consistent with the datafile, checks that premium as a percentage of unit reserves is reasonable.
• Benefits: Checks that sum at risk is zero or positive, checks that per policy death benefit looks reasonable in comparison to the unit reserves (e.g., 101%/110% of unit reserves).
• Charges/commissions: Checks that observed charges and commissions as a percentage of unit reserves/premium are as expected.
• Expenses: Checks that expenses are consistent with assumptions, checks that observed expense inflation is consistent with assumption.
• Decrements: Checks that observed surrenders, deaths and maturities are consistent with assumptions.
• Growth/discount rates: Checks that growth and discount rates are consistent with yield curves.
• Runoffs: Checks on runoffs of reserves, risk margin and solvency coverage ratio for reasonableness.
• Independent calculations: For example, independently calculate gross/net best estimate liability (BEL) based on cash flows and discount rates and then check consistency with BEL output from model run.

Comparison checks on outputs from different runs in the same time period

Some examples of types of runs that can be compared include:

• Solvency Capital Requirement (SCR) shock runs to base runs.
• Own Risk and Solvency Assessment (ORSA) scenario runs to central runs.
• Comparison of different steps in analysis of change (AOC) runs.
• Comparison between BEL in Solvency II run and IFRS 17 run.

Outputs can be checked to ensure that only the outputs expected to change have changed and that the changes are as expected. Some examples of types of output fields or values that may be compared between the two runs include:

• Premiums
• Benefits
• Charges/commissions
• Expenses
• Decrements
• Growth/discounting

Comparison checks on outputs from different runs in different time periods

This involves checking movements in outputs over different time periods, for example from one valuation period to the next. Some examples of types of output that can be checked include:

• Movement in business volumes
• Checks that cash flow movements are consistent with movements in business volumes
• Decrement variances
• Expense changes

The importance of results validation

Results validation has wide-ranging benefits to firms, as discussed below.

• Confidence in results: Implementation of output validation mechanisms and governance around controls and checks aids insurers in gaining confidence in their results. Working with accurate and reliable outputs allows insurers to use their results with confidence for various strategic needs and decision-making processes. For example, the results can be used for:
  • Investment strategy
  • Capital management
  • Pricing of products
  • Management information and reports, including key performance indicators (KPIs)
  • Deciding on appropriate reinsurance arrangements

Reliable output from actuarial models is crucial to arrive at appropriate, informed and timely decisions to gain strategic competitive advantage.

Avoiding errors will also gain market confidence from various stakeholders including investors, shareholders, the board, policyholders and rating agencies, amongst others. It speaks to the financial stability and integrity of the insurer and can positively influence the reputation of the insurer in the market.

• Regulatory and audit requirements: The European Insurance and Occupational Pensions Authority (EIOPA) addresses output validation as part of the Solvency II Delegated Acts in Article 267.³ This requires companies to have effective systems and controls to ensure that valuation estimates of their assets and liabilities are reliable and appropriate to ensure compliance with Article 75 of Directive 2009/138/EC. It further requires companies to establish internal control processes that include an independent review and verification on a regular basis of the information, data and assumptions which are used in the valuation approach, its results and the suitability of the valuation approach.

Auditors also conduct a review of results used in the preparation of financial statements. Auditors are required to report whether, in their opinion, the financial statements give a true and fair view of the assets, liabilities and financial position of the company. They are also required to state whether they have obtained all the information and explanations necessary for the purpose of their audit. Having comprehensive clean and transparent checks and controls on model output will help to meet audit requirements and avoid any material observations.

In the following sections we discuss the key challenges regarding outputs and results that insurers may face and how these challenges can be addressed.

Issues concerning output management

Key issues that insurance firms may face in respect of managing and validating output results include:

• Inaccurate results: Insurers may be faced with inaccurate results, which can be due to:
  • Inaccuracies in the underlying data or assumptions that feed into the model
  • Errors in the underlying code in the model
  • Errors in the implementation of the model
  • Errors in manual adjustments

As insurers deal with a large amount of information, there is a critical need for output validation to ensure that this information is accurately processed and reflected in the results. If the data, assumptions and model are all OK, the main source of inaccuracy is in any manual intervention needed to process the results.

The impact of any errors may be augmented if inaccuracies and errors in model outputs are identified at a late stage in the process. This may require reruns and production of updated results. By implementing appropriate checks and controls on the results at each stage of the process this operational inefficiency can be avoided.

• Technological advancements: A large number of insurers now use various technologies such as artificial intelligence (AI), machine learning (ML) and advanced analytics for speedy actuarial calculations and analysis. There is a need for thorough testing and validation of these technologies to ensure they are not producing inaccurate outputs. Sound model validation will help to ensure this.
• Regulatory and business changes: Insurers may face issues with implementing control frameworks and governance if there are regulatory changes or changes in the business from portfolio transfers etc. Additional checks may be needed that were not needed or were not applicable for other regulations or the original types of business. The frameworks should be designed to be dynamic and easily adaptable for any changes that arise.

Addressing output issues faced by insurers

Output validation framework

A validation framework is crucial for insurers to ensure accuracy, reliability and regulatory compliance of their outputs. It helps verify the correct functioning of processes involving sensitive data and complex algorithms, thus reducing errors that could lead to financial, legal and trust issues. This framework ensures that output meets industry standards, maintaining the integrity and credibility of the insurance sector. Systematic validation enhances operational efficiency, improves decision-making, and supports fair and accurate service delivery to policyholders.

One potential example of an output validation framework has been outlined in Figure 1.

1. Define validation objectives

The key objective of the framework could be to ensure the accuracy, consistency, reliability and compliance of actuarial models and their outputs. The objective should further include building, maintaining and monitoring robust checks and controls on outputs from various actuarial models used by the insurers to ensure that the output has been validated and is accurate and reliable for the insurer’s use.

Insurers may add other objectives to the above which may be specific to their company requirements and previously identified issues, for example checks on unmodelled reserves.

2. Identify key outputs for validation

Insurers should clearly define which outputs will be validated, for example outputs from pricing, reserving or risk models. Consideration should also be given to the suite of calculations that are produced, for example calculations for valuations, pricing, shocks and stresses, sensitivities, risk and projections.

Insurers should define the type and nature of output from the models along with their purpose. For example, the different model outputs contributing to balance sheets, income statements, disclosures or quantitative reporting templates should all be defined. The purpose for the output could include analysis of results for risk assessment, investment strategy, reinsurance arrangement and pricing, among others.

3. Define roles and responsibilities

Insurers should define clearly the roles and responsibilities within various teams involved to demonstrate independence between the individuals responsible for producing the outputs and those responsible for conducting checks and controls. This will also reduce any undesired duplication of work and improve efficiency.

4. Develop validation rules and criteria

Insurers should develop rules and criteria for the validation of output from the actuarial models, for example BEL is expected to be between X% and Y% of unit reserves, risk margin is expected to be Z% of BEL etc. Regulatory requirements such as specific rules based on local and international insurance regulations, e.g., Solvency II, IFRS 17 or other applicable frameworks should be included in developing these rules. When setting criteria for validation, (re)insurers should also consider actuarial guidelines, market practices for risk assessment and financial projections. Insurers may also take into account acceptable levels of tolerance for the firm and criteria for accuracy, completeness and reliability.

5. Implement validation checks and controls

In addition to the validation of input data, assumptions and models, insurers should define and implement checks and controls on their results.

The following types of checks and controls may be employed:

• Replication: Independently replicate outputs to verify accuracy.
• Benchmarking: Compare outputs against benchmarks or industry standards.
• Reasonability checks: Assess the plausibility of results given historical data and business context.
• Trend analysis: Identify and explain deviations from historical trends.
• Reconciliation: Ensure consistency between different actuarial outputs (e.g., reserving vs. pricing).

Checks and controls are discussed further at the beginning of this note.

6. Documentation and reporting

As part of this step, companies should develop and maintain comprehensive documentation for various aspects of validating outputs. They have been listed below:

• Documentation to outline the validation process, findings and any anomalies or outliers detected.
• Documentation for any expert judgements applied and any one-off or manual modifications made.
• Documentation to describe the steps taken to correct any identified issues.
• Reports for stakeholders highlighting any key findings and noting any key areas of concern.

This will aid insurers in maintaining transparency and increasing the reliability of the results for various stakeholders.

7. Review and feedback

Companies should incorporate internal peer review of results to mitigate the risk of errors and improve quality assurance and credibility. Further, insurers may also consider undertaking an independent review of their models, methodologies and results to get an objective assessment of their results and identify issues or potential errors that may have been overlooked.

The review process is critical for risk management and accountability. It also supports compliance with professional standards and guidelines, including for example the requirement for a peer review of Technical Provisions (TPs) and related Actuarial Opinion of TPs (AOTPs) and Actuarial Report on TPs (ARTPs) under the Central Bank of Ireland’s domestic actuarial regime and related governance under Solvency II.

Companies should allow for feedback by incorporating learnings from validation processes to refine and improve validation rules and processes. They can use validation insights to improve data collection, modelling and reporting processes.

8. Regulation and governance

As part of this step, companies should ensure that the validation process is in compliance with relevant regulations, standards and guidelines relevant to their markets. They can establish a governance framework to this effect to oversee the validation process and ensure accountability. Companies can also conduct audits by external bodies to ensure compliance and validate the framework for checks and controls.

9. Continuous monitoring

Regular monitoring of the actuarial models and outputs is crucial to ensure that they remain accurate. Frequent monitoring, for example when model updates are made or after defined lengths of time, aids insurers to identify and correct errors early and contributes to process efficiency and the production of accurate results.

Companies can regularly update validation tools and technologies to keep up with advancements and emerging trends in the market. They should also allow for continuous training and creating awareness for staff on the importance of validation and their roles in the process.

Tools and technologies to consider for model output validation

• Data management and visualisation tools: Excel remains a popular tool with numerous insurers for data manipulation, ad hoc calculations and data and output checks. It can be used in conjunction with other visualisation tools such as Tableau and Power BI for creating visualisations and dashboards that can be used by actuaries to analyse model results, compare key indicators against previous periods and identify any outliers in results movements.

• Statistical and analytical software: R and Python are open-source programmes which can be used to validate model output. These programming languages offer statistical libraries for data analysis, data visualisation and model validation. These programmes can be used to create functions or routines that assess whether model output meets specific criteria. These programmes or scripts can be made a part of the valuation and governance processes to perform the checks and controls on the actuarial outputs on a quarterly/annual basis for production runs.
• Artificial intelligence (AI) and machine learning (ML): AI and ML can play a crucial role in validating outputs from actuarial models for insurers by enhancing the precision and efficiency of the validation process. These technologies can automate the analysis of vast datasets, identify subtle patterns and detect anomalies that might be overlooked by traditional methods. This can lead to more accurate and reliable actuarial results.

However, there are significant risks to consider, such as the potential for algorithmic bias, which can skew results if the training data is not representative. As with traditional models, AI and ML models require continuous updates and human oversight to ensure they remain accurate and relevant, preventing overreliance on automated systems that could lead to flawed decision-making.

Conclusion

In conclusion, output validation is crucial for insurers to ensure the accuracy, reliability and compliance of their actuarial results. Robust validation mechanisms are essential to prevent incorrect decisions, regulatory sanctions and loss of market confidence. The challenges of output management, including model errors, data inaccuracies and technological changes, underscore the need for a comprehensive validation framework.

A structured framework with clear objectives, roles, responsibilities, validation rules and continuous monitoring is vital. This not only meets regulatory and audit requirements but also enhances efficiency and supports strategic decisions. Employing checks such as recalculations, benchmarking and reasonability tests ensures accuracy and consistency.

Advanced tools and technologies, such as data management software, statistical programs and AI and ML, enhance the validation process by automating and refining data analysis. However, addressing risks like algorithmic bias and data privacy through continuous updates and human oversight is imperative.

A well-implemented validation framework fosters transparency, accountability and trust, reinforcing the insurer's financial stability and integrity. By adhering to these principles, actuaries can significantly contribute to the insurance industry's sustainable growth and resilience.

---

¹ McGinley, D., Gleeson, C., Mcllvanna, M. & Stack, E. (April 2021). Structuring an Effective Model Risk Management and Validation Framework. Milliman Research Report. Retrieved 6 December 2024 from https://ie.milliman.com/en/insight/structuring-an-effective-model-risk-management-and-validation-framework.

² King, E. & Mittal, A. (June 2024). Harmonising Data: The Art of Validation and Management. Milliman White Paper. Retrieved 6 December from https://ie.milliman.com/en-gb/insight/harmonising-data-art-of-validation-and-management.

³ The full text of the regulations is available at https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32015R0035.