Quantification of model risk

For this task, the first step is to assess the material models.

How to identify a material model? This could be based on two criteria:

Scope

Is the model used for regulatory purpose? Is it used to provide pricing information binding for the financial institution? Is the model’s outcome helping to take strategic decisions?

Complexity

This is the combination of many elements such as the number of formulas used in the model, the approach applied, the number of data inputs, assumptions, and limitations relevant for the model.

In practice, the combination of the above criteria can be used to determine the rating for the model materiality as low, medium and high for example. This is an extremely important information which can drive the priority in the validation plan and calibrate the effort in model validation.The rating system of the models is an essential step which leads to the next one: how to quantify model risk?There are many challenges including the following:

Model failure criteria are hard to define as they are subjective

Non-homogeneity such as model risk can have many sources

There are direct and indirect effects (e.g. outcome of the model and decision taken based on it).

Change to one or more data inputs (variables)

Recalculation of the outcome of the internal model (A)

Recalculation of the outcome of the internal model (B)

The quantity A) – B) gives the measure of the uncertainty of the model.

Having said that, there are many ways to quantify model risk. Some of them are more qualitative while others can be more quantitative.A reasonable approach is represented by the following three steps⁹:

Identification of the model risk sources (model risk factors)

Quantification of the inherent model risk of each source

Mitigation of the identified model risk.

Table 1 provides some specific examples.

Identification	Quantification	Mitigation
Data error / missing/ insufficient	Sensitivity to data inputs	Capital buffer / conservatism / back-testing
Computation error / invalid assumptions	Benchmarking / measure of statistical method	Robust model governance
Incorrect / improper use/execution	Decay of predictive power over time	Data quality assurance

A more quantitative approach is sensitivities analysis which consists in the following steps:

Model risk monitoring

What model performance is acceptable?

Aggregation and reporting of model risk

Application to stress testing models