Consider the case of the railway worker tasked with tapping the wheels of trains arriving at a small Karoo town with a hammer. At his retirement party the mayor brightly asked: “What was the purpose? Was that a safety check?”. “I don’t know,” answered the railway worker. The regulation of artificial intelligence (AI) carries the same challenge: diligent supervisors conducting checks with output they cannot interrogate.

One challenge that has received a lot of public discussion is the risk of model bias. Biased model outcomes could reflect the use of unrepresentative data with which to train a model or aspects of the model. For example, AI models trained on specific demographic information might generate inaccurate results when applied to people outside that demographic group.

Such bias could result in unjust or even illegal treatment of specific groups, such as racial discrimination, or affect the elderly, lower-income households or less financially astute investors. Clearly it is something a just society would want regulators to have a firm grip on, but there is no industry consensus on how to measure bias.

Though there are statistical methods for testing for skewed sample distributions, a regulator would need to be able to interpret the complex data and model outputs used by firms or test these models itself. But even this approach might be insufficient given the nature of AI.

Imagine a bank uses a data set of 100 properties, 90 with red roofs and 10 with blue, to train its AI model. The model could produce biased estimates of the value of properties or treat certain credit customers unfairly if the ratio of red roof to blue roof houses is different from 9:1 in the country, or if certain demographic groups tend to prefer red roof houses and such customers also tend to have different creditworthiness from other borrowers.

AI models are useful for unearthing complex relationships between things such as property size, location, age or other property features, but judging whether their predictions are reasonable requires the ability to interpret the relationships between property features and property prices. The contribution of different features to AI model predictions can change dramatically over time and as new data is used.

This problem could become even more challenging as AI models begin being trained on web-based data or the output of other AI models — so-called synthetic data. Regulators may well want to prohibit the use of synthetic data and require tests to differentiate between actual and synthetic inputs and the extent to which synthetic data improves model performance.

There is also the possibility that firms might draw on data from vendors that could be protected in terms of privacy law. The creation of large, linked data sets creates new and complex data breach risks and potentially new systemic vulnerabilities.

This leads to the question: “When and how often should a regulator require that institutions test for bias in their models?” One-and-done tests or even set-frequency periodic tests would not guarantee that models are compliant with regulatory requirements.

Another challenge is the extent of human involvement. A prudent approach might be to require that companies keep a firm hand on the tiller, so to speak. This means the output of an AI model that, for example, provides a credit score or assesses whether an applicant should be granted a mortgage should have a human at the assessment stage to check the results and the model’s rationale.

This raises the question of how heavy the hand and how big the tiller should be. A sophisticated AI model might involve many millions of calculations to arrive at a model output. Requiring human intervention is like giving a paddle to a strongman to steer an oil tanker. With thousands of daily decisions to make, it is not practically possible to have constant human intervention.

This is not to say financial regulators should surrender responsibility. On the contrary, this will require development of new governance structures and greater involvement of regulators not just in model deployment but in the ongoing operation of AI systems.

Supervisors also need to invest and run their own AI models internally, if only to better understand these technologies. Regulators need to employ more data scientists and statisticians and give them the needed authority to lead on policy formulation. Regulators need domain expertise and must be capable of responding to a changing AI landscape.

Expect AI to reshape banking regulations. AI offers the prospect of automating much of the regulatory reporting process, reducing regulatory compliance costs and improving many banking services. But realising these benefits, mitigating emergent risks and promoting the accountability of financial firms will require a rethinking of regulations and regulatory processes.

Rapid changes in technology and monitoring challenges for AI models mean regulations need to become more principle-based, with new accountability and responsibility mechanisms to promote self-regulation and market discipline. If regulators do not understand AI they may fall into the same trap as the railway worker in our story: hearing the wheels ping but not knowing what to make of it.

Dr Steenkamp is CEO at Codera Analytics and a research associate at Stellenbosch University. Roos is an associate with Codera.