A sensor can fail and keep sending the same value. This state is known as stuck data. When it happens, the screen shows the same figures as when nothing is wrong. Some of our water level gauges have a self-diagnostic function that detects this state, but even so, in an emergency, the last safeguard is still a human.
AI is a good partner. It organises vast amounts of information and quickly offers plausible answers.
But there is a danger in this. The more plausible an answer, the more people tend to stop checking it[1]. Entrusting thinking to tools is an old practice, and it has been studied extensively[2]. Yet there is a risk that, as we keep entrusting it, the very ability to check will wither, as evidence from the medical field suggests[3]. Before long, we may be left only approving AI’s answers.
It is the same structure as the sensor story. Stuck data is dangerous because it does not look wrong. Plausible answers are the same. When a mistake does not look like a mistake, only a human with a habit of doubt can notice it.
The real danger is that humans give up their judgement.
There are four safeguards.
First, decide in advance which judgements remain with humans. Decisions that cannot be undone, or that have a large impact, are made by humans.
Second, layer physical safeguards. Self-diagnosis and the eyes of people in the field. If one is breached, the next one works.
Third, have mechanisms for doubt. Technology to verify that information is genuine, and a habit of pausing before information that stirs up emotion.
Fourth, change the human role from passive to active. Not only giving instructions, but asking, checking and deciding.
Keeping systems running, and not handing over too much. Holding these two in balance is the work of human care. We keep watch with a question mark after ‘All Clear’. The last line is drawn by our own hand.
Sakae Mugiuda, President