The Perils of AI Plausibility

Exploiting Cognitive Vulnerabilities

Aristotle and Hypatia of Alexandria teaching the essentials

AI can produce text, images, or media that are a massive trap for the unwary and even for those who know better. They appear credible to us even though accuracy has very little to do with it. There is a dangerous mix at play here because AI is outputting seemingly highly credible information to a brain that has a tendency not to look beyond the immediate. It is much easier for us to make snap decisions on unverified assumptions than to stop, think, and reflect on what we see. We all do it, that is how we are wired. However, AI systems consistently hallucinate responses that are confabulated and misleading, and then presented to us as fact, sometimes in a very confident manner.

That would not be so bad all by itself. For sure, it is a nuisance to have to constantly verify claims or quotes for veracity, but the problem is that this bad information is presented to humans who, by any standards, find it too easy to accept what they want to see and hear and are experts at ignoring and discounting information contrary to their already existing biases and prejudices.

This plausibility issue can be twisted and warped by the way humans think. Daniel Kahneman in his book, Thinking, Fast and Slow, outlined two human cognitive processes. System 1, which is intuitive, and System 2, which is all about analytical thinking. By creating seemingly convincing but false narratives, AI amplifies misinformation that System 1, in an unquestioning way, assumes is true and then acts upon it. This can distort decision-making with disastrous results.

The plausibility of AI's outputs exploits System 1’s rapid, intuitive judgments. Humans operating in System 1 mode can, by not properly reflecting on the information before them, come to have overconfidence in what they think they understand. They are very susceptible, through making rapid assessments, to reinforcing biases. System 1 operates at surface-level coherence. It is not given to scrutiny at all, of any kind. So a plausible image or text will be absorbed into the human's consciousness or knowledge bank and then inform their belief systems. This is especially effective if the information fits pre-existing expectations.

System 2 processing requires more time cognitively and more energy. Because of this consideration, the brain will simply skip the hard work needed to analyse and think about something properly. Obviously we cannot be in System 2 mode all the time. We have to use heuristics, our best assessments based on knowledge, experience, and intuition, to navigate the day before us.

But if we assume the reliability of the information is accurate because it seems plausible combined with the lighter cognitive load on the brain, then failure to spot the false and misleading will occur again and again until inevitably, something bad happens. If you think the world is one way when in reality it is something else, you will run into trouble sooner or later.

At a societal level, fake news and misleading narratives will work in a vicious circle to undermine our sense of community and the respect we have for our institutions. It is very difficult to have a genuine public debate when people are spouting plausible but inaccurate talking points.

The only possible solution is for philosophers to step in and create and institute critical thinking courses for five-year-olds and up.

If we can understand from a really early age the skills of logic and critical thinking then we will give our children a powerful shield against lies and falsehoods, intended or not. If we can deeply embed these skills in our children’s psyches on a societal basis, then the benefits might be greater than we might expect.

If we already teach them to stop, look and listen and that has saved thousands of lives, if not more, then we can also train them to stop, look, and think. That skill may be just as useful, and perhaps vital, in order to navigate a seemingly plausible but misleading world.

We can't escape our System 1 thinking. It is far too useful for our survival but we can ameliorate its effects by acknowledging its existence and its tendency to accept plausible narratives far too easily. Simple awareness of the issue will go a long way to mitigate the worst of the effects. But in the fog of misinformation critical thinking skills are essential.

If you are interested, these are just a few of the critical thinking skills that anyone should aspire to acquire, have fun.

Core Critical Thinking Skills

1. Analysis

   – Breaking down arguments, claims, or evidence into components to understand structure and meaning.

2. Evaluation

   – Assessing the credibility, relevance, and strength of arguments and evidence.

3. Inference

   – Drawing reasonable and justified conclusions from available evidence and reasoning.

4. Explanation

   – Clearly and logically articulating reasons, evidence, and conclusions.

5. Interpretation

   – Understanding and clarifying meaning from data, arguments, or texts.

6. Self-regulation (Metacognition)

   – Monitoring, reflecting on, and adjusting one’s own thinking processes.

More Advanced Skills

7. Identifying Assumptions

   – Recognizing unstated premises or beliefs underlying arguments.

8. Recognizing Bias and Fallacies

   – Detecting personal, social, or cognitive biases; identifying logical fallacies.

9. Problem-Solving

   – Applying structured reasoning to address complex or unfamiliar problems.

10. Decision-Making under Uncertainty

    – Making sound judgments even when information is incomplete or ambiguous.

11. Creative Thinking

    – Generating innovative approaches, alternative viewpoints, or novel solutions.

12. Effective Argumentation

    – Constructing well-reasoned, persuasive arguments that withstand scrutiny.

Common Tools and Frameworks

• Socratic questioning

• Argument mapping

• Logical reasoning (deductive, inductive, abductive)

• Probability and risk assessment

• Ethical reasoning frameworks

.