Risk and AI: It's Tricky

Oftwominds.com‘s eclectic range of timely topics include finance, economy, stocks, housing, Asia, energy, long-term trends, social issues, urban planning, work/tradecraft, health/diet/fitness, sustainability, Les Paul guitars and The Great Transformation ahead: www.oftwominds.com/blog.html.

The possibility that AI will end up unleashing waves of ‘Anti-Progress’–malicious uses, untrustworthy output and uncontrollable floods of slop–also doesn’t occur to those confined in the current belief construct.

A funny thing happens on the way to understanding risk: we discover it’s tricky. We think we see all the risks, and think we can mitigate or hedge those risks, but by its very nature, risk evades such simplistic filters and metrics. Risk remains hidden, offscreen, invisible, building up out of sight, awaiting a catalyst that’s equally undetectable until it manifests, and after the fact, we look back and ask, why didn’t we see that coming?

Risk is tricky like that. It can lay dormant for decades and then erupt with little warning.

Risk is tricky in other ways. In our hubris, we see the power and might of our technologies, systems and foresight, and reckon these are so robust they will easily survive any tectonic shift, as we’ve planned for emergencies.

But our faith in the might of our civilization is itself a source of risk because the risk of Model Collapse–the breakdown not of a supply chain or technology but of our entire conceptual construct of how the world works–goes unrecognized because our confidence that our model maps the real world is so high that we are incapable of recognizing its drift into hallucination and civilizational psychosis.

In other words, our confidence that our conceptual mythologies are accurately mapping the real world is itself a source of civilizational risk because this confidence makes it inevitable that we do more of what’s failing, as the alternative–recognizing our conceptual models and mythologies are self-serving rationalizations that substitute artifice for realistic appraisals–is conceptually and emotionally impossible.

Put another way: Emperor Norton’s delusions of power and grandeur were harmless as long as he was recognized as delusional. But should Emperor Norton actually be given the power he believed was his to wield, then risk rises accordingly.

Consider the bet being made globally that the current iteration of AI will be 1) immensely profitable (the most important thing in the Universe) and 2) immensely productive (secondary to immensely profitable but necessary as a motivation for everyone to throw trillions of dollars at purveyors of AI). The risk that this bet–and the assumptions that make it not only rational but pressing–is the equivalent of handing Emperor Norton the keys to the kingdom with little evidence he will be a wise leader, is unimaginable in the current model / mythology, and so therefore it doesn’t exist.

The worst that could possibly happen in the current model / mythology is a brief spot of bother in the stock market as euphoric overvaluations come down to Earth, and then the immense profits start flowing and markets rocket higher in a multi-decade Bull Market of AI Productivity.

The possibility that the current iteration of AI is innately incapable of metaphorically boiling away the seas is not on the screen, any more than a stock market crash or social upheaval is on the screen. Yet if the fantasy of vast, unstoppable floods of profits driven by vast increases in productivity fail to materialize on a very short timeline, then both a stock market crash and social upheaval move from “impossible” straight through “unlikely” to “happening now,” leaving everyone who thought they understood risk and were properly hedged against unwelcome change in a state of disbelief and wonderment.

Risk is tricky that way. What’s “impossible” in our current belief construct–a construct we mistakenly believe maps the real world perfectly–is a source of system-breaking risk that is invisible within the confines of this self-congratulatory belief construct.

The possibility that AI will end up unleashing waves of Anti-Progress–malicious uses, untrustworthy output and uncontrollable floods of slop–also doesn’t occur to those confined in the current belief construct. The risk may be of a magnitude and scale that switching AI vendors or platforms and approving policy tweaks won’t fix the problem.

My book Investing In Revolution is available at a 10% discount ($18 for the paperback, $24 for the hardcover and $8.95 for the ebook edition). Introduction (free)

Become a $3/month patron of my work via patreon.com

Subscribe to my Substack for free

---

NOTE: Contributions/subscriptions are acknowledged in the order received. Your name and email remain confidential and will not be given to any other individual, company or agency.

Thank you, Peter ($70), for your superb generous subscription to this site — I am greatly honored by your support and readership.		Thank you, Mark ($7/month), for your marvelously generous subscription to this site — I am greatly honored by your support and readership.

Thank you, Douglas H. ($70), for your massively generous subscription to this site — I am greatly honored by your support and readership.

Thank you, Vermont R.P. ($70), for your splendidly generous subscription to this site — I am greatly honored by your support and readership.

Source: http://charleshughsmith.blogspot.com/2026/07/risk-and-ai-its-tricky.html