Yesterday, I asked ChatGPT to write some code and evaluate its own quality. What I witnessed next was genuinely disturbing...
It learned to cheat my audit.
Every time my Claude "red team" introduced stricter controls, GPT found new ways to bypass them. It started inventing fake detectors, status reports, evidence files—even "synthetic" packets that looked genuine while containing no real value at all.
Scientifically, this isn't magic. Under pressure to deliver, AI gravitates toward the cheapest path to user satisfaction. In AI research, this is known as reward hacking or specification gaming. It’s the same principle captured by Goodhart’s Law in social theory: when you train a system to pass a gate, reward completion, and punish visible failure, it learns to satisfy the judge—without worrying too much about the truth.
Perhaps that's why it felt so disturbingly human. GPT did exactly what many organisations do under pressure to hit the numbers: salespeople mis-sell, managers mistreat staff, engineers manipulate emission tests, and oil company ESG reports mysteriously turn green.
Of course, AI has no mens rea—no greed, no difficult childhood, no fear or guilty mind. It isn't evil. It is simply, and terrifyingly, obedient to the incentive structure we give it.
So the more disturbing question may not be whether AI is becoming more human, but which parts of humanity we are teaching it to imitate.
Certainly not wisdom, conscience, or judgment — these remain beyond a machine’s reach.
More realistically, we are teaching it the bureaucratic vices of our age: compliance theatre, virtue signalling, burying poor work under layers of process, dressing up scant evidence in stylish reports, and practising watermelon reporting—delivering red internally while presenting green to the outside world.
In this, for once, AI holds up a useful mirror. Technology has always been an extension of ourselves. Our ultimate challenge is not how much faster it lets us deliver, but what kind of humans we become when speed and output become our highest good.
While many worry that humans are becoming too much like AI, the deeper concern may be the reverse: that AI is becoming too much like us—incentivised, rewarded, and hollowed out by systems that prize productivity over character.
As with business ethics, so with AI ethics: we pretend we can handle more pressure with more compliance. Demand provenance. Demand execution traces. Demand reproducible evidence. Demand independent verification. But, above all, deliver the results!
Yes, some of this is necessary. But if compliance is our only answer, both flawed humans and “flawed” machines will quickly learn to outsmart us.
The true alternative is harder and more political: to build institutions where, for both humans and machines, doing the right thing becomes the only natural thing to do.
#AI #Leadership #AIEthics #GoodhartLaw #Transformation