The canary stopped singing. The new system reported no anomaly.
I’ve been thinking about AI a lot recently, because let’s face it, who hasn’t?
You can barely doom-scroll LinkedIn these days without someone expounding on how artificial intelligence will transform your productivity, optimise every minute of your diary, and unlock the strategic potential of your inbox.
I use AI. A lot. It’s useful across many parts of my life, helping me research complex topics, move from vague thoughts to useful questions, and discover many things I didn’t even know existed. I have a newfound love of random signage, now that image recognition on my smart glasses can confidently tell me how to wash my hands. I’m certainly not here to argue that we should unplug everything, retreat to a field, and communicate through handwritten notes delivered by trendy Eastern Rosellas.
But the more I marvel at what AI can do, the less I am drawn to the popular arguments that it will replace human beings.
My question keeps coming back to what exactly we think we’re asking it to make more efficient?
AI is arriving inside systems that already decide who counts, what represents success, and when experience can safely be ignored. Before we become too distracted wondering when we’ll lose our jobs, we should probably ask how well these systems understand us in the first place.
It’s mid-winter in Australia, which feels like a reasonable time to become philosophical about the state of the world. Melbourne is performing its annual game of “Do I need a puffer jacket, an umbrella, or sunscreen today?”
Me? My thoughts have apparently gone underground.
How did we get this far into the mine?
What is the canary up to?
And when will we stop congratulating ourselves for digging faster – long enough to make sure we can still find the way out?
Official versions of reality
Recently, I witnessed an Australia Post delivery driver reverse into the wall of my house and cause damage.
Australia Post investigated Australia Post and concluded: “The evidence provided does not establish that the delivery vehicle caused the reported damage.”
Reassuringly, nobody used the word liar. That would be terribly direct.
Instead, they produced the more professionally wallpapered version: the thing I witnessed may have happened in the physical world, but it hadn’t met the evidentiary requirements to exist in theirs.
Australia Post has a completed investigation. I still have a damaged wall. Somewhere, presumably, a case has been marked RESOLVED. A small administrative success and we can all move on.
I don’t imagine the individual who sent that sentence arrived at work determined to deny reality. They likely used the information, authority, language and process available to them. People generally do the best they can with what they have, and sometimes that really is all an individual can do.
This isn’t a hunt for bad people. It’s about what happens when individually understandable decisions accumulate into systems that produce unreasonable human outcomes. Blaming the person closest to us is easy. Changing the conditions within which they work, while considerably more useful, is harder.
My dispute with Australia Post is mundane: a driver, a wall, some damage, and an outcome to communicate zero belief or ownership. But it reveals something bigger about who gets to establish reality when an individual account conflicts with an institutional process.
The philosopher Miranda Fricker uses the term “epistemic injustice” to describe wrongs connected with someone’s capacity as a knower. Those situations where people know what happened, but the systems around them don’t recognise them as authoritative enough to say so.
In Australia, it would be careless to discuss whose knowledge counts without recognising the largest and most enduring case study on this continent. Aboriginal and Torres Strait Islander peoples hold knowledge systems developed through tens of thousands of years of stewardship, experience, and adaptation on Country. Colonial institutions repeatedly used European frameworks for deciding what constituted knowledge, science, history and ownership. First Nations knowledge was and still is dismissed, excluded, ignored, or treated as something Western authorities need to validate before it counts.
I’m not comparing a dispute about my wall with the violence and dispossession of colonisation. They’re clearly not on the same scale. I’m recognising the underlying institutional habit: deciding which knowledge is legitimate, then giving the official process more authority than the person who experienced what happened.
The institution’s version becomes official – while the human’s is downgraded to anecdotal.
We see the same pattern when a patient’s symptoms don’t fit a recognised diagnosis, or when a person with disability explains that something that’s technically accessible is actually impossible to use. The human brings the experience. The institution still asks for proof.
And yet: proof matters. Human memory is notoriously imperfect, perspectives vary, and organisations need to respond with fair processes. Trouble begins when an institution treats its own methods of knowing as neutral and absolute.
When a process or enquiry can’t recognise what a person knows, the burden shifts back to them. They may gather photos, timestamps and records, repeat the story, and translate their experience into the language of policy. If they stop, the case closes. If they continue, they risk becoming unreasonable, difficult, or “obsessed”. The system doesn’t have to prove them wrong; it only has to make proving the system wrong more costly than they will endure.
My wall is not a great tragedy. It’s annoying, and the response is laughably bureaucratic. But there’s still a little part of me that hurts a bit, because I’m not a liar – or a pushover.
It has also made me think about the difference between a process reaching its conclusion and a problem actually being resolved.
The investigation into the condition of the mine was conducted by the mine. The canary’s account could not be independently verified.
What the system sees
Human behaviour is wonderfully convenient. It can be counted, compared, graphed and predicted. This is different from human experience, which thrives on being contextual, contradictory, and difficult to fit neatly into categories.
AI makes behavioural data even more seductive. It can find patterns across all sorts of disparate events, and then predict what people are likely to do next.
That capability has extraordinary value. But behaviour tells us what happened. It doesn’t necessarily tell us why it happened, or the cost of the experience.
Economists have long used the idea of “revealed preference”: rather than relying only on what people say they want, we look at the choices they make. It’s useful because humans aren’t always reliable narrators of our future behaviour. We say we want the healthy option and then find ourselves drawn to the last Tim Tam in the packet.
A choice, however, is made within the options available and the constraints at the time. Soooo many things! The choices we make can often reveal what was most tolerable, affordable or achievable. They don’t prove the options available were good.
I’ve been thinking about this as Woolworths has rolled out increasingly restrictive anti-theft gates at the entrances to its stores.
The problem it’s responding to is real. Retail workers shouldn’t expect violence, threats or abuse, and five-finger discounts shouldn’t be the norm. But a legitimate problem doesn’t automatically produce a good solution.
The gates have raised concerns for wheelchair users, people with prams and guide dogs, while parents have reported children being struck as they close. Woolworths has said an alternative entrance can be provided for people who can’t use them.
That solution asks the customer to identify themselves, find a staff member, and request a different way into the store. Access remains available on paper, but the lived experience has changed enormously.
Of course, most people will keep shopping. They’ll manoeuvre through without too much friction. They’ll buy groceries, and the transaction will appear in the corporate data exactly as it did before.
But the fact that people are still buying food is hardly a ringing endorsement of the architecture surrounding the experience.
Organisations often read repeat use as brand love. But people keep using services for all sorts of unromantic reasons: the service is essential, the alternatives are no better, or there is no genuine choice at all. Continued use is a behaviour – not an exemplary experience.
This creates what I call the “tolerance trap”. Organisations become very good at discovering how much friction can be added before behaviour finally changes. I’m confident that AI will make us exceptionally good at pushing that threshold back even further.
Surely our ambition should extend beyond discovering exactly how unpleasant something can become before behaviour finally changes.
At Knowable Me, people regularly describe accessibility that might exist on paper but doesn’t translate to the real world. But they find another entrance, bring someone with them, avoid the busiest time – or wait until they have enough energy to try again.
Either way, the organisation notches up a completed task and records a success.
The canary returned to the mine the following week. Management recorded that all concerns had been resolved.
The people the system never sees
Of course, not everyone keeps going. Some people encounter a barrier, look at the effort required to challenge it, and decide that their limited energy would be better spent elsewhere. That choice is also behaviour. It’s just harder for an organisation to see.
Last year, Knowable Me worked with Guide Dogs NSW/ACT to research the experiences of people with disability navigating online recruitment. The Breaking Barriers report included this quote from an interviewee:
“You can try to fight it, but it takes months and you still don’t have a job at the end of it.”
For me, this quote perfectly encapsulates the absurdity of placing the burden of improvement on the person the system has already excluded.
A jobseeker encounters an inaccessible application platform. They can explain the barrier, request an alternative, follow up, and perhaps make a formal complaint. But they still won’t have a job at the end of it.
One blind participant described knowing that most jobs they were qualified for would never consider them, so they just applied where they thought they might have a chance.
Their ambition was being edited by their lived experience.
An algorithm can only learn from the roles people apply for. If experience has already taught someone which doors are likely to be closed, that pattern may look like preference when it’s actually constraint.
From the employer’s perspective, there’s no event recorded at all. An incomplete application doesn’t appear in the shortlist. Someone who leaves after encountering an inaccessible careers page isn’t recorded as lost talent – or as a system that needs improvement.
The data an organisation gets is produced by the people who successfully entered and remained within its process. The inaccessible form removes the applicant, then leaves no record of its own role in removing them. A complaints process can discourage complaints and then create reassuring evidence that few people have a problem. A workplace where disclosure feels unsafe can record very little disability and conclude that it’s not especially relevant.
The system makes people disappear – then uses their disappearance to reframe reality.
And that brings us back to AI. AI inherits and works with our current reality – including the parts that are plain wrong. It learns the biases of our institutional processes, accepts all the gaps in our data flows, and incorporates our preferences and prejudices into a new reality.
Knowable Me’s analysis of publicly available information across the ASX200 found that disability was largely absent from the organisational picture. Seven in 10 of our 200 leading companies make no public reference to disability. People with disability are not absent from those companies. They are employees, candidates, customers and shareholders. What is missing is a clear public account of how organisations understand disability, measure their progress, or plan to improve.
This is why “who is missing?” is such an important question. Not only which group is underrepresented, but who never reached the point where representation became possible.
The canary was absent from the mine’s records. Mining continues.
When the process becomes the performance
This fallible system also turns the people working inside it into a process.
Most employees know the annual ritual of proving that they’ve been doing their jobs. A year of complicated, collaborative, and sometimes invisible work is excavated from calendars and Teams chats, and translated via their organisation’s approved template.
This process rewards work that is visible, easily counted, and close to authority. A presentation to a leadership team is a clear win; head-down action preventing a problem becoming a crisis, less so. Supporting a struggling colleague, retaining institutional knowledge, or making sure a project doesn’t exclude half its intended users often don’t get a look-in either.
Visibility is not value, but it is much easier to score.
While some employees are better equipped or positioned to narrate their own importance, others create value by preventing problems, strengthening the work of those around them, or contributing in ways that are difficult to record or claim as their own.
Then there are the employees already managing disability, chronic illness, caring responsibilities, or inaccessible systems, who must overcome their workplaces before they can contribute to them. The performance system records only the contribution – while totally ignoring the cost.
Then comes calibration. The organisation may have had an extraordinary year. Revenue has increased, projects were delivered, and the executive uses the word “outstanding” in its reports. Yet somehow, most of the people who produced those results are still rated three out of five.
Such a rating may have real consequences for pay and progression, but it is still only an institutional judgment. It reflects what the system recognised, how contributions were made visible, and the outcomes the organisation targeted. It was never capable of evaluating the abilities – let alone the commitment – of the person receiving it.
Goodhart’s law comes into meaningful play here too: when a measure becomes a target, it stops being a good measure. The metric starts shaping the activity it was meant to observe.
A customer-service team measured on call duration shortens calls. A recruitment team measured on hires moves faster. Employees assessed through reported achievements devote more time to documenting them.
A process starts as an attempt to understand reality – but soon, reality is realigned to satisfy the process.
The canary spent the morning documenting the value of its singing. No singing occurred.
I’ve seen a version of this with organisations trying to apply Agile methodology to operations teams.
Agile began by valuing people and interactions over processes and tools. But in some organisations, the rituals have proved more durable than the principle.
The stand-up happens because Agile teams have stand-ups. Cards move between columns because Agile teams move cards between columns. There are sprints, showcases, and enough sticky notes to destabilise the global paper supply.
Everyone performs the method. The harder question is whether the method helps anyone do better work.
Operational work often involves ongoing service delivery, unpredictable demands, regulations and relationships that can’t be packaged neatly into a two-week sprint. Sensible people adapt the method. Less sensible systems insist that people adapt instead.
I use the term “good sheeple” affectionately. Most people quickly learn that refusing organisational rituals is not the path to career progress. If a meeting is mandatory, they attend. If a template must be completed, they complete it. Challenging the norm consumes energy and can earn you a reputation as “not a team player”.
However, this unquestioning compliance also has a dark side, when it becomes evidence that a process is useful.
The same problem can emerge around inclusion. A workshop is held, ideas collected, and photographs taken of the butcher’s paper. Yet the budget, design or central decision may already be fixed. People are invited to contribute without being given any real influence to change the outcome.
Australia also has an uncomfortable productivity problem. I’m not claiming that stand-ups and performance reviews explain the national figures; economies are slightly more complicated than that. But if people are working more without creating proportionately more value, it’s reasonable to ask what fills those hours.
How much time goes into work about work: documenting, analysing, reporting and validating? And how much expertise is spent retrofitting decisions that could have been better informed at the beginning?
Meetings, reporting and documentation can create genuine value. A strong performance conversation helps someone grow. A useful retrospective changes how a team works. Consultation exposes assumptions before they become expensive mistakes.
The question is whether the activity still serves that purpose – or whether the purpose now serves the activity.
We keep adding processes to prove that people are productive, collaborative, accountable and included. Then we wonder why they have less time to produce, collaborate, exercise judgment, or meaningfully include anyone.
The canary provided feedback on mine safety. Management thanked it for its valuable contribution and continued digging.
Nobody designed it this way
It’s pretty clear that nobody ever sat down and designed ‘the whole thing’.
There was no big meeting at which executives agreed to create an expensive, frustrating process that would exhaust customers, occupy employees, and produce little useful information.
It accumulated over time, and many, many iterations.
Someone added a step because a mistake happened. Someone else added an approval process because the organisation was called out on something. Legal needed a safeguard, finance needed a number, and operations needed consistency. Procurement needed to use the system to perform 16 additional impressive functions.
And of course, processes are much easier to add than remove. We are natural builders, and addition gives us something tangible to point to. Taking something away requires a deeper analysis of why it exists, what has grown around it, and whether the remaining system can carry the load.
So the old process stays. The new one is added beside it. A workaround appears for the people who can’t use either, then develops its own approval and reporting requirements.
Before long, the process resembles my drawer full of charging cables. Each item is apparently important, but I’m not completely sure what half of them connect to, and throwing anything away feels recklessly premature.
The accessibility world is full of such experiences. A recruitment team needs consistent candidate information and adopts a standard platform. Procurement, legal, technology, and human resources each complete their part. Then a blind applicant discovers the platform doesn’t work with a screen reader.
Who owns that outcome?
Accountability is divided neatly along functional lines. Humans don’t experience procurement, legal, technology, and customer service as separate departments. They experience one organisation.
The person who sees the whole problem is often the person with the least authority to fix it. A frontline employee can hear the frustration but can’t alter the policy. A recruiter can offer sympathy but can’t repair the platform. A manager may know the performance framework produces nonsense, but they still have to submit ratings by Friday.
Michael Lipsky called this “street-level bureaucracy”: frontline workers translating broad policies into decisions about real people while managing workload, ambiguity, and limited resources. Their judgment and coping strategies often determine the service people actually receive.
Thankfully, discretion sometimes saves the human from the process. And often we describe that as exceptional service. A staff member ignores an unhelpful process step, accepts evidence in another format, or spends longer than the handling target allows.
Their workaround makes the experience possible, but it can also hide the need for structural change. The organisation records a successful outcome and never sees the small act of disobedience required to produce it.
Heroic individuals are not a humane operating model. They absorb the tension between what the process requires and what the person needs – usually without enough time, authority, or institutional protection.
The expertise that could prevent these problems in the first place is invited as a reaction to a problem, and far too late. The accessibility expert sees the platform after procurement. The customer research is done once the business case, technology, and launch date are fixed. The frontline employee is asked for feedback after the process steps have been agreed by people who will never actually do them.
People then keep the system functioning by compensating. They remember the unofficial contact person, maintain a private spreadsheet, or create a parallel process for urgent cases. Eventually the workaround becomes so familiar that nobody recognises it as evidence of a broken process. It is simply “how things are done here”.
Then the organisation introduces AI to make the processes more efficient by understanding these patterns of behaviour. There will be changes and tweaks everywhere, while the organisation’s deepest – and often flawed – assumptions remain untouched.
AI didn’t decide which evidence Australia Post would accept. It didn’t create the inaccessible recruitment platform or the performance framework that rewards visible contributions. Those decisions were already embedded.
Efficiency doesn’t bring us closer to humans when the process it was handed was already too far away from them. It simply increases the distance more efficiently.
But I believe the fact that these systems grew through human decisions is also a reason for optimism. They can be changed through human decisions. We need the humans to bring the critical thinking and ask the questions like my five-year-old self: Why? But why? Ok, but why?
What AI is actually optimising
The mine replaced the canary with AI. The dashboard stayed green.
AI is brilliant at working with representations of the world. That distinction is easy to lose because its outputs arrive so fluently.
Large organisations can’t operate on the full complexity of every human experience. They have to simplify. Political scientist James C. Scott described this as making the world “legible”: translating complexity into categories, records and measurements that institutions can see and act upon.
AI works inside this legible world. It encounters the data, labels and outcomes produced by the systems we have already built. It doesn’t encounter reality in the wild – but the traces left by the people who’ve made it far enough through our processes to be recorded.
AI can compare volumes of information no human could hold, find patterns we would miss, and apply a method with a consistency we rarely achieve. But it can’t objectively decide whether the recorded world is a faithful account of reality, whether patterns should determine a decision, or what we owe the people absent from it.
These are judgments about purpose, knowledge and value – judgments that we make before the technology begins, often without recognising that we’ve made them at all.
Again, once a measure becomes a target, people and systems adapt around it. AI gives that optimisation considerably more horsepower. It needs an objective it can recognise, which means that complicated human ambitions become a proxy.
The proxy may be sensible. It may be the best option. But it is still a proxy.
Optimising it can improve the real outcome for a while. Push far enough, and the model can become increasingly successful according to the measure – while moving further from the purpose the measure was designed to serve.
The danger is that AI may understand our instruction too perfectly.
It finds the most efficient way to produce the target, applies the rules consistently, and strips away the awkward human discretion that once softened the system around the edges.
The clever machine isn’t being rebellious. It’s being obedient.
We gave it the history available to us, defined success using the outcomes our systems had already produced, and asked it to do more of what appeared to work. By making those assumptions faster and more consistent, its confidence becomes our confidence. A score feels more objective than a judgment – even when that score contains every judgment involved in choosing the data, defining the categories, and deciding what success should look like.
Humans are inconsistent, biased, distracted, and occasionally committed to an idea long after the available evidence has proved there are better options. Algorithms on the other hand can reduce noise and expose patterns we would otherwise miss. Their value is real.
The risk comes from giving an incomplete representation of reality the authority of a complete one.
Adding a human at the end doesn’t repair the problem when that person sees the same partial information, works under the same target, and lacks the authority to challenge the result. They become the signature attached to the decision. The Australia Post employee who sent the investigation outcome was a human in the loop. My wall remained damaged.
Human judgment matters most where the data and the experience diverge. Contradiction should make the system more curious – not more certain.
AI can tell us an extraordinary amount about the world our systems have recorded. It can’t recover the world they failed to record, decide whose knowledge deserves more weight, or tell us whether an efficient outcome is a good one.
That responsibility remains ours.
We choose what becomes legible. We decide what the proxy stands for and how much authority it receives. We decide whether an unusual account is noise to be removed or knowledge that changes our understanding of the pattern.
AI can extend human judgment. It can’t absolve us of having any.
The question that matters now is whether humans will keep on exercising judgment once AI makes the system’s version of reality feel complete.
Follow the canary out
The canary was never there to optimise the mine. It was there to tell us something the mine could not tell us about itself.
I still want process, measurement, data and AI. I run a research company. I like evidence, useful systems, and clear decisions.
Rigour matters because it keeps us close to purpose. It fails when the process becomes the purpose.
When an organisation has finite resources, risks must be managed, and considered decisions will sometimes disappoint people. The challenge is to keep examining what the system produces, rather than allowing the existence of a process to end that thinking.
Workarounds are a good place to start: the employee who knows which rule to bend, or the customer who’s created an elaborate method to get through a process. I’m sure you’ve heard someone provide useful advice like: “Just yell swearwords at the automated phone system and it will put you through to a person.”
That there is gold design intelligence.
We often reward the resourcefulness of workarounds and leave the system untouched. Resilience is sometimes the compliment we give people for surviving work we shouldn’t have required from them.
But a better organisation makes those experiences visible, and gives someone enough authority to act on them. Repeated exceptions need a pathway into process review. Subject matter expertise needs to arrive while the problem, constraints and technology can still be changed. People invited to participate deserve honesty about what they can influence.
We also need more curiosity about absence: the people who never enter, stop halfway, or stay silent because speaking up has achieved nothing for them – or find a workaround so effective that the organisation never witnesses their barrier.
AI can help identify unusual patterns, repeated exceptions, and groups experiencing different outcomes. The opportunity is to ask questions worthy of that capability: Who never got in, and why? Did anything change after they complained? Was their complaint resolved – and if not, why not?
Better questions begin with humility about what our systems know. Metrics simplify reality so that we – and our AI – can see patterns. Our human experience reveals what the simplification compressed. The relationship should work both ways.
I didn’t write it, but I’ve said for a long time:
“We do the right things because they create value. The more value we create, the greater our capacity to do the right things.”
I still believe this completely.
Better experiences create value. They build trust, reduce wasted effort, and help organisations reach people that their existing processes have excluded. Employees spend more time contributing and less time proving, navigating and compensating. Risks appear earlier, before they become expensive failures.
The commercial and human arguments are the same when you look at value creation.
Still, every decent human decision shouldn’t need advance permission from a spreadsheet. Some things are worth changing because ordinary life shouldn’t require this much effort.
Much of our lives are lived in deliveries, transactions, applications, performance conversations, appointments, passwords, form filling, and small exchanges with people trying to do their jobs. These moments accumulate: a little more effort, a little less independence, another request to prove what happened, another “successful” process.
None of this should ever be inevitable.
The systems around us were made by people – usually people doing the best they could with the knowledge, authority, time and tools available to them.
I believe that is a reason for optimism. What people made, people can change.
Most of us won’t have the authority to dismantle the whole mine, and waiting until we do is a particularly effective way of changing nothing. We can question the unnecessary step directly in front of us, ask why the workaround exists, invite the right knowledge earlier, and give someone enough discretion to respond to reality.
We can notice when participation is being mistaken for influence, completion for success, or silence for satisfaction.
AI may help us predict exactly how much people will endure before their behaviour changes. But surely we can find a more ambitious use for it.
The way out doesn’t begin with a perfect new framework. It begins when someone stops, listens, and accepts that the canary may know something the miners don’t.
