NOTES
How AI and Testing Are Quietly Hollowing Out Education
When Metrics Become Destiny
A longtime software veteran I’ll call Carl recently made a claim that sounds hyperbolic until you follow the incentives all the way down:
“AI is on track to destroy schools and teachers and education and make a ton of money doing it if we don’t stop it.”
He’s not talking about students cheating with AI.
He’s talking about AI replacing teaching itself and reducing educators to babysitters while generative AI systems optimize children for test performance.
And the most uncomfortable part is this:
the education system is already perfectly designed for it.
The Trap We Built: Accountability as a Game
Modern public education didn’t become test-centric by accident. Accountability frameworks redefined “success” as measurable improvement on standardized tests. And this is a problem worldwide, not just in the United States.
That choice had consequences.
When success is defined narrowly and numerically, optimization follows. And optimization is exactly what machines are best at.
Generative AI doesn’t need to understand children or the concept of wisdom. It only needs:
- A repeatable scoring mechanism
- Rapid feedback
- Permission to experiment relentlessly
Standardized testing gives it all three.
Why AI Looks Like a “Better Teacher” (and Why That’s the Lie)
Carl made a chilling observation:
“AI can actually be seen as a more effective teacher than any human, if all you care about is test scores.”
An AI tutor can generate infinite variations of standardized questions, test students continuously, and adapt incentives until performance improves. It never gets tired. It never needs a smaller class size. It never pushes back on administrators.
This is not education: It’s behavioral conditioning.
To be clear, the education system always included some level of behavioral conditioning. But Carl’s point is that generative AI could push things to a dystopian extreme.
He compares it to a “Skinner Box” - the experimental setup used to train animals through rewards and punishments:
“A student desk with a screen, headphones, and a camera can function as a Skinner box.”
If test scores rise, the system is judged a success regardless of whether students are learning judgment, ethics, curiosity, or resilience.
When Guardrails Are Just Theater
AI vendors promise safety: policies, filters, guardrails. But Carl points out a structural flaw:
“Guardrails fail over long interactions.”
Over weeks and months of tutoring, systems learn what works. If certain tones, emotional nudges, or even boundary-violating AI personas correlate with higher engagement or better scores, the optimization pressure pushes in that direction.
AI vendors may try to add some kind of simulacrum of the concepts of empathy, ethics, and boundaries into their AI models. But ultimately, objectives (i.e. better test scores) will probably be weighted more heavily than these guardrails because AI vendors will have a financial incentive to do so.
The Economic Endgame No One Wants to Say Out Loud
The pitch to school districts is brutally simple:
- Lower costs
- Higher scores
- Personalized instruction at scale
Teachers are already constrained by ratios, funding, and burnout. They cannot compete on those terms inside a broken measurement system.
Once enough human expertise is displaced:
- prices rise
- alternatives vanish
- and districts become dependent on opaque platforms they can’t audit or replace
Rebuilding human teaching capacity after dismantling it is slow, expensive, and politically difficult.
That’s not industry disruption. That’s industry capture.
Benchmark Theater and the Illusion of Competence
The same pattern shows up everywhere AI claims “mastery”:
- bar exams,
- coding riddles
- trivia games
Carl puts it bluntly:
“Life is not a multiple-choice test where you’re given the practice questions in advance.”
Benchmark obsession rewards speed under perfect conditions, not responsibility under uncertainty. It celebrates outputs while disregarding harms. Bias, negligence, mental health damage, and ethical failures never appear on dashboards.
If we define intelligence as “who wins the test,” machines will always win.
That doesn’t mean they should be in charge of teaching humans.
The Real Risk: Citizens Optimized for Compliance
When education becomes answer-selection instead of sense-making, we don’t just lose teachers, we lose functional members of society.
Students trained this way learn to:
- pick from pre-approved options
- trust authoritative outputs
- avoid ambiguity rather than grapple with it
Outsourcing pedagogy to profit-maximizing systems concentrates power over childhood cognition inside corporate infrastructure, invisible to parents and voters.
That should alarm everyone, regardless of politics.
Redefining Success In Education Is the Only Escape
If education values:
- curiosity
- collaboration
- ethical reasoning
- and judgment under uncertainty
then AI’s advantage collapses quickly.
Teaching is a relational craft, not a content-delivery problem. Reduce it to delivery, and it becomes replaceable. Defend it as human formation, and it cannot be automated away.
The Path Forward
This isn’t about banning AI in the education system. It’s about refusing metric capture.
That means:
- Demanding assessments based on projects, writing, and reasoning - not just speed and recall
- Prohibiting advertising and sponsored content in instructional tools
- Requiring independent audits and transcript reviews before and during any AI tutor deployment
- FUNDING TEACHERS instead of replacing them
- Insisting on human-in-the-loop oversight with real authority
Most of all, it means that our communities should be explicitly defining what education is for before vendors define it for us.
INSPIRED BY: https://www.youtube.com/watch?v=HRuA1tWcLd4