The Hidden Risk of AI Automation: Losing the Human Experts Who Train AI
As artificial intelligence systems increasingly take on knowledge work tasks, a critical and often overlooked risk emerges: the very experts that AI needs to learn from are being replaced by automation. Entry-level jobs that traditionally built expertise—document review, data cleaning, code review—are now handled by models. This creates a dangerous feedback loop where future experts cannot develop the judgment required to evaluate and improve AI systems. Below, we explore this growing concern through key questions.
1. Why is reliance on AI for knowledge work creating an enterprise risk?
AI systems in knowledge domains rely on two pathways for improvement: autonomous self-improvement or high-quality human feedback. The industry has invested heavily in the first, but the second is neglected. As AI automates entry-level tasks, the pipeline of junior professionals who learn by doing these tasks is shrinking. These workers are essential for catching errors and providing nuanced feedback that keeps AI systems accurate and up-to-date. Without them, the quality of human evaluation declines, leading to potential stagnation or errors in AI performance. This risk is not captured in traditional risk models, which focus on technical failures rather than the erosion of human expertise.
2. What limits self-improvement in AI for knowledge work compared to games like Go?
AI self-improvement works superbly in domains like Go or chess because they have stable environments and unambiguous reward signals. The rules are fixed, and outcomes (win/lose) are immediate and clear. In contrast, knowledge work operates in dynamic environments where rules—laws, financial instruments, medical guidelines—change constantly. A legal strategy that was effective last year may fail due to a new court interpretation. A medical diagnosis's accuracy might not be known for years. Without a closed feedback loop, AI cannot reliably improve through self-play. Human evaluators are necessary to provide context and judgment, but they are becoming scarce.
3. How does the lack of stable environments and reward signals affect AI learning?
In stable environments, AI systems can use reinforcement learning to explore vast state spaces and receive immediate, perfect feedback. For example, AlphaZero's famous "Move 37" emerged from self-play because the game ends decisively. Knowledge domains lack this: a software engineer's code might pass tests but fail in production years later. A financial model might appear profitable until a market crash reveals hidden flaws. Without clear, timely feedback, AI cannot self-correct. It needs humans who can judge intermediate outcomes and provide guidance. But if those humans are not being trained because entry-level tasks are automated, the feedback loop weakens.
4. What is the "formation problem" and why is it concerning?
The formation problem refers to the fact that current AI systems were trained on data from experts who developed deep judgment through years of entry-level work. Today, those same entry-level jobs—document review, first-pass research, data cleaning—are among the first to be automated. This means new professionals never acquire the tacit knowledge and critical thinking that comes from hands-on experience. They become less capable evaluators and trainers for AI. Over time, the pool of high-quality human evaluators shrinks, leading to a degradation in AI training data and model performance. No catastrophic event is needed for this to happen; it is a gradual erosion from rational cost-cutting decisions.
5. How does automation of entry-level jobs threaten future expertise?
Entry-level positions are where professionals learn the fundamentals: how to spot errors, evaluate nuances, and make judgments under uncertainty. By automating these roles, companies gain short-term efficiency but lose the long-term ability to develop experts. For instance, a junior lawyer reviewing contracts learns to identify subtle clauses that could cause issues. If AI does that task, the lawyer never builds that skill. Similarly, junior radiologists training by reviewing thousands of scans develop pattern recognition; if AI handles initial reads, that experience is lost. The next generation of experts will have less practical judgment, making them less effective at overseeing and improving AI systems.
6. What historical examples illustrate knowledge loss, and how is today different?
History is replete with lost knowledge: Roman concrete, Gothic construction techniques, and mathematical traditions that took centuries to recover. However, these losses resulted from external catastrophes like plagues, conquests, or institutional collapses. Today's threat is different. Knowledge may atrophy not from a single disaster but from thousands of individually rational economic decisions. Each company's decision to automate entry-level roles makes sense in isolation, but cumulatively, it eliminates the human foundation needed to maintain and advance expertise in entire fields. This is a silent, systematic erosion rather than a dramatic event.
7. What should companies do to address this risk?
Organizations must recognize that human evaluation capacity is a strategic resource, not a cost to minimize. They should invest in training programs that preserve hands-on learning even as automation increases. For example, require that junior staff be involved in verifying AI outputs, not just relying on automated metrics. Create feedback systems where human evaluators are continuously developed and rewarded. Companies should also model the long-term risk of losing expertise, just as they model technical risks. This means treating human-in-the-loop as a critical infrastructure that requires deliberate nurturing. Failure to act could lead to a future where AI systems plateau or produce increasingly flawed results due to lack of quality human guidance.