IT leaders now recognize AI expertise as essential to organizational success, but a new analysis published by InformationWeek reveals that such knowledge can't be transferred from one team to another. This creates a major problem that appears destined to worsen over time, forming a new and complicated form of technical debt that traditional documentation methods can't solve.
Traditional IT systems rely on documented logic such as code, configurations, workflows, and system architecture, while AI expertise is far more context-dependent, according to Jamie Dearnley, CTO at Resolver, an enterprise risk and incident management software company. With conventional software, the code itself is the truth, explained Guy Maliar, CTO at AI developer Latent AI. A new engineer can examine the code and reconstruct why the system behaves the way it does, because the behavior is deterministic and it's all visible. AI breaks this model entirely. The behavior doesn't really exist in the code—it emerges from the model, the prompts, the data, and thousands of small judgment calls the team made along the way. Almost none of this gets written down, Maliar noted, and since the system is non-deterministic, you can't just read it and predict what it'll do. Instead, you have to rebuild a feel for its quirks by watching the system work and fail.
The report identifies the "black box" problem as a core challenge: the inability to see or explain exactly how an AI system processes inputs to reach a specific output. In complex models like deep neural networks, billions of mathematical interactions create decisions that are virtually impossible for human developers to manually trace or interpret. Organizations are accumulating a dangerous type of technical debt, Maliar warned. The system runs fine right up until it doesn't—a model update, some data drift, an input nobody anticipated—and at that point no one has the context to figure out what went wrong. According to Prateek Mishra, CTO at AI-focused recruitment firm Joveo, the biggest risk isn't that AI makes mistakes—it's that organizations don't know why it made them. "If users can't understand the decisions, trust breaks down immediately," he said.
The report recommends treating hard-won empirical knowledge as a real deliverable, not a byproduct. In practice, this means keeping versioned evaluation sets that capture what "correct" looks like and maintaining decision traces so successors can see the reasoning behind the design. The evaluation sets matter most, since they'll let a new team confirm they haven't broken anything, even before they fully understand it. For AI, knowledge transfer is less about explaining the system and more about preserving the ability to test it. Sanjay Kukreja, CTO at business process management firm eClerx, advised creating an AI system inventory that identifies which models, agents, datasets, prompts, tools, and business processes exist, as well as who owns them and what decisions or actions they support. The analysis warns that without this approach, organizations become dependent on something they can neither fix nor safely change, pushing them toward one of two bad outcomes: too scared to touch it or forced to rip it out and rebuild because understanding it costs more than starting over.