Early AI: Painful observations
When generative AI entered the mainstream with the debut of ChatGPT, the industry moved quickly – almost overnight. A wave of stochastic chatbots appeared as developers combined Large Language Models (LLMs) with vector databases such as Pinecone. With modest Python skills, teams could assemble a Retrieval‑Augmented Generation (RAG) chatbot in only a few days.
The speed of adoption was remarkable, but it was also misleading. Very early on, it became clear that probabilistic retrieval layered on top of probabilistic models introduces structural weaknesses. These systems can generate plausible responses, but they struggle in environments that require accuracy, repeatability, and trust. For highly regulated industries, the difference between plausible and correct is the difference between success and operational failure.
Yet years later, much of the industry conversation still focuses on explaining RAG itself rather than examining its limitations. This disconnect has been striking. The situation was not the result of incompetence – it was the result of pressure. Development teams across industries were asked, often urgently, to “do something with AI.” Fear of missing the technological shift drove rapid experimentation and rapid deployment.
Developers reached for architectures that were fast to demonstrate. RAG pipelines were relatively easy to prototype, impressive in demos, and capable of producing convincing responses. But systems that are easy to demonstrate are not always easy to harden into reliable production infrastructure.
From the outside, many implementations appeared brittle. From the inside, however, the tradeoffs were often unavoidable. Teams used the tools available to them and delivered solutions under real organizational pressure. That pressure has not disappeared. Competitive forces and vendor marketing have only intensified it. Organizations must signal innovation, while vendors must demonstrate product momentum. In that environment, architectural limitations often persist long after they are widely understood.
The teams that recognized these issues earliest often shared a common background – formal knowledge management. Practitioners in that discipline understand that effective AI systems depend less on clever prompts and more on disciplined information architecture: structured content, explicit context, semantic relationships, and governance. In other words, reliable AI systems are built, not bolted on. The work that makes them trustworthy happens long before a model generates its first token.
The precision paradox in AI
As AI systems become more accurate, they paradoxically become less tolerant of error – a phenomenon known as the precision paradox.
Early RAG systems produced approximate answers. Because overall accuracy was relatively low, users expected occasional mistakes and ignored minor inaccuracies. The bar for performance was modest. As models improved, expectations rose, and users began to assume systems were correct by default. That shift dramatically changes how errors are perceived.
Small inaccuracies that once went unnoticed suddenly become highly visible. A single incorrect step in documentation guidance or a misinterpreted configuration parameter can immediately undermine trust. A tiny factual mistake or incorrect context fragment can break user confidence entirely, especially in technical and regulated environments.
Improving precision, therefore, has an unexpected consequence: It reduces tolerance for error. The better a system appears, the less its mistakes can be ignored.
Several implications follow. Probabilistic approaches have practical limits. Vector‑based RAG delivers broadly plausible answers but struggles to achieve deterministic accuracy. Precision requires structure and governance. Knowledge graphs, ontologies, and semantic relationships provide verifiable context that probabilistic systems lack. Perception also matters as much as performance – users judge AI systems by their worst errors, not by their average quality.
For simple support bots, this gap may be acceptable. In agentic workflows – where AI systems perform multi‑step tasks autonomously – imprecision compounds quickly and can become catastrophic.
The false hope of LLM model‑only progress
For several years, the dominant narrative suggested that the solution to AI reliability problems would come primarily from bigger and more capable models. Each generation promised more reasoning, more context, and more intelligence.
In practice, these improvements have been incremental rather than transformational. Large Language Models remain probabilistic systems. Their outputs are generated through statistical prediction rather than deterministic reasoning. No model upgrade fundamentally changes that mathematical foundation.
Large Reasoning Models illustrate this limitation clearly. Despite the name, these systems do not perform formal reasoning. Techniques such as chain‑of‑thought prompting encourage models to simulate reasoning through recursive probabilistic checks. The output can look convincing and structured, but it is not equivalent to deductive reasoning, constraint solving, or grounded inference. The result often resembles reasoning without actually being reasoning – something that sounds logical but lacks verifiable guarantees.
Large Context Models – more context, more noise
Large Context Models (LCMs) follow a similar pattern. Increasing the context window allows models to process more tokens at once, but additional text does not automatically create understanding. Instead, it increases surface area.
More tokens introduce more retrieval noise, more opportunities for irrelevant correlations, and more chances for the model to produce responses that sound coherent without actually being correct. Providing a model with enormous context is somewhat like giving a parrot a larger library. The information may be present somewhere in the pile of text, but the system has no grounded understanding of authority, validity, or contradiction.
Context without structure becomes noise at scale. Even more concerning, large context can actively mask failure. Longer responses feel richer and more thoughtful, making hallucinations or invented connective logic harder to detect.
This dynamic also has a financial dimension. Large context systems dramatically increase token consumption, shifting costs from training toward continuous inference. If models cannot become reliably smarter, the economic incentive becomes making prompts larger. But context is not cognition, and scale alone does not produce knowledge.
The uncomfortable truth
Technologies that reduce token consumption – knowledge graphs, semantic models, governed content, and external inference engines – often conflict with the economic incentives of Large Language Model providers.
These approaches move intelligence outside the model and into structured systems where reasoning can be verified and constrained. Real reasoning systems frequently require fewer tokens, not more. That tension helps explain why deterministic augmentation often receives less attention than probabilistic scaling, even though it is essential for trustworthy AI.
The early days of AI in technical documentation
For practitioners in knowledge management and information architecture, these developments were oddly familiar. For years, the documentation and knowledge management communities have advocated semantic models such as taxonomies, ontologies, controlled vocabularies, and knowledge graphs. These tools were designed to capture meaning and relationships explicitly rather than relying on implicit interpretation.
At conferences and industry events, these concepts were often dismissed as academic or overly theoretical. Many organizations technically employed knowledge management specialists, but rarely prioritized their recommendations.
The rise of generative AI changed that conversation almost overnight. Suddenly, everyone was discussing semantic meaning, provenance, context, determinism, reasoning, and inference. The limitations of purely probabilistic approaches became obvious once organizations attempted to deploy AI in real operational environments.
Accuracy and trust emerged as the true constraints. Agentic workflows – systems where AI performs multi‑step tasks autonomously – cannot operate reliably on surface plausibility alone. They require grounded context and deterministic validation.
How the technical documentation community responded
The initial reaction within the technical documentation community was predictable: panic. Many writers feared that generative AI would eliminate documentation roles entirely. In the moment ChatGPT appeared, this fear was premature. The technology was not mature enough to replace professional technical communication. Nevertheless, the anxiety was understandable.
As the panic subsided, experimentation began. Writers explored ChatGPT and quickly discovered its strengths and weaknesses. Prompt engineering became the first collective adaptation because it was accessible and delivered immediate productivity gains.
Some practitioners advanced to custom GPTs tuned for specific domains and documentation tasks. These systems improved localized workflows but did not address broader structural challenges such as governance, accuracy, and lifecycle management.
Meanwhile, Retrieval‑Augmented Generation systems required deeper engineering skills. As a result, ownership of documentation corpora often shifted toward software engineering teams. In many organizations, documentation content quietly migrated away from the documentation function and into AI engineering pipelines. Sometimes this shift happened collaboratively. In other cases, it occurred with minimal coordination. The architectural and organizational consequences of that shift are still unfolding today.
The documentation supply chain
Modern documentation organizations increasingly need end‑to‑end automation and governance across the entire content lifecycle. This includes authoritative knowledge capture, structured authoring and validation, controlled publication, continuous maintenance, and the eventual retirement of outdated content.
In short – the entire supply chain of technical knowledge.
The quiet winners
A small number of documentation organizations have already begun building this future internally. These teams possess engineering resources and approach documentation as a systems problem rather than a publishing task.
Instead of waiting for vendors to deliver complete solutions, they are building internal pipelines that encode institutional knowledge directly into automated workflows. Rather than experimenting with isolated AI features, they are engineering documentation supply chains.
The transformation is not painless. Changes of this scale rarely are. But the organizations willing to make the investment are already seeing the benefits: improved consistency, faster content maintenance, and AI systems that operate with far greater reliability.
For the technical documentation profession, this moment represents both a challenge and an opportunity. The role of the technical writer is expanding beyond authoring into knowledge engineering, system design, and governance. Those who embrace that evolution will help shape the next generation of intelligent systems.
Next
Stay tuned for Part 2 of this article series: “The real deal: Case studies of Agentic AI documentation”