“Why won’t my coffee maker turn on?” “How can I pair my device with Bluetooth?” “What does this error code mean?” – The questions your customers ask search engines and AI chatbots can help you identify documentation gaps and improve your content.
Many documentation sites have a built-in search box, and analyzing the keywords entered here can give tech writers a first indicator of what customers are searching for. Bear in mind, however, that these keywords alone do not reveal the intent of the search, and false assumptions may produce false analytics.
With the introduction of chatbots and AI search engines on many documentation sites, gone are the days when technical writers needed to guess the intent behind search keywords. Prompts entered into AI chatbots reveal the intent of the search without leaving much room for interpretation.
The purpose of analyzing search keywords and prompts is to clearly understand our customers’ questions without any assumptions or ambiguity.
Analyzing prompts
Once the intent of the search is known, tech writers can determine whether they need to tweak the content and how, or whether the prompt is irrelevant and can be ignored. Since most prompts start with interrogative words (“what”, “when”, “how”,…), understanding these helps technical writers to act smarter.
Interrogative word | Inference about customer intent | Content changes |
What | Interested in understanding some definitions and wish to have an explanation | Focus on content changes to offer more in-depth information |
Why | Trying to reason | Add content that explains a phenomenon, offers reasoning, and provides detailed analysis |
How | Interested in understanding procedures or methods | Build tutorials and step-by-step guides |
Which | Interested in choosing/evaluating | Content must be tailored to compare things so that the customer can make a choice |
Are/Is/Was/Were | Seeking confirmation | Add content on classification information, state of being |
The type of prompt or question offers a clear value proposition to technical writers by highlighting the customer intent. However, analyzing prompts along with their responses offers more insights, such as whether the chatbot or AI search delivered a helpful answer.
Gathering prompt analytics
Figure 1: Classification of prompts from AI-search
Clustering similar questions into relevant topics provides a holistic perspective of trending topics. Adding a time dimension to this topic analysis shows how the trend is evolving and the changes in customer search behavior over time. This lag indicator metric helps to understand and evaluate the uptake of new product releases by customers. It can aid the documentation and marketing teams to understand the impact of customer education and product marketing activities.
Figure 2: Topic identification using prompts clustering
Source: Document360 Eddy AI Analytics
Unanswered prompts
Any prompts that resulted in a “Sorry, I cannot help you here” response from the AI engine can reveal underlying gaps in your knowledge base. Once the unanswered questions are validated to be genuine, undertaking a clustering approach to group prompts by their topic will reveal the themes that technical writers need to focus on. Going through prompts in each clustered topic will show all the questions that have been asked on a particular topic. This offers insights into existing documentation gaps and enables technical writers to fill these gaps.
Answered prompts
Answered prompts help you to understand trending prompts. Citations are now included in modern AI-based searches, revealing highly cited articles. These articles must be kept up to date and must have a high degree of accuracy.
The answered prompts also help to quantify the value of documentation. If a customer received a response, we can assume that they have acted on it. However, if the response is inaccurate and customers provide negative feedback, then technical writers need to train the AI engine to produce accurate responses by providing ground truth to correct the AI response. In this fine-tuning process, technical writers play a crucial role in ensuring the AI engine delivers high-quality, accurate answers.
Number of conversations
Given the conversational nature of AI engines and their ability to remember previous context, it is important to track the number of prompts in a single conversation. If a conversation contains multiple prompts, it means that your customers are using the AI engine as a chatbot and are more inclined to make follow-up questions. Alternatively, if a conversation contains only one prompt, then customers are using AI-based search like a traditional search engine and may not completely understand the technological capabilities of AI engines.
Figure 3: Understanding customer behavior using conversational analytics
Source: Document360, Number of conversations
Prompt drift
Prompt drift is a phenomenon where customer prompts change over time, but the response to those prompts remains unchanged. Prompt drift shows the shift in customer persona and their preferences in using different words to find the same information. Understanding prompt drift helps technical writers to tweak content and include terms that are synonymously related to actual business terms. A typical example of prompt drift is
Where is the United States of America? → Where is the USA? → Where is America?
Product feature adoption
Analyzing all prompts pertaining to product features will provide valuable insights into how new product features are being adopted. If there are many prompts about a particular product feature, it might signal that it is harder for customers to adopt. These insights can be shared with product managers so that they can understand whether to make changes to UX, functionality, or other aspects of the product feature.
Change in terminology
Analyzing prompts for the presence of terminology or business terms reveals the impact of customer education initiatives. If customers are using synonyms to find product features rather than exact business terms, it shows that:
- Customers are not getting educated about certain product features and business terminologies. Either the product manager needs to change the product feature name or the marketing team needs to ramp up the product marketing activities.
- Customers prefer simple terms to look for information, and product feature names must align with customer preferences and industry-wide accepted terms.
Observing customer behavior
By comparing keyword-based search volumes with prompt-based search, we can understand how customers are transitioning to AI-based search. As per ResearchGate’s “Diffusion of Innovation”, about 14% of early adopters wish to try new technology. Volume data from traditional search (keywords) and AI-based search (prompts) will provide a clear picture of customer behavior. Tracking keywords based on the number of words over time also helps to measure the impact of AI search.
- If customers use more prompts than keywords in their searches, maybe it is time to switch off traditional keyword-based search.
- If keywords are appearing in prompts rather than questions, it might be time to educate your customers about how to use AI search efficiently.
- If customers search by using keywords rather than prompts in an AI-based search, your customers are still in the early stages of adopting the new technology.
Analyzing prompts for keywords such as “please” will unearth information about customer personas. Also, checking whether prompts start with a capital letter and end with a question mark reveals how they are treating your AI search engine.
Closing remarks
Prompt analytics represents a paradigm shift in how technical writers understand and respond to customer needs. By analyzing the language, intent, and patterns within these prompts, documentation teams can create more targeted, effective content that truly serves their customers. As AI-based search continues to evolve, the tech writer's role will increasingly involve interpreting these digital conversations, bridging the gap between customer queries and knowledge delivery.