December 2016
By Laura Bellamy

Image: © Axel Bueckert/123rf.com

Laura Bellamy is the Director of Information Experience at VMware, Inc. and manages a global team of writers, editors, and architects. Laura has 15 years of industry experience as a technical writer and information architect. She has worked extensively in XML authoring technologies, is a co-author of the book DITA Best Practices, and a technical writing instructor at the University of California Santa Cruz Extension.


lbellamy[at]vmware.com
www.vmware.com


 


 

Tips for getting started with metrics


  • Start with a few key metrics and grow from there.
    Too much data can be overwhelming. Limit the data that you
    collect so your team can adapt to the strategy.

  • Design for automation in mind.

    It takes time to
    collect data and present it in effective ways. As you are defining your
    metrics strategy, try to automate the collection and presentation of
    data as much as possible.

  • Start with existing capabilities and grow from there.
    It might not be necessary to start a new IT project to begin
    collecting data. Evaluate the functionality of your existing tools. Can
    you use SQL queries in your current database to gather data? Can you use
    your company’s default web analytics tool?

  • Socialize the strategy with your teams.
    Your team
    might feel that metrics are being used to measure and judge their work.
    Discuss your metrics strategy with the team so they understand the
    value of the data and how they can use the data to do more impactful
    work.
  • Report out key insights to show the value of your
    content.

    Metrics can help you make decisions that improve
    customer success and demonstrate the value of your organization. Be sure
    to collect and share data that shows your use of the data and how it
    improves business outcomes.

The golden triangle of content metrics

Feedback can be a gift and an asset that content-producing teams can use to demonstrate their value and improve the success of their content. But how can we measure the value of our content?

As technical writers and content producers, we provide information that serves our customers so they can successfully use our products and services. To produce effective information, we rely on customer feedback. Too often the feedback that we receive focuses on a single channel of information or a limited set of metrics. This narrow view limits our ability to make effective decisions.

Discussions of metrics and customer feedback often revolve around customer satisfaction (CSAT) and net promoter score (NPS). As a content-producing organization, you might be unable to get a single metric that shows satisfaction, let alone be able to use that metric to make data-driven decisions for your organization.

As opposed to focusing on a single metric or a single channel of customer feedback, consider expanding the scope of customer feedback data that you gather to include data across three dimensions: customer engagement, content value, and market reach. The data from these dimensions creates a golden triangle of customer feedback that will help you to better demonstrate the value of your information, focus your resources on the most impactful areas of work, and better serve your customers.

Metrics strategy

Just as you have a content strategy, you should also create a metrics strategy that answers three key questions:

  • What do you want to know?
  • How will you measure it?
  • How will the information be actionable?

A metrics strategy can help prioritize what data you collect and ensure that the information will help both managers and technical writers to make decisions that improve the success of your content. The data from a well-defined metrics strategy can provide actionable information down to your writing team as well as provide valuable business information that you can provide to your management team.

Avoid false metrics

If you are just getting started defining your strategy and capturing data, you might be tempted to center your strategy around metrics that are easy to produce, such as:

  • The number of topics, pages, or deliverables that you create
  • The number of topics, pages, or deliverables pages that you publish
  • The number of languages you support
  • The number of output formats that you support
  • The degree of reuse within your content

These internal metrics explain what your team has produced. This could be effective information if you are trying to evaluate the internal productivity of your team. This data also helps to show how you are using resources and spending your budget to produce content. However, this information does not answer whether your content is of value to customers.

Customer engagement metrics

Customer engagement metrics answer the question, "How are customers using your content?" These metrics are the easiest to gather because you can quickly measure how customers access and interact with your content.

How to get metrics

If you publish content to a website that is publicly accessible, use a web analytics tool to gather customer access data. Your corporate web or marketing team might have an existing analytics tool that you can use. Otherwise, consider using free tools such as Google Analytics or Piwik.

If you do not publish publicly accessible HTML, consider other ways you can evaluate how often customers access your content. For example:

  • Number of downloads of a PDF
  • Number of documentation downloads from the app store

What to measure

Web analytics tools provide such a wide range of information that it can be difficult to decide what to track. For measuring customer engagement, consider tracking the basic metrics listed in Table 1:

 

Question

Metrics to measure

Uses for this information

How many customers access my content?

 

  • Sessions – The number of times that users access and engage with your site
  • Total users – The total number of visitors to your site
  • Unique users – The number of visitors that have visited your site once
  • Returning users – The number of visitors that have visited your site more than once
  • Page views – The total number of pages viewed on your site

 

Use this information as a data point for determining how wide your customer reach is. For example, if you have a specialized product that serves niche customers, you might expect only a few hundred users to access this content. If your content is targeted toward a mass-market audience and you do not see a large number of views, you might need to address SEO or marketing issues to ensure your content reaches your customers.

How long are customers engaging with my site?

 

  • Pages per session – The average number of pages viewed per session
  • Session duration – The average length of a session
  • Bounce rate – The number of visits where users left the site before engaging with a page

 

Use this information to determine the most popular and the most engaging pages so you can apply positive characteristics from these pages to other content. For example, if your topics with code samples are widely viewed and have a low bounce rate, consider creating more code samples and adding samples to other pages.

 Table 1: Basic metrics for measuring customer engagement

 

Content value metrics

Content value metrics answer the question, "Do customers find your content useful?"

As the Nielsen Norman Group showed, the core characteristic of a good user experience is utility – meaning that the content is useful to customers. You can invest a great deal of effort creating perfect, polished content that doesn’t meet customer needs.

A well-defined metrics strategy that includes value metrics will help you to determine the usefulness of your content as well as the usability of the information.


Figure 1: Nielsen Norman Group user experience characteristics

How to get metrics

  •   Number or star ratings
  • Like and upvote systems
  • Yes/No questions
  • Surveys and feedback forms
  • Comments
  • Email

Be aware that feedback mechanisms range in technical complexity. Adding an email address to your site is easy, while setting up a database and notifications for your rating mechanism can be more difficult to implement.

What to measure

Because there are so many different feedback mechanisms, defining the goal of your value metric is critical. Defining the goal might help you to choose which mechanism to implement, see Table 2.

Goal

Recommended feedback mechanism

Receive feedback about errors in the content

Email or feedback form – A private message that is directly routed to your team ensures that you receive the actionable information. You can track the number of emails you receive or forms submitted.

Track social engagement of your content

Comments – Commenting can create a dialog for your user community. You can track metrics related to comments including number of users that comment, number of comments received, number of threaded comments, etc.

Determine customer satisfaction

Survey – Ask a small set of questions to determine how satisfied customers are with the content and your site.

Understand utility of content

 

  • Number or star ratings – Give customers a range to rate the value of your content on a fixed scale. Measure the average ratings of your content.
  • Like – Allow customers to Like or Upvote content that they find useful and to Downvote content that is ineffective. Measure the number of Likes for your content.
  • Yes/ No question – Provide a direct question for users to specify if they find content helpful or not. For users who answer No, provide a follow-up question to gather actionable information. Measure the number of users who find your content helpful.

 

Table 2: To collect value metrics, define your goal first before deciding on the feedback mechanism.

Market reach metrics

Market reach metrics answer the question, "Are you effectively reaching your customers?"

How to get metrics

While some of this information is provided from web analytics, you might need to contact your sales, marketing, or support organizations to gather additional data. Contact other parts of your company to understand the potential customer base for your content.

What to measure

Your sales or support organization might be able to answer the following questions:

  • How many customers have purchased the product or service?
  • How many customers have downloaded the product or service?
  • How many customers have installed or activated the product or service?
  • What are the key accounts or users for the product or service?
  • What global regions and languages have the most users?

Market reach data is essential for you to determine if your customer engagement is successful. Your products and business units could vary widely and understanding market reach helps to avoid an apples-to-oranges comparison. For example, consider what success looks like for the following two products:

  • End-user SaaS product – For a new Software-as-a-Service product with a single market focus, the expected adoption might be slow growth among a small, key customer base.
  • Enterprise installed product – For a mature, enterprise product that serves multiple markets, the expected adoption might be rapid growth among a majority of the existing user population.

Correlate data to create the golden triangle

When you have data from the three key dimensions, you can correlate the information to gather a holistic view of the content and to make insightful observations and decisions.

Figure 2: The triangle of customer metrics

 

The triangle answers three basic questions:

  • Engagement – How do customers use your content?
  • Value – Do customers find your content useful?
  • Reach – Are you effectively reaching customers?

Example: Evaluate the success of an effort

Consider how the data from the triangle helps a manager to determine how successful the effort was for two projects, see Table 3.

Project description

Evaluation

Scenario 1

  • Product has only a handful of customers
  • Two technical writers work for six months to create content
  • The 100 pages are all highly rated
  • Documentation is important for a key account

 

Successful – Although this product has a limited customer set and very few users, delivering the documentation to a key customer helped to serve a key customer account. This effort was highly impactful for the business.

 

Scenario 2

  • Mass-market product has a large number of customers
  • One technical writer produces a video series that is viewed by only 2,000 users
  • Most viewers don’t finish watching the video

 

Not successful – It was costly to produce the video series and the result of only 2,000 views is not a significant number for this customer base. The cost of this project did not return the investment to the business.

Table 3: Measuring the success of a project

 

Example: Make forward-looking decisions

 

The data from the triangle can also be used to evaluate trends and make decisions for upcoming projects and resource decisions. This is especially useful when you consider the opportunity cost of a project. By focusing your resources on one project, you are unable to complete another project. Use your customer data to help determine which projects you should prioritize, see Table 4.

Project description

Proposed decision

Scenario 1

  • Three topics are rated as poor by customers
  • Topic A has 100 page views
  • Topic B has 2,000 page views
  • Topic C has 10,000 page views

 

 

  • Priority #1 – Improve Topic C because it will benefit the most customers.
  • Priority #2 – Improve Topic B because it reaches a limited set of customers.
  • Do not update Topic A. It might be poorly rated, but your investment will have little impact.

 

Scenario 2

  • Should you document a new feature for an upcoming release, or incorporate bug fixes to existing topics?
  • Sales data shows that customers don’t download the product until three months after the release
  • There are a lot of negative comments and ratings that show poor quality in the current release

 

 

  • Priority #1 – Complete the bug fixes and quality improvements. Helping to improve the content that customers are currently using will improve CSAT.
  • Priority #2 – Document the new feature after the bug fixes are complete and deliver it within three months of the release so customers have the content when they use the new version.

 

Table 4: Using customer data to determine which projects to prioritize

 

Conclusion

Defining a metrics strategy is important because it focuses your effort and ensures that you have actionable data. Documenting your strategy is also important because this effort might involve multiple teams, such as Marketing, Sales, and IT, which need to understand your requirements and goals.

By using data from multiple dimensions and correlating the data, you have a comprehensive view of your content and can use insights to improve the success of your content.