May 2016
By Giuseppe Scavo and Christine Perey

Image: © www.microsoft.com

Giuseppe Scavo is a PhD candidate at the Knowledge Media Institute, the Open University, UK. He is investigating how the combination of Augmented Reality and the Internet of Things can enhance performance and training of blue-collar workers in the manufacturing industry. In conjunction with his pursuit of the keys to the future of work, he provides research services to the AR for Enterprise Alliance.


beppescavo[at]gmail.com


 


Christine Perey is the founder and principal analyst of PEREY Research & Consulting. She provides highly strategic research, business- and market-building related services with an emphasis on building robust strategies leading to the successful introduction and widespread adoption of Augmented Reality products and services.


christine[at]theAREA.org
http://theAREA.org
www.perey.com


 


 

Christine Pereyís partnership with leading firms, universities, experts and users brings a total view of the landscape in which products and services will compete and be combined for the best customer outcomes. These partnerships and collaborations led to the establishment of the AR for Enterprise Alliance, the only global, member-driven industry organization focusing on accelerating AR adoption in enterprise.


http://theAREA.org

The new frame of our enterprise information

With Augmented Reality, information is no longer restricted to a paper or screen. The focus of the future of information design will be on presenting information to users in context and using AR technologies. But what are its limitations and how can we exploit its opportunities?

Augmented Reality (AR), the delivery of information synchronized with the real world surrounding a user, provides an emerging alternative for providing information. By being able to access crucial information in context and in real time, the user can perform complex tasks more quickly and with fewer errors, even with no prior training or experience. By addressing only the current needs of the user, AR-assisted systems will evolve to offer other organizational benefits when implemented in enterprise knowledge products.

AR is the result of combining content, software, hardware and the physical world. When developing AR experiences for enterprise users, the designer must consider the strengths and limitations of the presentation systems available to the user.

AR presentation systems

AR experiences can be delivered through different devices and technologies. When choosing the best system, the userís needs, device costs and complexity as well as the environment must be taken into consideration. The devices currently available for enterprise AR can be classified according to their position in relation to the userís senses. The three categories in such a classification system are:

  • Handheld displays such as smartphones and tablets, which must be positioned between the userís senses and the real world target.
  • Head-mounted displays such as smart glasses and camera-equipped visors are already positioned near the userís senses and the user only needs to turn his gaze at the real world for the system to be aligned. Both hands are free.
  • Stationary projectors using beams of light directed on real-world objects remain in the userís field of view as long as the object and user are correctly positioned with respect to the projector and camera.

Another important and equally valid approach to classifying AR presentation systems is based on the presentation technology. In this case, we make distinctions between:

  1. Video see-through: The environment is captured by the systemís sensors (e.g. a camera) and, using computer graphics, the elements of the AR experience are blended together before being rendered on the display. This is the AR technology used on tablet screens, smartphones and some types of head-mounted displays.
  2. Optical see-through: The environment is detected by the systemís sensors, but only the augmentations are rendered while being displayed using systems of semi-transparent prisms and small projectors, so that the userís optical system integrates the images. Many smart glasses fall into this category.
  3. Projection AR: a projector displays illuminated text, images and instructional symbols directly onto real objects in the userís environment.

When designing technical documentation for AR delivery, the developer must take into account the strengths and limitations of the systems in each of these categories.

Letís take an example to illustrate this: Augmented Reality devices are particularly efficient in providing spatially registered multimodal information. However, given their limited field of view (FOV), small screen size and low resolution, it is not convenient, and thus difficult for the user, to read text blocks. Also, in order for the user to be able to read the text using a hands-free AR display, the font must be significantly larger than on a tablet or laptop (precisely how much depends on the resolution and FOV). Text also needs to be formatted to be digested quickly, as the user does not have a mouse with which to scroll.

Another important and new design consideration is depth perception and sensing. The depth perception of a user is based on millions of years of evolution using two eyes (stereo vision) and a dense system of optical nerves wired in the brain. It is fundamental to how users experience the real world. In order for digital annotations (augmentations) to appear "correctly" in the userís context, these systems must also sense and use depth information.

Augmented Reality presentation systems manage depth in different ways. Due to the stationary nature of the light source, projection AR does not support depth. However, when using optical see-through displays to blend text, graphics or even a 3D object into a userís environment, depth is critical to rapidly understanding the purpose of the experience with a low cognitive load. Some optical see-through displays have limitations if they lack stereo "vision" or alternatives to capture depth.

Those devices with only one camera ("monocular systems") are unable to accommodate differences in depth as the digital content appears in only one plane while the real environment is perceived on a wide range of different focal planes. Hence, despite other drawbacks, using video see-through technology where the userís perception is already adjusted to looking at a 2D image is recommended in cases where the perception of the spatial position of technical annotations and 3D objects is of crucial importance.

Consider the impact when an arrow or annotation that is intended for a real-world target is hidden. Or the userís confusion when it seems that the annotation is intended for another object in the scene. Using systems with depth sensing and "mapping" enables more natural AR experiences, which are able to convey to the user the relative position and distance of the target with respect to the annotation.

Additional considerations that one must take into account when designing for different presentation systems include the expected frame and refresh rates of the user system and a variety of fully mobile technology constraints. For example, when designing for hands-free presentation systems, the user has limited power and memory and, if the system is required to use a network connection for offloading content and computational tasks, the designer will need to assure continuous connectivity between the device and the network servers.

The table below compares the features of existing information delivery systems with regard to interactivity, depth support and other modalities for the three categories of AR presentation systems.

 Feature

 

 

Paper delivery

PDF

With video see-through AR

(handheld AR devices)

With optical see-through AR (hands-free AR, smart glasses)

Projection AR

Mobile

Mobile

Mobile

Mobile

Mobile

Not mobile

Interactive content

No

No

High

Low

Low

Support for depth

No

No

Yes

Yes

No

 

Support for large text

Yes

Yes

Yes

No

No

Power limitations

No

Depends

Yes

Yes

No

Memory limitations

No

No

Depends

Yes

No

Mobile connectivity

No

No

Depends

Yes

No

Table 1: Features of information delivery systems

 

Providing multimodal information

When we want to provide information assets today, thereís usually a link or a page. The content is designed to accommodate many different linear or non-linear modes of consumption and exploration. There are many links on a page and the technical writer doesnít control when or how the user looks or reads. Is he gathering information from top to bottom, from side to side, or from bottom to top?

In contrast to paper and PDFs, with Augmented Reality systems, the information provided to users is designed to be "consumed" only when it is needed and in a manner thatís closely tied with a specific real-world object. This presents a new set of challenges that designers of technical documentation will need to overcome. Systems must provide users with an intuitive way to see or use complementary information that may be of value Ė other than the AR experience. The system may detect when a user wants to change modes (no longer in AR view) or offer a way for the user to put the AR experience on "pause" while focusing on the interactive digital content.

There are other important differences between how we deliver information today and how AR works. It is widely recognized that combining text with audio, images, videos and 3D animations reduces the time to process information and improves knowledge acquisition and memory recall. Augmented Reality authoring and presentation systems allow multimodal information to be seamlessly integrated with the object of the workerís attention. An animated 3D model can float right above the real-world object to illustrate the movement the user should perform, while a side-by-side image can help a technician with quality assurance.

Background sensitivity

When designing technical documentation for Augmented Reality, itís important to remember that the user has the full world as the "background". There are many sources of distraction and conflicts that can reduce the effectiveness of the information delivered. For example, color design is an important consideration.

The choice of colors in labels and 3D content can draw attention to particular details. In addition, since AR overlays are spatially positioned in the real environment, a designer must consider the contrast of the augmentation (the overlay) with respect to the background (e.g., white text on a white background is invisible).

In the table below, we compare the features of existing information delivery systems with the three categories of AR presentation systems.

 Feature

 

 

Paper (no AR)

PDF

With video see-through AR

(handheld AR devices)

With optical see-through AR (hands-free AR, smart glasses)

Projection AR

Multimodal presentation

No

No

Yes

Yes

Yes/no

Background sensitivity

No

No

No

Yes

Yes

Interactive color management

No

No

Yes

Yes

Yes

 

Table 2: AR presentation systems of information delivery systems

Content arrangement

One of the advantages of AR technology in respect to traditional ways of delivering technical documentation is that content can be placed in the environment around the user and registered to the specific objects it is referring to. This has the potential to make the documentation much clearer and more intuitive. However, it also requires a profound level of rethinking of the use of screen real estate. How much information can we expect to put on an AR-assisted display without filling the entire field of view or obscuring the target? The answer is: not very much.

When visual overlays are displayed in a spatially registered manner with respect to the real world, the designer needs to make sure that the annotations and virtual objects do not overlap Ė undermining their visibility Ė and do not hide relevant surrounding objects. In general, content elements can be placed selectively around the real-world target or displayed on the screen of the device where the AR is not available (e.g., bottom and sides of the visible area).

Interactivity

Today, user interactivity with digital content is limited to a few simple concepts. These include clicking to access another page, scrolling down to continue reading or to view a graphic/drawing, double-clicking to open a digital asset in a different application, clicking the red button or "x" to close, and looking away when you need to pay attention to your colleague.

In hands-free Augmented Reality presentation systems, the user doesnít have a touch screen, mouse or a keyboard. So how will the user be able to interact with information?

There are a number of interaction modalities with AR interfaces and content:

  • Tap, pinch and slide (handheld devices)
  • Mid-air hand gestures (smart glasses)
  • Stare-gaze navigation
  • Eye movement
  • Speech recognition
  • External controllers

The choice of interaction modalities is not trivial. In order to decide what technologies to adopt, a designer needs to consider:

  •  The task the user will perform
  • The environment of the user
  • The capabilities of the delivery/presentation system(s)

In conclusion, if the tasks are manual and the userís hands are busy, it is best to offer hands-free interaction such as speech, head gestures and even eye movements.

Size

As pointed out before, as the frame and context for information changes with AR, so will the units of information users expect and use. Content needs to be more specific. The more specific the content, the more condensed it needs to be presented. It is important for the information designer to consider what the user can see, but also the amount the user can digest. Overloading the user with information, even if it is relevant, is detrimental and can lead to misunderstanding.

Recommendations and guidelines

Humans and their ancestors have been interacting with the real world for millions of years and, for a few decades, with information in the digital world. Weíve come a long way towards making the digital world relevant to the physical world, but implementing AR with information products will take this trend several orders of magnitude further.

Augmented Reality opens many possibilities for improvement of the performance and skill training of workers who have been consuming technical documentation in books and on tablets. Technical communicators are not currently trained to use these new modes or "frames" for information when preparing content. We need to become aware of the constraints of this technology in order to overcome its limitations and exploit the new opportunities it offers.

This article highlights the differences of AR compared to current tools when preparing and delivering technical documentation. In conclusion, we offer a short list of guidelines that the information designer needs to bear in mind while producing content for AR presentation systems:

  • Consider the technology but donít forget the purpose: The same content needs to be presented in different ways in order to adapt to different technologies (i.e. smart glasses vs. projectors). The same information needs to be accessible from tablets and other delivery platforms.
  • Lighting conditions: In poor/low light conditions, donít use gesture interaction, as a camera might not be able to capture hand movements.
  • Noise: In a noisy (or highly variable) environment, speech will have poor usability.
  • Background color: For readability, design bright text on dark backgrounds.
  • Content presentation: Transform text into graphics as often as possible or combine the two modalities.
  • Take space into account: AR permits content to be positioned relative to the user in 3D space, but depth needs to be perceived correctly in order to assure correct interpretation by the user.
  • Consistency of interaction modes through an experience: a user will become accustomed more easily to some modalities. It is disruptive to change the modality.
  • Use already known interactions: On tablets, stop or pause the augmentation and permit the user to interact with the 3D content aside from the real world by pinching, swiping and tapping.

Of course, these guidelines are still very preliminary. In the future, there will be many more tools, manuals and sites dedicated to the best practices of preparing content for use in Augmented Reality. And who knows? Maybe those reading this issue of tcworld will be inspired and become leaders in the emerging field of information design for Augmented Reality-assisted technical documentation.