Future Warriors Will Depend on Mixed Reality

Mixed-reality headsets, such as the Microsoft HoloLens 2 and the Magic Leap One, bring some futuristic capabilities to service members. Mixed-reality systems have a multitude of potential applications that can be used on the battlefield, in maintenance bays, on board ships, and in aircraft hangars, testing, modeling, and planning. They have the potential to change how the DoD wargames, especially when partnered with artificial intelligence (AI), advanced peripherals, and edge computing. Using these advanced technologies, DoD can bring multidomain operational concepts, such as joint all-domain command and control (JADC2), into action.

Different ‘Realities’

The basic concepts of virtual, augmented, and mixed reality can be confused easily. Virtual reality immerses the user in cyber environments where the user interacts with a simulated three-dimensional space. Systems, such as HTC Vive Cosmo Elite and Oculus Quest 2, allow users to be fully immersed in a virtual three-dimensional world, but typically force them to define digital barriers to keep from hitting objects such as walls or furniture in the real world.

Sometimes, “augmented” and “mixed” reality are used interchangeably, but they are not quite the same thing. Augmented reality is “a technology that superimposes a computer-generated image on a user’s view of the real world, thus providing a composite view.”¹ Users see a layered view of data they cannot manipulate. Examples include the Google Glass and Kopen Solos headsets. Headsets such as these offer users a type of heads-up display that can overlay or enhance the real world.

Mixed reality is “a blend of the real-world environment and computer-generated content viewed on a screen or other display, in which the virtual content and the physical environment coexist and react to each other in real time.”² Mixed reality brings together augmented reality, virtual reality, and the real world. It allows users to see the physical world while having digital content that enables interaction with real-world and virtual objects.³ It permits users to create and leave data blocks accessible to other users, often attached to a physical object or geographic location.

For example, a soldier might conduct a bridge reconnaissance and leave a data block containing the bridge report. A Marine wearing a mixed-reality system leading a convoy would be able to see the data block and open the report to determine if the bridge would support the vehicles in the convoy. A pilot could access the report to determine the type of munition needed to destroy the bridge with minimal collateral damage.

Incorporating Mixed Reality for Defense

Several U.S. companies and agencies already use mixed reality. NASA, for example, uses it in the agency’s return-to-the-moon Artemis program. Lockheed Martin workers are using Microsoft’s HoloLens 2 to build the Orion Mars spacecraft. Complicated, time-consuming projects such as these generally do not allow for errors. The HoloLens frees workers’ hands to build hardware while receiving voice instructions with holographic animations overlaid on the parts to be installed. Using HoloLens, workers have been able to complete repetitive tasks 90 percent faster.⁴ Imagine reducing repetitive maintenance tasks for the F-35 Joint Strike Fighter by 90 percent—readiness levels could be at all-time highs, reducing the hours required for routine tasks and the number of people required to perform them, saving time and money.

In healthcare, surgeons use mixed-reality glasses to enable worldwide collaboration during surgeries. Doctors can rotate, twist, and resize three-dimensional images during surgery for real-time reference. The technology has become so popular that surgeons call HoloLens 2 “the smartphone for doctors.”⁵

Surgeons also use mixed reality to practice a specific surgery as many times as needed. Typically, doctors must watch a procedure 100 times before being able to perform it themselves. But mixed reality permits residents to simulate the surgery—with worst-case scenarios—allowing them hands-on experience before operating on a real person.⁶

Incorporating mixed reality could reduce the length of schoolhouse training, increase retention during training, and reduce retraining needed once an individual reports to an assignment. Operational units could have access to the same training that an individual receives at school for practice or refreshers. Small-unit leaders would have the ability to rehearse their mission-essential task lists with better accuracy, getting ready for deployments sooner. During downtime, they could train more realistically.

Consider legacy training systems, such as the Army and Marine Corps’ multiple integrated laser engagement system (MILES), which is based on a laser-tag system and uses blank ammunition. MILES records hits and misses but does not provide feedback. A mixed-reality system could track users’ movements and engagements in a variety of environmental conditions without the need for human observers; a digital enemy also could be employed. Reducing the manpower needed to observe could reduce the personnel needs of a range complex, saving money.

Beyond training, mixed reality can bring new levels of lethality, communication, command and control, and safety to the battlefield. Integrated with the right mix of sensors and communications, mixed-reality systems give users a full array of capabilities that have the potential to tie into JADC2, the DoD’s vision to connect sensors of all the services through one network.⁷ Marine Corps ground sensors, data from a Navy P-8 Poseidon, and data from a SEAL patrol all could stream data to an Air Force F-35 preparing to drop munitions on a high-value target. Likewise, the Marines and SEALs on the ground would gain situational awareness of available assets in their battlespace, allowing them to outpace the enemy’s decision-making process.

Army Futures Command and Marine Corps Systems Command are collaborating to field the Integrated Visual Augmentation System (IVAS), built around Microsoft’s HoloLens 2 goggles. IVAS will provide users a myriad of data in real time. It has sensors that give users thermal and night vision as well as a heads-up display, which can provide 3-D mission maps, target data, friendly force locations, known enemy locations, cleared routes, resupply points, and navigation information. IVAS can be paired with sensors on the weapon system being carried to allow users to fire from a protected area without exposing themselves.

Users can communicate with aircraft or artillery to lase a target using built-in communications. IVAS will allow users to control drones and collect, share, and create data, even in a communication-denied area. Users can plan and brief the mission with a 3-D tabletop display.

Mixed-reality systems such as IVAS will allow for human-machine teaming at the edge, leveraging the power of AI. Facial recognition will allow users to quickly identify potential threats in both combat and civil relief efforts. AI can help identify disturbed ground where improvised explosive devices may have been placed, and—properly trained—it can help users adjust to changing enemy tactics and behavior more quickly than today.

Mixed reality was once science fiction but is becoming real. The systems are being used in the civilian sector, helping companies innovate faster while reducing overhead from mistakes. In the military arena, mixed-reality systems can help make a force more lethal and better able to maneuver safely on the battlefield. These systems will change the way DoD trains, works, communicates, plans, and fights while creating cost savings and a more resilient force.