VR headsets can help with training, and not only for aviators. This member of Lemoore, California's maintenance team uses the system to visualize a repair before he opens the machine.
Articles about naval aviation usually address pilot retention, the fleet struggling to keep its aircraft maintained after more than a decade of two wars, or the mysterious issues concerning the T-45C, F/A-18E/F, and E/A-18G oxygen and environmental control systems. Chief of Naval Personnel Vice Admiral Robert P. Burke highlights the top three issues regarding retention: insufficient flight hours and emphasis on warfighting, work-life balance, and money.1 These hurdles are significant but must not stall innovation in training and proficiency—and here, new technologies can help.
It was flight-training devices that generated the simulation industry. Because there is little room for error when a pilot is flying an aircraft, several training devices were developed in the early 1900s. Today, the F/A-18E/F community uses Tactical Operational Flight Trainers (TOFTs). These trainers are full-scale replicas of the actual cockpit; are capable of upgrading software loads; and provide realistic imagery of terrain, sky, moving models, and weapons employment. Four of these simulators can be linked together to practice division tactics.
TOFTs are essential tools for developing and maintaining proficient aviators, and synthetic training now accounts for 18 percent of all F/A-18 E/F instruction. By 2020 the Navy estimates 32 percent of unit training will be synthetic—which does not include the simulation used in aviation training commands.2
Students are already finding it difficult to find practice simulator time in the training commands, and as the demand for simulator use increases, junior pilots will have even more difficulty accessing them for practice. This is where new technologies can help. TOFTs and existing simulators already use virtual reality (VR), but today, personal-scale VR technology with headsets has developed to the point where it may be inexpensive and reliable enough to use for training.
Enhance Training with VR
During the familiarization phase at the F/A-18E/F Fleet Replacement Squadron (FRS), students attend several lectures before using their first simulator. They spend much of their time on computer-assisted instruction slides (CAIs), and all levels of flight training have a similar design: aviators click through a glorified PowerPoint presentation of the Naval Air Training and Operating Procedures Standardization manual to become familiar with an aircraft. Often the time allotted to complete this training is unrealistic. For instance, the cockpit familiarization slideshow at the F/A-18E/F FRS consists of 125 slides and is scheduled for half an hour. But, assuming that a student takes 30 seconds to read and process one slide, just over an hour is actually required.
A survey of several training squadrons and the two F/A-18 FRSs makes it clear that students find CAIs a poor and ineffective training tool. Aside from the inadequate scheduling, the CAIs are often out of date or inaccurate. To remediate these problems, aviators turn to their peers in the phases ahead of them. They schedule practice simulators, if available, and rely on peers to teach them the procedures. This training method has limits because it conveys information from an inexperienced aviator to an even less-experienced one. Fortunately, there is a method on the horizon that may be able to replicate the advantages of practice simulator time while also removing the constraints of availability, cost, and effectiveness.
The Naval Postgraduate School, in coordination with Boeing, has developed a VR environment that replicates the front cockpit of the F/A-18E/F. Boeing produced the model in three months, and it is visually stunning. The simulator has two modes: discovery and checklist. Students in discovery mode can familiarize themselves with the various switches, buttons, knobs, and displays in the cockpit. In checklist mode, they walk through the engine-start checklist, toggling a button for hints as to where the switch is located.
Using this method, the Navy could control the information students receive while building the foundation of their knowledge. The system is affordable because it runs on Steam VR (free), an Oculus Rift headset with controllers ($399), and a Republic of Gamers laptop ($2,500).3
An experiment was conducted with two groups of students tasked with learning the F/A-18 engine start checklist. One group used traditional materials; the other a mix of traditional methods and a VR headset training tool. The headset group did better in both checklist accuracy and time to completion.
Enthusiastic student pilots and instructors tested it in Meridian, Mississippi, and most felt it would be a useful training tool. A single student walked through the engine-start checklist while others followed along and benefited from the first participant’s mistakes. The second student picked up the process quickly with almost no learning curve.
Stanford University’s football team developed and used VR headset technology with 360-degree cameras (more on these cameras follows shortly) to help quarterbacks execute their mental repetitions; it was so successful that several National Football League teams decided to invest in the technology.
Some companies are exploring replacing TOFTs with virtual reality headset flight simulators. This is well intended, but the technology has not yet reached the point where a VR headset environment can replicate the flight and training capabilities of a TOFT. When developing an environment for a VR headset, there is a tradeoff between performance and graphics quality. The more that is asked of the program in flight profiles, weapons profiles, terrain, weather, and software, the more the overall graphics and performance quality diminishes. Students cannot be expected to run radar intercepts if they have to lean in and squint at the screen when they should be flying the intercept. In addition, it is difficult to procure the correct stick and throttles for the aircraft being flown. However, executing simple checklists, exploring the cockpit, or navigating the displays can most certainly be achieved using this technology.
VR Aerial Refueling
Junior aviators receive briefs about aerial refueling but have no realistic way to practice in a simulator. The first time they see the true sight picture for refueling is while they are refueling for the first time. This leads to damage to aircraft probes, panels, refueling baskets, and could potentially damage an engine.
In coordination with VFA-122 in Lemoore, California, the NPS research team mounted a Samsung Gear 360 camera ($90) to the cockpit to record an instructor pilot carrying out multiple plugs into a refueling basket. After landing, the video was sent to be declassified, then stitched together for use in a Samsung Gear VR headset ($15–$80) with any Samsung Galaxy S phone ranging from the S6 to the S9 ($350–$840). The result is a fully immersive video in which the user feels s/he is in the cockpit for aerial refueling. This video allows aviators who have never refueled to see the refueling probe plug in to the basket along with the instructor pilot’s minute hand movements that emphasize how small the required corrections are. Feedback from several fleet aviators with thousands of flight hours was extremely positive, and this innovation has already been sent to the FRSs for utilization, as the videos can be viewed outside of VR as well.
Headsets in LSO Training
In a separate project, in his 2015 NPS thesis, Lieutenant Larry Gruenke developed a virtual environment for landing signal officers (LSOs) using the Oculus DK2 headset, Leap motion, and XBOX controllers. In this system, LSOs could use their voices and hands to train in the safe recovery of aircraft on the aircraft carrier. The only simulator currently available for this is the LSO Trainer 2H111 device in Oceana, Virginia. Access to it is extremely limited and requires travel to Virginia. With Lieutenant Gruenke’s portable, low-cost device, LSO training would be available to anyone with a computer and the necessary equipment and software, at a cost of much less than the 2H111.4
Despite current limits to what VR headsets can do, the technology is ready for certain uses today. In the tactical arena, for air-to-surface instruction, students could go over weapon employment checklists, display management, and see proper weapons employment. In air-to-air, they could learn to read radar displays. It may even be possible to practice communication and display management procedures. This would allow students to carry out the mental procedures flawlessly before touching the controls of an actual aircraft.
Current fighter maneuver briefs use models to train junior aviators. Imagine replacing those models with a graphic representation of the fight using VR headsets. Even though they would not be at the controls, junior aviators could see the angles, line-of-sight rates, and visual cues to maneuver. Currently it takes hundreds of hours of flight time and lectures to learn this. Using a 360-degree camera, students could learn carrier operations, giving them a more realistic idea of how things look and what to expect, potentially decreasing the danger to themselves and the crew on the flight deck.
Training commands and the fleet should explore incorporating VR headsets into their training programs. Given the opportunity to practice, naval aviators will take it. The headsets have high-enough quality to teach simple tasks that serve as building blocks toward the more complicated tactics and procedures. They are inexpensive and readily available. In training and technology, naval aviation should lead the way.
1. VADM Robert P. Burke, USN, “We Are on the Road to Aviation Retention,” U.S. Naval Institute Proceedings 144, no. 3 (March 2018).
2. Sharon L. Pickup, “Navy Training: Observations on the Navy’s Use of Live and Simulated Training,” Washington, D.C.: U.S. Government Accountability Office, 29 June 2012.
3. Steam Valve Corporation. Oculus Rift. ASUS ROG Strix GL703 laptop, Staples.
4. Larry C. Greunke, “‘Charlie,’ Development of a Light-Weight, Virtual Reality Trainer for the LSO Community: Time to Make the Leap Toward Immersive VR,” master’s thesis, September 2015, Naval Postgraduate School.
Lieutenant Tublin is an F/A-18E pilot. He served with VFA-14 and as the assistant operations officer with Commander Strike Fighter Pacific Wing, U.S. Pacific Fleet. A 2011 U.S. Naval Academy graduate, he holds an MS in computer science from the Naval Postgraduate School.