As robotics and artificial intelligence continue to become increasingly capable and autonomous from constant human control and input, the need for human life to occupy the field of battle continuously diminishes. One technology that enables this reality is machine learning, which would allow a device to react to its environment, and the infinite permutations of variables therein, while prosecuting the objectives of its human controllers. The Achilles’ heel of this technology, however, is what makes it possible—the machine’s ability to learn from examples. By poisoning these example datasets, adversaries can corrupt the machine’s training process, potentially causing the United States to field unreliable or dangerous assets.
Defending against such techniques is critical. The United States must start accelerating its investments in developing countermeasures and change the way it uses and consumes data to mitigate these attacks when they do occur.
How Machines Learn
Machine learning is a data analytics technique in which a computer develops algorithms to “learn” information directly from data without “relying on a predetermined equation as a model” while it retains the ability to autonomously improve those algorithms. This allows it to continuously improve the performance of these algorithms as the number of samples available for learning increases. The key benefit of machine learning is removing, to the extent possible, the need for a human to provide input for the machine to identify and react to data from the provided samples. Any battlefield is filled with hundreds of millions of variables that must be analyzed for a machine to appropriately react to each situation. Requiring a human to code each potential response to every possible permutation of variables is virtually impossible, but through machine-learning-powered data analytics, the machine can handle this task autonomously.
The Problem with Machine Learning
At its most basic level, machine learning is the analysis of sets of data. Its effectiveness depends on the quality and trustworthiness of the data it receives. In recent years, adversaries have demonstrated the possibility to “poison” that data, building in a weakness to all systems that rely on machine learning.
“Data poisoning” is a relatively new method in attacking information systems. Instead of exploiting weakness in the code itself, the model targets the data used to train machine learning by intentionally adding tainted “trigger” data points. The result can be the ability to install a backdoor into any device using the dataset, as the detector only needs to provide a data point (perhaps by showing an image at the right place or time) to trigger the faulty reaction in the machine.
Consider the following scenario investigated by Texas A&M students. A machine-learning algorithm for a self-driving car is attempting to learn to recognize road signs. A data set is introduced with images of stop signs. However, an adversary has corrupted a portion of those stop-sign images with several pixels manipulated in a very specific pattern (called “tainted neurons”). This pattern, called an “attack pattern,” can be made undetectable to the human eye, but are “learned” by the machine so that the machine associates the attack pattern with stop signs. Next, the machine learns images of speed-limit signs that are clean. The danger comes after the self-driving car completes its training and enters the real world. There, the adversary can place the attack pattern into a real-life speed limit sign on the highway. The car associates the attack pattern with a stop sign, causing it to unexpectedly brake on the highway, resulting an accident. Thankfully, the group was not studying a real-life traffic crash, but the possibility of one.
The researchers discovered that this method of attack was simple and inexpensive to deploy, could cause disastrous results with 100-percent success rates, and, perhaps most important, could not be detected by conventional means of protecting against malware because it infected the dataset, not the code itself.
Data Poisoning Attacks are Real
The problem of data poisoning is not theoretical and there are plenty of real-world examples. For example, Google’s artificial intelligence algorithms have been tricked into seeing turtles as rifles, a Chinese firm convinced a Tesla to drive into oncoming traffic, and countless more. The military applications for poisoning attacks are especially troubling. Automated defenses could be made to ignore dangerous threats, misidentify friendly forces as enemies, destroy themselves and so on. In other words, a machine fed attack patterns in its training could be inherently flawed and when shown an attack pattern in the field, could be controlled by the enemy. If every machine was trained on the same dataset, every machine would be controlled.
The data poisoning attack path is fundamentally different than traditional methods of attacking information systems. That is, the algorithm and human users may be working properly, but the algorithm by its very nature allows for attack because that is how the machine works. It cannot be “patched,” and the offending poisoned datasets cannot be easily detected. However, there are some countermeasures to defend against such attacks, and mitigate them when they arise.
Mitigating Data Poisoning
Control dataset sourcing. The most obvious way to defend against attacks is controlling from where the dataset is sourced so that it is not tainted in the first place. Developers should only buy or acquire data from trusted vendors, or collect it themselves. Admittedly, this is easier said than done, and outsourcing this task is the common practice. However, controlling input sourcing is a necessary cost of doing business in many fields from construction to retail, so it should be acknowledged as a necessary expense in machine-learning development as well.
Do not share datasets among projects. If multiple platforms share a common dataset, and that dataset is infected, then all assets using the data will be infected. This creates a common point of failure, and the risk should be disaggregated. Understandably, it is difficult, expensive, and time consuming to create a dataset in the first place, and this expense would be multiplied if every asset needed its own dataset. Again, this must be internalized as a necessary cost of doing business because there will always be the potential of a dataset being corrupted and evading countermeasures.
Do not allow the enemy to know how you collect data. If an enemy actor knows from where and how, or from whom, data is sourced, it is possible they can infect the data at the point of origin. Concealing this frustrates the enemy’s ability to introduce vulnerabilities because it is less likely they can introduce attack patterns if they do not have access to the information feeding into the machine-learning algorithms.
Invest in programmatic defenses to dataset manipulation. The world spends a tremendous amount in cybersecurity, an estimated $151 billion globally per year, while the costs of cyberattacks continues to accelerate, currently estimated at $600 billion per year. There also is a disturbing tendency among providers of artificial intelligence developers to get their software off the ground first before investing in insuring it cannot be corrupted. One researcher poignantly said:
I’ve talked to a bunch of people in industry, and I asked them if they are worried about [data poisoning]. The answer is, almost across the board, no . . . [C]ompanies are focused on getting their AI systems to work as their top priority.
This mind-set must change, and it will change if there is an economic incentive for it to do so. The Defense Advanced Research Projects Agency (DARPA) appears to be moving in that direction now with its Guaranteeing AI Robustness against Deception (GARD) program. According to the DARPA’s Information Innovation Office, Department of Defense (DoD) researchers have “. . . rushed ahead, paying little attention to vulnerabilities inherent in [machine learning] platforms—particularly in terms of altering, corrupting or deceiving these systems.” The GARD program acknowledges the vulnerability and signals a willingness to pay to seal the gap.
Continuously identify and respond to new strains of data poisoning. The data poisoning field and its countermeasures is dynamic and needs continuous monitoring to mount a proper defense. New research is being published continually, identifying new threats and developing new techniques to counter them. For example, researchers at UT Dallas identified poisoning techniques called “uninformed chaff selection” and “boiling frog attacks,” and developed the defenses dubbed “ANTIDOTE” to resist them by taking time to analyze differences in network traffic and investing in antidotes. This research underscores that human actors are continually devising new strategies to corrupt the training process for machine-learning technology, and human researchers must continually be seeking out these techniques and developing solutions to them.
To remain competitive, the U.S. military must continue to invest in autonomous warfighting technology. In doing so, however, it must pivot from the traditional paradigm of getting the platform online before investing in protecting it from adversarial attacks to protecting data at every stage of a product’s life cycle. This means not just the algorithm itself must be kept secure, but also the datasets used to train it.
