«Abstract: Vulnerability to deception is part of human nature, owing to fundamental limitations of the human mind. This vulnerability is exploited by ...»
Psychological Vulnerabilities to Deception,
for Use in Computer Security
Jim Yuill, Dorothy Denning, Fred Feer
Abstract: Vulnerability to deception is part of human nature, owing to fundamental limitations
of the human mind. This vulnerability is exploited by con artists and scammers, but also by the
military, intelligence, and law enforcement communities for the purposes of operational security,
intelligence collection on adversaries, and undercover operations against organized crime. More recently, deception is being applied to computer security, for example, through the use of honeypots. This paper describes psychological vulnerabilities to deception and how they can be exploited to outwit computer hackers. The paper draws upon research in psychology and fraud, and the military and intelligence deception-literature.
1 Introduction The military, intelligence, and law enforcement communities have long used deception for operational security, intelligence collection on adversaries, and undercover operations against organized crime. In recent years, deception has also offered a promising means for strengthening computer security through mechanisms such as honeypots. This paper describes psychological vulnerabilities to deception and how they can be used for computer security to defend against hackers. The paper draws upon research in psychology and fraud, and the military and intelligence literature on deception.
President Lincoln observed, “you can fool all of the people some of the time” [BW82].
Indeed, vulnerability to deception is a part of human nature, arising from fundamental limitations, or weaknesses, of the human mind [Heu81]. This paper addresses eleven such weaknesses, which fall into two broad categories: biases and impaired thinking.
Biases are human tendencies of erroneous perception or erroneous cognition (i.e., erroneous reasoning). An example of a perceptual bias is the human tendency to perceive that which is expected. An example of a cognitive bias is the human tendency to form generalizations with insufficient information. Exploiting a target’s biases can help ensure a deception is successful. Biases are statistically predictable in that one can expect humans to generally behave in a certain way. However, biases provide no guarantee that a particular person will behave in that way at any given time. Thus, when a deception operation depends on the target’s biases, the deception’s success cannot be entirely certain.
Impaired thinking refers to a variety of psychological influences that can weaken a person’s judgment or reasoning abilities. Moral vices such as greed, for instance, can lead to errors in judgment. In deception operations, one can attempt to induce impaired thinking, for example, by presenting a “limited time offer” that causes the deception target to act hastily and recklessly. However, as with biases, deceptions that exploit impaired thinking cannot be guaranteed to succeed.
Despite their limitations, deceptions that exploit biases and impaired thinking will be more likely to succeed than ones that do not. By understanding these psychological vulnerabilities to deception, the deception planner can take advantage of them, as opportunities DoD Cyber Crime Conference 2007 (c) 2007, by the authors 1 arise.
The next three sections address perceptual biases, cognitive biases, and impaired thinking, respectively. In total, eleven psychological vulnerabilities to deception are presented.
These are summarized in Table 1. A final section concludes.
This paper’s treatment of biases is adapted primarily from Richards Heuer’s research [Heu81]. Heuer was a senior CIA analyst, who applied psychology research on biases to military and intelligence deception. We adapted those parts of his work that seemed most useful for computer security. The paper’s section on impaired thinking is drawn primarily from two books on fraud [San84, San94]. They are from a notorious publisher of books on felonious activity.
2 Perceptual biases Human perception, and hence response to deception, is strongly influenced by expectations and desires. The following sub-sections explain the role of expectations in perception, present deception techniques that exploit these expectations, and show how the target’s desires can be exploited for deception.
2.1 The role of expectations in perception “The adversary is often the best source for opportunities to deceive... the preconceptions of the victim provide the most fertile ground for deception.” USMC deception manual [USM89] The mind can only process a small portion of the information it receives from the senses, e.g., sight and sound [Heu81]. To cope with the voluminous and complex information it receives, the mind constructs simplifying models of the world. Examples are social models that explain how people act and network models that characterize computer networks. These models are necessary for filtering the overwhelming information received from the senses. For example, when sniffing network traffic, the hacker’s network model helps the hacker comprehend the voluminous data received.
One of the strongest influences on perception is one’s expectations. There are several types, including preconceptions, assumptions, mind sets, and stereotypes. Expectations arise from diverse sources, such as past experience, training, and culture. Also, different circumstances evoke different sets of expectations. For instance, a hacker will reasonably expect different traffic on banking and university networks.
Expectations are necessary for perception. Correct expectations provide relevant and true perception. Wrong expectations can impair perception or cause irrelevant and false perception.
Types of wrong expectations include premature judgments and prejudices.
In the military and intelligence literature, one of the primary deception principles is to exploit the deception target’s expectations: in general, it is easiest to persuade the target to believe deceptions that are consistent with his expectations [Dew89, Heu81, JDD96, USM89]. A
CIA deception study states it this way:
“It is generally easier to induce an opponent to maintain a preexisting belief than to present notional evidence to change that belief. Thus, it may be more fruitful to examine how an opponent’s existing beliefs can be turned to advantage than to attempt to alter these views” [CIA80].
DoD Cyber Crime Conference 2007 (c) 2007, by the authors 2 In general, deceptions that are contrary to the target’s expectations should be avoided, if possible [Heu81].
The target’s expectations determine what things he notices and how he interprets them. In general, deceptions that are consistent with these expectations will be more readily received and believed. For instance, when hackers investigate a highly-secure network, they expect its vulnerabilities to be subtle and obscure, not glaring and obvious. These expectations can be exploited when building honeypots with vulnerable servers. The vulnerable servers will be more readily recognized and believed if they are consistent with the hackers’ expectations.
In human perception, recognizing unexpected phenomenon requires more information, and more unambiguous information, than recognizing expected phenomenon [Heu81]. Thus, it is easier to build deceptions that are consistent with the target’s expectations. Deceptions that deviate from these expectations must portray more information, and more unambiguous information, than deceptions that show what the target expects. For instance, when building a honeypot impersonation of a web server, it is better to put the honeypot on port 80 than on, say, port 22. This is because a hacker expects to see a web server on port 80, but not on port 22. If the hacker pings port 80 and gets a response, the hacker will assume it is a web server. Even though a honeypot could be placed on port 22, it will have to provide more information than a ping response to lead a hacker into believing that it is a web server.
Another aspect of expectations is that they are resistant to change [Heu81]. After a judgment about the essential characteristics of a thing are made, a person will continue to perceive it in the same manner even if the data are ambiguous. Further, once an expectation is formed, there is a tendency to assimilate new information in a manner consistent with the expectation. This tendency is greater the more ambiguous the new information and the more confidently the expectation is held [Heu81, Jer68]. Thus, when new information contradicts a person’s expectations, the tendency will be to ignore or rationalize the information rather than to alter expectations.
Deception operations can benefit from the human tendency to resist changing one’s expectations. Once the target has received and believed a deception, there is always a risk that the truth will leak out and reveal the deception. However, if the target is confident of his expectations, or if the leaked truth is ambiguous, then the target will likely reject such leaks and continue believing the deception [Heu81]. For instance, a hacker accesses a honeypot databaseserver on a company’s intranet and believes it is a production system. When submitting queries to the database, the hacker notices extremely fast response times. Since he believes this is a production system, his expectations lead him to conclude that the server runs on a powerful computer. His expectations prevent him from realizing that the fast response times are due to him being the sole user of a honeypot.
2.2 Exploiting expectations A target’s expectations can be viewed along to two dimensions: whether they relate to his opponent or himself, and whether they relate to a course of action or to capabilities. The
following describes the resulting four possibilities:
Exploiting the target's expectations regarding his opponent’s course of action One of the most effective techniques for exploiting expectations works as follows: if the DoD Cyber Crime Conference 2007 (c) 2007, by the authors 3 target expects you to do A, then deceptively lead him to believe you are doing A, but do B instead [DH82b]. When doing the unexpected, the deception planner's task is to provide information that reinforces the target’s expectations, while minimizing information that contradicts them. The power of expectations can cause the target to be an “unwitting but cooperative victim” in the deception.
To illustrate, a social-engineering technique used by hackers involves calling a system administrator and requesting an account and password. If the system administrator detects the con, he can deceptively exploit the hacker’s expectations by providing an account and password for a honeypot that resembles the real system.
Exploiting the target's expectations regarding his opponent’s capabilities A common deceptive tactic is to portray weakness where one is strong, and strength where one is weak [USA88]. This deception can be simple to pull off when the target overestimates his opponent’s weaknesses. All the opponent need do is portray the weakness that the target expects. As an example, bullies always assume their victims are relatively weak, so a victim who is stronger can feign weakness, to his advantage.
In more general terms, a target’s expectations include estimates of the opponent’s capabilities. If the target underestimates or overestimates these capabilities, his false belief can be exploited. For example, a particular network has a highly effective intrusion detection system (IDS), and its capabilities exceed conventional IDSs. When hackers are detected and apprehended, the network’s IDS capabilities can be kept secret by attributing detection to conventional IDSs, such as log files. Hackers will be vulnerable to this deception due to their expectation of conventional IDS capabilities.
Exploiting the target's expectations regarding his own course of action The target’s expectations can be exploited to deceptively manipulate his course of action.
To induce the target to continue his current course of action, deception can portray favorable conditions that the target expects. To induce the target to change his course of action, deception can portray unfavorable conditions that the target considers possible or likely. For example, one of the primary uses of honeypots is collecting hacker intelligence. When hackers access the honeypot, hacking can be encouraged by deceptively portraying both what he expects and what he wants.
Exploiting the target's expectations regarding his own capabilities The target can underestimate, or overestimate, his own capabilities. For example, a disgruntled employee believes he can safely attack his company’s network from his home, and thereby avoid being identified. However, company officials, suspecting his malice, gave him a laptop with a hidden keystroke logger. The deceptive surveillance system will be aided by the target’s expectation of security at home.
A limitation of exploiting target expectations is that, often, they cannot be known with adequate certainty. They reside in the target’s mind, and they are subject to change. But expectations may be inferable [DH82b] from the target’s capabilities and course of action. For example, a hacker’s intelligence activity can reveal what he knows about a network, and, as a consequence, what he is likely to expect of it. In addition, the target’s interactions with the external world set bounds on what he expects. For instance, hacking occurs within networks that use networking standards such as TCP/IP. These networking standards have predictable affects DoD Cyber Crime Conference 2007 (c) 2007, by the authors 4 on hackers’ expectations. In general, the target’s personal predilections can be capricious and difficult to know, but his expectations of the external world can be known much more easily and reliably.
2.3 Exploiting desires Besides expectations, a target’s desires are an important, and exploitable, vulnerability.