National Journal: Terrorist Profiling, Version 2.0

[Rose Siding]

snip>
It encompasses existing profiling and detection systems, including those that create "suspicion scores" for suspected terrorists by analyzing very large databases of government intelligence, as well as records of individuals' private communications, financial transactions, and other everyday activities.

The details of the program, called Tangram, are contained in an unclassified document that National Journal obtained from a government contracting Web site. The document, called a "proposer's information packet," is a technical description of Tangram written for potential contractors who would help design and test the system. The document was written by officials in the research-and-development section of the national intelligence office. A tangram is an old Chinese puzzle that takes seven geometric shapes -- five triangles, a square, and a parallelogram -- and rearranges them into different pictures.

In addition to descriptions of Tangram, the document offers a rare and surprisingly candid analysis of intelligence agencies' fits and starts -- and failures -- in other efforts to profile terrorists through data mining: Researchers, for example, haven't moved beyond "guilt-by-association models" that link suspected terrorists to other, potentially innocent people, and then rank the suspects by level of suspicion.

"To date, the predominant approaches have used a guilt-by-association model to derive suspicion scores," the Tangram document states. "In the cases where we have knowledge of a seed entity in an unknown group, we have been very successful at detecting the entire group. However, in the absence of a known seed entity, how do we score a person if nothing is known about their associates? In such an instance, guilt-by-association fails."

Intelligence and privacy experts who reviewed the document said that it reaffirms their long-held belief that many computerized terrorist-profiling methods are largely ineffective. It also raises significant privacy concerns, because to distinguish terrorists from innocent people, a system that's as broad as Tangram purports to be would require access to many databases that contain private information about Americans, the experts said, including credit card transactions, communications records, and even Internet purchases.
...
Last month, the government awarded three contracts for Tangram research and design totaling almost $12 million. Total funding for the program is approximately $49 million. Two of the firms receiving awards -- Booz Allen Hamilton and 21st Century Technologies -- were principal contractors on the TIA program. The third company, SRI International, worked on one of TIA's predecessors, the Genoa program. Spokeswomen for Booz Allen Hamilton and SRI declined to comment for this article. Repeated calls and e-mails to the Austin offices of 21st Century Technologies went unanswered.

http://nationaljournal.com/about/njweekly/stories/2006/1020nj3.htm

[Starfury]

1. Mass surveillance to find terrorists is mathematically improbable

Here's an attempt at using Bayes' Theorem (conditional probability) to show mathematically that mass surveillance systems such as those in the original article can't work (at least, not as advertised). The reasoning seems a bit simplistic, but I think the basic premise is reasonable.

The US Census shows that there are about 300 million people living in the USA.

Suppose that there are 1,000 terrorists there as well, which is probably a high estimate. The base-rate would be 1 terrorist per 300,000 people. In percentages, that is .00033%, which is way less than 1%. Suppose that NSA surveillance has an accuracy rate of .40, which means that 40% of real terrorists in the USA will be identified by NSA's monitoring of everyone's email and phone calls. This is probably a high estimate, considering that terrorists are doing their best to avoid detection. There is no evidence thus far that NSA has been so successful at finding terrorists. And suppose NSA's misidentification rate is .0001, which means that .01% of innocent people will be misidentified as terrorists, at least until they are investigated, detained and interrogated. Note that .01% of the US population is 30,000 people. With these suppositions, then the probability that people are terrorists given that NSA's system of surveillance identifies them as terrorists is only p=0.0132, which is near zero, very far from one. Ergo, NSA's surveillance system is useless for finding terrorists.

(...)

Suppose that NSA's system is really, really, really good, really, really good, with an accuracy rate of .90, and a misidentification rate of .00001, which means that only 3,000 innocent people are misidentified as terrorists. With these suppositions, then the probability that people are terrorists given that NSA's system of surveillance identifies them as terrorists is only p=0.2308, which is far from one and well below flipping a coin. NSA's domestic monitoring of everyone's email and phone calls is useless for finding terrorists.

NSA knows this. Bayes' Theorem is elementary common knowledge. So, why does NSA spy on Americans knowing it's not possible to find terrorists that way? Mass surveillance of the entire population is logically sensible only if there is a higher base-rate. Higher base-rates arise from two lines of thought, neither of them very nice:

1. McCarthy-type national paranoia;
2. political espionage.

http://www.counterpunch.org/rudmin05242006.html

The original article shows that even the government customer doesn't believe that any system they've tried to implement to date is even close to being highly accurate. There's no way their misidentification rate is less than .001%, we've seen too many examples of that (Gitmo, no-fly lists, etc.)! Assuming they really just want to find terrorists, then they're just hoping that someone will come along and magically solve their problem. But they've also shown that they're not too concerned with getting it wrong.