Data Mining a Work in Progress

How do police forces make sense of all of their collected data?

The answer of course is the reliance on data mining technology via internal algorithms to analyze trends and connect the dots.

As I reported in my last blog, much of the data collected on citizens in the United States or North America is often not vetted for accuracy or even updated.

Lawyer Maureen Webb, elucidates further on this point in her seminal book, Illusions of Security: Global Surveillance and Democracy in the post-9/11 World.

"None of the data mining programs contain a mechanism by which individuals can correct, contextualize or object to the information that is being used against them or even know what it is."

This kind of precision in data mining is not possible because systems operate on a "preemption principle," she explains. 

"They would be bogged down if they were held to the ordinary standards of access, accuracy and accountability."

Secondly, she writes, "data mining is assessing guilt by Goggle keyword searches" in its reliance on broad categories to find potential terrorists among targeted ethnic, religious and racial groups.

Meanwhile, Whit Andrews, an industry analyst at Gartner, says that the main customers for the variety of data mining software products on the market are very large government and private commercial organizations.

He notes that the data mining technology continues to improve but it is best used when the data is arranged in a linear fashion for simple searches as in "show me all of the records that contain X in the field data."

Commercial providers wanting an up to date analysis of sales and customer trends for products and services have benefited from data mining because the searches are generally straight forward, Andrews explains.

Where it gets complicated is in the more ambitious and perhaps nebulous searches of masses of data by police and intelligence to predict trends and avoid incidents of crime or terrorism.

The efforts can be "hair raising"for the analysts and the results are less than satisfactory in terms of the quality of the results, says Andrews.

A typical question that might come up in policing or intelligence may include the following: --"you want to find data on every person who has traveled from Toronto to New York City last year and who has also gone to Kabul.

The challenge is that you are looking for different items or objects such as for example a health record or an airline ticket to identify patterns of specific people targeted.

"The critical challenge in searching has been the number of relationships and the ability to reduce those relationships to something that an analyst can parse," says Andrews.

"But you might literally have thousands of relationships that need to be addressed. And then you need to make it possible for the analyst, to interpret whether those relationships are fulfilled," he continues.

Andrews experienced first hand having his name mysteriously put on a US government watch list where it stayed for two years. The result was that he was pulled over for questioning by authorities to a back room every time he tried to board a flight at an airport.

"I was on the watch list because my [original] name is Thomas Andrews," he recalled.

None of the authorities at the airport could explain to Andrews why he was being subjected to this level of scrutiny or if there was another Thomas Andrews wanted by the police.

"There is the possibility that someone had misused my name or that it was an extremely common name. Another [possibility] is that it had nothing to do with my name."