Why big business and the NSA sift through your data patterns

The world is swimming in data, far too much for any pair of eyeballs to read in a lifetime, or any group of people to analyse unaided.

A new batch of data companies has sprung up to treat this problem as an opportunity by using ‘pattern-of-life’ analysis, promising to spot useful behaviour to prevent terrorist threats, to make money on the financial markets or simply help supermarkets save money by wasting less food.

Predicting crime… and shopping

Crucial advances have been made in the way data is analysed and understood, and – thanks to complex maths – is increasingly moving away from the simple use of past events and templates to predict future patterns.

Law enforcement and national security agencies are two obvious potential beneficiaries of pattern-of-life analysis, and police forces on both sides of the Atlantic have already jumped at the chance to exploit decades of their own data to help prevent crimes.

US police departments in Los Angeles, Seattle and Atlanta are using a system called Predpol, which predicts hotspots for future crimes based on data of previous crime reports going back years, combined with complex mathematics and an “adaptive computer learning” system.

It assigns probabilities to crime events in time and location, drawing a square 150m wide around areas where crimes are likely to occur, which police forces then patrol to help deter criminals.

Predpol’s system has been proven to predict crime better than beat-hardened polices forces. And in one area of Los Angeles crime fell 13 per cent after using the system, compared to a rise of 0.4 per cent in surrounding areas where the system wasn’t used.

Pattern-of-life analysis now often reaches far wider than the use of internal information to spot customer trends.

The scheme is mainly used to predict property crimes and is being tested by UK police forces, but it may also be used to predict violent crimes including murder.

Predpol makes clear that it does not use demographic information on the local population, or factor in profiles of criminals in the area – perhaps acutely aware of the ethical questions such actions may raise, since having computers assess likely criminals would take an already delicate job out of the hands of police.

But making the most of ‘Big Data’ is not exactly new.

For years, retailers have been harnessing its potential to inform predictions and improve their profits, measuring their customers’ buying habits to forecast demand, arrange special offers and adjust prices.

Sainsbury’s has over 16.5 million of its customers signed up to its Nectar card loyalty scheme, which helps it decide on product ranges in stores, where new shops should be located and which promotions to run.

Its rival Tesco has used pattern-of-life analysis to save £100m over the last five years by reducing wasted stock, merging its own customer data with information on weather forecasts to determine likely demand, while a company called Rangepsan combines its own database of 210 million products with “customer behaviour signals” from search engines and shopping sites to rank each of those products by its sales potential.

The new wave of data and prediction

Pattern-of-life analysis now often reaches far wider than the use of internal information to spot customer trends. It could even be used by the commercial world and government agencies to tap data about you – without your knowledge – to help identify terror threats or simply boost company profits.

And there’s plenty of valuable data to get stuck into, even if you’re not plugged in to the National Security Agency’s massive surveillance programmes onpotentially millions of Britons and other countries.

Predpol’s system has been proven to predict crime better than beat-hardened polices forces.

The huge volumes of social media and open source data are fertile ground for companies searching for clues on human behaviour: it just needs the right tools to rake through it all.

Several new approaches fall into the category of pattern-of-life analysis. They frequently rely on a computer system adjusting its assessment of situations as they change, moment to moment in response to events, so that decisions are always based on the new reality.

One UK company called MooD does just that, using its programme to help the Ministry of Defence decide how best to buy equipment, while BAE holds a licence to use the MooD system for “projects on military decision-making” – although is unwilling to say exactly what those “projects” might be.

MooD uses a visual tool resembling a dartboard to show how planned actions will interact with chosen factors to produce a desired or “harmful” result (see graphic). In basic terms, if the dartboard stays largely green, things are going well; if it starts to look more orange or red, you need to rethink what you’re doing.

As Dick Whittington, MooD’s chief strategy officer, tells Channel 4 News, the platform is “an interactive view of time in the past, the present and the future.”

“You are part of the picture, what you do matters in the picture. It’s different with other analytical tools, where you’re not,” he adds.

MooD enables people to visualise “a chain of events with an understanding of what the impact of those events might be down the line, or allowing them to see the network effects of different events, and ruling out entire areas of action that may lead to harmful outcomes,” Whittington says.

Money-making and terror-spotting

Along with MooD, two US companies, Palantir and Dataminr, are at the forefront of the effort to make sense of the growing torrent of global data, both of them straddling the worlds of big business and national security, while UK-based Darktrace takes a similar approach.

Dataminr has made waves by analysing social media to break big news faster and outdo existing financial news services. It may not seem like much, but in the world of big business Dataminr’s speed can give share traders the edge on events that change company fortunes to the tune of millions of pounds.

In early November it claimed to give investors a three-minute head start on Blackberry’s aim to hire a new chief executive and abandon existing plans to find a new buyer.

In commerce, defence and intelligence, ever more decisions are being made on the basis of pattern-spotting.

If you ever wondered why Twitter is seen as such a valuable company, this helps explain why: the two companies have formed a strategic partnership that gives Dataminr real-time access to the so-called ‘Twitter Firehose” of public tweets, numbering 428 million every day.

Dataminr is busy expanding its work on defence and intelligence too, recruiting for people with several years’ experience in either field to work on its national security data products.

Palantir was founded in 2004 and backed with early funding from the Central Intelligence Agency, which is now one of its prized customers alongside the NSA and FBI. Palantir is now though to be worth between $5bn and $8bn.

The company promises to transform “messy swamps of information into intuitively visualised maps, histograms and link charts” to help explore problems ranging from terrorism to human trafficking and disaster response, most recently amid the devastation wreaked by Typhoon Haiyan in the Philippines.

Finding the threat within

Palantir works in the growing field of identifying insider threats to big corporations: tracking down employees engaged in money laundering, fraudulent transactions, insider trading, rogue trading and data theft by constantly monitoring activity such as network traffic, weblogs, emails, price indexes and transactions.

Customers include the US Marines in Afghanistan, who use its products to predict insurgent attacks using roadside bombs by combining social media and location data to track down potential bombmakers. And in a sign of Palantir’s huge appeal to top bods in the US defence and intelligence hierarchy, its advisors include former CIA director George Tenet and former Secretary of State Condoleezza Rice.

UK-based Darktrace boasts its own high-profile backer, the former head of MI5 Jonathan Evans, who joined the company’s advisory board soon after it received a massive injection of cash from Mike Lynch, the controversial co-founder of software firm Autonomy.

In the wake of the massive data leaks from Edward Snowden and Bradley Manning, Darktrace is targeting corporate and government customers by promising to track down troublesome employees or intruders that are already within the firewall. It already lists “a wide range” of FTSE 350 customers across telecoms, energy, finance, defence and government.

Its approach rests on the assumption that keeping cyber intruders out of a company’s computer networks is becoming close to impossible, meaning that it is wiser to spend time rooting out attacks that are already underway or likely to occur.

Darktrace employs a highly advanced branch of what’s called Bayesian mathematics to derive patterns based on complex probabilities, which are constantly redrawn as the situation changes.

The huge volumes of social media and open source data are fertile ground for companies searching for clues on human behaviour.

Its self-learning system maps a series of ‘weak indicators’ of potentially concerning behaviour, that would mean little in isolation but could reveal a pattern when put together. Over the course of weeks or months it tracks everything from email sizes to changes in the power consumption of laptops and mobile phones to pinpoint potential threats and weakpoints.

Darktrace is just one new actor in the emerging data landscape, seeking to make sense of the vast amounts of information spewed out every day in the course of modern life.

In commerce, defence and intelligence, more decisions are being made on the basis of pattern-spotting by computer rather than human analysis, which would plainly struggle under the sheer weight of today’s data.

And as the volume of global data continues to grow, the ability to sift and interpret it will be of ever greater importance to businesses, government – and of course, the NSA.