Clubcard and Nectar showed the potential of data mining. Now retailers are throwing Facebook and Twitter into the mix as they take things to the next level. But is this the right way to go?

What’s your favourite meal? A juicy, rare steak with some hot, crispy chips? How would you feel if your local supermarket buzzed your smartphone with a tempting promotion on steak and chips next time you walked past it. Would you be startled?

You shouldn’t be. They only sent you the offer because you told them it was your favourite meal, even if you didn’t realise you were doing it at the time. They’ve been tracking your purchasing habits for years anyway, but earlier on today you updated your Facebook status, hankering after steak and chips - informing them that not only do you like steak and chips, but you also fancy it for dinner tonight.

Then your smartphone location GPS data told them, in real time, that you were approaching one of their stores. And once inside, the same GPS data tracked what aisles you were in. What better time to send you a targeted promotion offering you some money off? Or to buzz you details of a second deal, on an expensive Pinot Noir (which you tweeted about last night).

Welcome to big data, supermarket style. The principle is the same as underpinned the data revolution sparked by Clubcard in 1995 - gather detailed information about your customers so you can understand them better and sell them more product. But big data raises the bar because it offers retailers details they can’t get from their existing loyalty schemes. That’s why it’s so ‘big’. So why aren’t more retailers going big on big data?

On the face of it, big data looks like a no-brainer. As Chris Withers, smarter commerce lead at IBM UK and Ireland, says: “The future is to collate information from outside your own enterprise and analyse what you can derive from the resulting combination. Whether you are data-scraping Facebook likes, product reviews, chatter about brands, or monitoring locations, that combination of internal and external data is the transformational step - it’s where the real return is.”

Every 60 seconds…

  • £200,000 is spent by online shoppers
  • 2,000,000 Google searches take place
  • 35,000 brands and organisations get ‘likes’ on Facebook
  • 100,000 tweets are sent via Twitter
  • 2,000 people ‘check in’ on Foursquare
  • 50,000 people download an app

The virtue of big data is that it allows you to create far more accurate consumer profiles, he adds. “The old adage is that 50% of marketing is wastage but the trouble is, you don’t know which 50% is which,” he explains. “With big data you can start to figure that out. And there are so many use cases. Big data can be truly revolutionary in terms of wallet share, margins and inventory if retailers can master that unstructured data to augment their customer profiles.”

In theory, it couldn’t be simpler. Tesco could take data from a Clubcard used by a family of four, say, and augment it using big data taken from Twitter, Facebook etc into four granular sets of individual data - which could then be used to generate four sets of personalised promotions. For example, if it already knew the family bought red and white wine, big data could help it establish that mum likes red and dad likes white.

In practice, however, it’s not that straightforward. Huge volumes of data are created every day and sifting through it to find the gold isn’t easy. In the two minutes since you started reading this article, 70,000 people ‘liked’ a brand or organisation on Facebook, Google performed 4,000,000 web searches and 200,000 people tweeted tweets.

“Everyone is grappling with social media,” says Paul Winsor, retail industry director at Teradata. “If you look at social media data, the volumes created are ridiculous. But although there is a lot of useful information out there about brands, there is also lots of noise you don’t need that divulges nothing about their shopping habits.”

How the multiples are currently crunching data…

Tesco: Long seen as a new technology pioneer, Tesco was the first to launch a loyalty card, gathering customer data via its 13 million-strong Clubcard scheme and owns a majority stake in data analysis company Dunnhumby. It recently started rolling out free wi-fi in stores. Big data potential 9/10

Sainsbury’s: It’s all about the Nectar card for Sainsbury’s - the biggest loyalty scheme of all with 18 million users. It’s also started rolling out smart trolleys to track shopper habits and has just invested in a giant data warehouse. Currently the biggest click & collect supermarket. Big data potential 8/10

Asda: No official loyalty card to speak of, but last month Asda rolled out a new credit card with a 0.5% cashback incentive for shoppers who paid for their groceries with the new card - a loyalty card by any other name. Also has access to Walmart’s data. Loves fun technology. Big data potential 7.5/10

Waitrose: Major problems after it launched a brand new website last year - forcing it to temporarily shut its customer forum to stem the tide of online complaints - not the sort of big data it would like. Has a MyWaitrose loyalty card and is the biggest fan of scan & shop personal scanners. Big data potential 6.5/10

Morrisons: No loyalty card. No online grocery, even. But arguably, that makes Morrisons the one to watch. It’s trialling “targeted coupons” offering customers deals based on the contents of their shopping basket - so it has finally dipped a toe in the water. What next? Big data potential 5/10

That noise is one of the big challenges of big data. It’s chaotic and messy compared with the information derived from a neat grocery shopping session. “Online shopping is far richer in terms of valuable data about customers individual preferences than social media,” says Winsor of online shopping. “And supermarkets are struggling with that online grocery customer data right now, let alone the rest of it.”

But social media data is still valuable, says Cindy Etsell, commercial marketing manager at SAS. Especially once you clean it up. “If you put garbage in, you get garbage out the other side,” she says. “The key is to extract the really useful information, harness it and use it effectively. You just have to make sure it is clean so you can talk to it, and there are so few retailers able to do that. Social media is just adding to the pile. The mobile area is adding to it too - it’s becoming more and more difficult to handle the increasing volumes of data involved. A lot of retailers don’t know where to start. It’s a mega challenge.”

One that retailers can begin to meet by applying high performance analytics to their data. The fun starts once the data starts “talking back”, as she puts it. “I want to walk into a store and get a message saying ‘Great to see you again, Cindy. I know four weeks ago you bought laundry detergent, so you’re probably running low. Here’s a special price on it today.’”

In short, a retailer can start targeting customers with personalised offers it knows they really want. So Sainsbury’s could use its Nectar data to establish that a shopper loves pizza, for example, and use external big data to establish that they drive past Sainsbury’s every day at around 6.30pm. Then it could combine the two data sets and ping that shopper an offer for pizza at 6.15 pm when his stomach starts to rumble and he starts to think about stopping off somewhere to pick up some dinner.

It’s genius. But it’s also a little bit creepy, which is where the other big issue with big data lies for many people. “Location data can be used to get customers into the store and prompt purchasing, but it also raises ethical dilemmas,” says Duncan Ross, director of data science at Teradata. “You can learn an incredible amount about people through this rich big data set that is generated nowadays, but perhaps it’s not a case of can I, but should I?”

Recent studies in the US indicated that people were prepared to divulge really personal data for just 60c, he says: “It’s surprisingly little.” But US retailer Target learned the hard way what happens if you get too close - and it was just using the customer data it had already, rather than any supplementary data.

While retailers are generating “actionable insight” from their analysis of big data, says Winsor, acting on it is proving difficult, often because the infrastructure is simply not there. “For a lot of retailers, the customer insight department is Bob on the third floor,” he says. “I’m exaggerating, but there are companies out there that have surprised me with the lack of organisation in place. There is a definite skills shortage. It needs investment and organisation. Whatever happens in the next five years, the lack of data is not going to be a problem. It’s the management of that data and how retailers use it. The challenge is going to be getting properly set up so they are ready to use it.”

“Big data can be truly revolutionary in terms of wallet share, margins and inventory if retailers can master unstructured data”

The good news is that there’s a wealth of data that if used effectively through relevant, personalised, well timed and therefore welcome promotions, will boost customer retention - and retail revenues, says Etsell.

“A retailer might have millions of customers with millions of variables, which can create huge amounts of scrambled data,” she elaborates. “But we can analyse it in three minutes. A 1% lift in retention for a retailer with millions of customers could equate to £120m in additional revenue every year, so making these improvements can have a major effect on sales. And retailers are struggling right now, they are pushing cost out of their stores, their warehouses, their people and the supply chain. So now is the time to get in there.”

Etsell believes a “gradual turnaround” is taking place in terms of retailers realising the potential of big data. “Then it’s going to go exponential,” she says. “If you know your customers inside out, that is the store of the future - one based on thorough data analysis of your customers’ personal preferences. Customers want you to talk to them personally and companies that don’t get it right will be left behind. And if you aren’t going to rise to the occasion, then what is the point?”

Tesco won’t be caught napping. In June, CEO Philip Clarke told the Consumer Goods Forum that in-store Wi-Fi would let it combine Clubcard data with smartphone data to target shoppers with “suggestions about what to buy” and “point out offers” as they wander the aisles. The era of “mass personalisation” had arrived, he said, offering up “the most valuable asset of all” as a reward - shopper loyalty. Tesco would also be “embracing social media”, he added. “We’re not simply following the customer, we’re standing alongside them, making suggestions, being their loyal partner. And that, ladies and gentlemen, is the way to thrive.”

In an environment where most retailers are currently focused on surviving rather than thriving, big data could make the big difference.

When supermarkets get too personal

US supermarket Target hit the headlines in February when it crunched its customer data to target women it believed were in the early stages of pregnancy - or, as a journalist at put it, “data-mine its way into your womb”.

First, Target analysed transactional data of customers it knew were pregnant to establish buying patterns that corresponded to each stage of pregnancy. It found a high proportion of pregnant women bought a high number of bottles of unscented lotion at the start of the second trimester.

It then laid this buying pattern across all its female shoppers. Those who displayed similar shopping patterns were targeted with ads and coupons for baby products. Clever stuff, and it’s hard not to have a degree of admiration for the customer insight team that dreamed up the idea.

But a negative PR storm erupted after one irate father discovered his teenage daughter was pregnant as a result of the scheme.

Storming into his local Target, the angry father confronted the store manager and demanded to know why his teenage daughter was being sent the coupons.

“My daughter got this in the mail!” he reportedly yelled. “She’s still in high school, and you’re sending her coupons for baby clothes and cribs? Are you trying to encourage her to get pregnant?”

The unfortunate store manager was baffled. But the upset teenage girl eventually ‘fessed up, the mystery was solved and Target came under fire for what was generally seen as a sinister level of snooping.

Target went on the defensive. “Like many companies, we use research tools that help us understand guest shopping trends and preferences so that we can give our guests offers and promotions that are relevant to them,” said a spokesman, in a statement that sounded a lot like ‘don’t just single us out, everyone is at it’.

Perhaps they are right. But perhaps it’s not up to a supermarket to tell someone they’re going to be a grandparent.