Big Data has been heralded as the next game changer in India’s increasingly digitised economy. Tales of Google predicting flu epidemics or Amazon boosting sales through its powerful recommendation engine are spreading in the media and in business circles. But many executives in India are adopting a “wait-and-see” approach regardless of the potential they see in Big Data. In most cases, this inertia is due to a lack of clarity about how and where to start rather than decisional or operational hurdles. This article seeks to provide a helping hand to executives aspiring to benefit from Big Data but who have yet to take the plunge.
Simply put, businesses developing Big Data capabilities want to generate and monetise insights from large amounts of seemingly disparate information. This definition is useful to capture the essence of the concept. However, to help executives unlock the Big Data opportunity it is important to explain what the key characteristics of big data are, why big data analytics differs from other methods, and what skill sets their organisation will require to implement a successful Big Data initiative.
First of all, how “big” are we talking?
As implied by the name, Big Data deals with big sets of information that cannot be processed by a statistician or an analyst using traditional tools like spreadsheets, or database query systems. The amount of data is a function of three elements:
Scale: How many entities are described and analysed? (e.g. the number of customers registered in a database)
Scope: How many attributes describe these entities? (e.g. the amount of information captured about each customer)
Speed: How fast does new information flow in? (e.g. the frequency at which information on customer purchasing behaviour is updated)
As the amount of information processed by an organisation grows, the potential to benefit from Big Data increases. However, Big Data is not restricted to companies processing quintillion bytes of data. If we take the example of retail, we can position different types of players along with a scale. At the higher end of the scale, we would find e-commerce businesses. These organisations deal with tremendous amounts of data, recording every click from tens of thousands of visitors each second.
At the lower end of the scale, we would find certain brick and mortar retailers recording the daily purchase behaviour of thousands of loyal customers. These companies are dealing with different amounts of information, but in both situations, their analysts would struggle to find insights using conventional methods. As long as this is the case, Big Data capabilities would benefit the organisation, even though - as we will see later in the article - the nature of the skills required to build such capabilities would be different.
What makes Big Data analytics different from other methods?
Besides the amount of information processed, Big Data analytics differs from traditional business intelligence techniques in three fundamental ways.
First, Big Data analytics is particularly effective in finding hidden or counterintuitive patterns that other forms of analysis would easily overlook. A common method used in the field is called data mining. It refers to the use of complex algorithms to explore a dataset and automatically detect notable relationships between variables. This technique does not require any prior knowledge or hypothesis about what the analyst is looking for. This comes with both advantages and drawbacks. On the one hand, it enables more scope for interesting and unusual discoveries. But on the other, it can induce false conclusions about causality leading executives in the wrong direction.
Second, Big Data analytics is a powerful tool to process unstructured datasets with incomplete, misspelt or unstandardised entries for example. This makes it different from other types of tools that require information to be stored in a precise format for it to be useful. Where traditional analysis would discard messy information, Big Data can include it to add precision and increase the scope of the insights generated.
Finally, Big Data also enables automation of certain decisions. If a model has the right information to predict something very accurately on a consistent basis, human supervision becomes unnecessary and can thus be avoided. A recent example of this in India is HDFC bank which has started granting loans to its existing customers by processing applications automatically online. Instead of relying on human operators, the bank is now using an algorithm to grant loans based on data relating to an applicant’s credit rating, demographic profile, and lifestyle. Through this initiative, HDFC is expecting major gains to its bottom line. Loan applications can be automatically processed round the clock in a matter of minutes which increases the company’s capacity substantially and also reduces workforce inefficiencies. Additionally, the algorithm offers better reliability in predicting an applicant’s solvency and is not prone to human errors and biases that can result in higher rates of defaulting customers.
What is the Big Data skill-set?
Executives are sometimes puzzled by Big Data because the concept is described differently depending on who is talking about it. Technically oriented people will speak of cloud storage, exabytes, and complex server infrastructures, making Big Data seem like a job for the IT department. Maths savvy people will describe algorithms and regression models making business analysts look best suited for the task. While customer-centric people will explain Big Data in terms of tailoring content and offers to each individual’s needs, advocating that it is the domain of the marketing department. In this context, top executives are often left wondering how to fit Big Data projects into their organisation’s plan and who should lead such initiatives.
In reality, Big Data requires all of these perspectives at once and being successful at leveraging it requires teams with complementary skills. The fundamental competencies to be combined all relate to three principal areas:
IT: Strong IT infrastructure is required to capture, store, and share large amounts of data across the organisation. This is the engineering part of the Big Data issue.
Data science: Experts in mathematics and computing will be responsible for the effective use of data, models, and algorithms. This is the insights generating part of Big Data.
Business acumen: People with functional expertise and market knowledge will frame the problems to be solved and the available options. They are required to translate insights into action and results.
