Article | May 2015 | Kanvic Grey Matter
Many executives in India are adopting a “wait-and-see” approach regardless of the potential they see in Big Data. However organisations can begin the journey toward building their Big Data capabilities by getting a clearer understanding of the opportunity and of the type of skills required to succeed.
Big Data has been heralded as the next game changer in India’s increasingly digitised economy. Tales of Google predicting flu epidemics or Amazon boosting sales through its powerful recommendation engine are spreading in the media and in business circles. But many executives in India are adopting a “wait-and-see” approach regardless of the potential they see in Big Data. In most cases this inertia is due to a lack of clarity about how and where to start rather than decisional or operational hurdles. This article seeks to provide a helping hand to executives aspiring to benefit from Big Data but who have yet to take the plunge.
Simply put, businesses developing Big Data capabilities want to generate and monetise insights from large amounts of seemingly disparate information. This definition is useful to capture the essence of the concept. However, to help executives unlock the Big Data opportunity it is important to explain what the key characteristics of big data are, why big data analytics differs from other methods, and what skill sets their organisation will require to implement a successful Big Data initiative.
As implied by the name, Big Data deals with big sets of information that cannot be processed by a statistician or an analyst using traditional tools like spreadsheets, or database query systems. The amount of data is a function of three elements:
Scale: How many entities are described and analysed? (e.g. the number of customers registered in a database)
Scope: How many attributes describe these entities? (e.g. the amount of information captured about each customer)
Speed: How fast does new information flow in? (e.g. the frequency at which information on customer purchasing behaviour is updated)
As the amount of information processed by an organisation grows, the potential to benefit from Big Data increases. However, Big Data is not restricted to companies processing quintillion bytes of data. If we take the example of retail, we can position different types of players along a scale. At the higher end of the scale we would find E-commerce businesses. These organisations deal with tremendous amounts of data, recording every click from tens of thousands of visitors each second.
At the lower end of the scale we would find certain brick and mortar retailers recording the daily purchase behaviour of thousands of loyal customers. These companies are dealing with different amounts of information, but in both situations their analysts would struggle to find insights using conventional methods. As long as this is the case, Big Data capabilities would benefit the organisation, even though - as we will see later in the article - the nature of the skills required to build such capabilities would be different.
Besides the amount of information processed, Big Data analytics differs from traditional business intelligence techniques in three fundamental ways.
First, Big Data analytics is particularly effective in finding hidden or counterintuitive patterns that other forms of analysis would easily overlook. A common method used in the field is called data mining. It refers to the use of complex algorithms to explore a dataset and automatically detect notable relationships between variables. This technique does not require any prior knowledge or hypothesis about what the analyst is looking for. This comes with both advantages and drawbacks. On the one hand, it enables more scope for interesting and unusual discoveries. But on the other, it can induce false conclusions about causality leading executives in the wrong direction.
Second, Big Data analytics is a powerful tool to process unstructured datasets with incomplete, misspelled or unstandardised entries for example. This makes it different from other types of tools that require information to be stored in a precise format for it to be useful. Where traditional analysis would discard messy information, Big Data can include it to add precision and increase the scope of the insights generated.
Finally, Big Data also enables automation of certain decisions. If a model has the right information to predict something very accurately on a consistent basis, human supervision becomes unnecessary and can thus be avoided. A recent example of this in India is HDFC bank which has started granting loans to its existing customers by processing applications automatically online. Instead of relying on human operators, the bank is now using an algorithm to grant loans based on data relating to an applicant’s credit rating, demographic profile and lifestyle. Through this initiative, HDFC is expecting major gains to its bottomline. Loan applications can be automatically processed round the clock in a matter of minutes which increases the company’s capacity substantially and also reduces workforce inefficiencies. Additionally, the algorithm offers better reliability in predicting an applicant’s solvency, and is not prone to human errors and biases that can result in higher rates of defaulting customers.
Executives are sometimes puzzled by Big Data because the concept is described differently depending on who is talking about it. Technically oriented people will speak of cloud storage, exabytes, and complex server infrastructures, making Big Data seem like a job for the IT department. Maths savvy people will describe algorithms and regression models making business analysts look best suited for the task. While customer centric people will explain Big Data in terms of tailoring content and offers to each individual’s needs, advocating that it is the domain of the marketing department. In this context top executives are often left wondering how to fit Big Data projects into their organisation’s plan and who should lead such initiatives.
In reality Big Data requires all of these perspectives at once, and being successful at leveraging it requires teams with complementary skills. The fundamental competencies to be combined all relate to three principal areas:
IT: Strong IT infrastructure is required to capture, store, and share large amounts of data across the organisation. This is the engineering part of the Big Data issue.
Data science: Experts in mathematics and computing will be responsible for the effective use of data, models and algorithms. This is the insights generating part of Big Data.
Business acumen: People with functional expertise and market knowledge will frame the problems to be solved and the available options. They are required to translate insights into action and results.
The balance of capabilities required will depend on the objective being pursued and the amount of data that needs to be processed. Taking our earlier example, an e-retailer will require the most complex infrastructure and a large team of data scientists to be able to automatically send tailored recommendations to site visitors. On the other hand, a brick and mortar retailer with no online presence might not need such sophisticated infrastructure. A team of business savvy marketers supported by a few data scientists could generate valuable insights, using only conventional computers connected to an existing intranet.
Top executives can start framing a high level Big Data game-plan for their organisation. To kickstart the board level discussions and give their plans a coherent shape they should consider organising a management roundtable on the topic. Executives from key departments such as I.T., marketing and operations should take part in the conversation because Big Data is likely to impact the organisation across functional boundaries.
The objective of the sessions should be to explore the opportunities or problems Big Data could help address, the potential roadblocks along the way, and the areas of uncertainty that would need to be clarified before going forward. To make the meetings productive, the moderators should take time to understand what their competitors are doing in Big Data and how pioneers from other industries and markets have successfully implemented such initiatives.
In conclusion, organisations can begin the journey toward building their Big Data capabilities by getting a clearer understanding of the opportunity and of the type of skills required to succeed. Leaders wishing to move their organisation in this direction should also consider initiating discussions between top executives as early as possible. This will help them give shape to and refine their vision of the role Big Data could play in their organisation.