Big Data Implementation for Advertising Channel Analysis in 10+

The Customer is a leading market research company.

Though having a robust analytical system, the Customer believed that it would not be able to satisfy the company’s future needs. Acknowledging this situation, the Customer was keeping their eyes open for a future-focused innovative solution. A system-to-be was to cope with the continuously growing amount of data, to analyze big data faster and enable comprehensive advertising channel analysis.

After deciding on the system’s-to-be architecture, the Customer was searching for a highly qualified and experienced team to implement the project. Satisfied with a long-lasting cooperation with DataDesignCenter, the Customer addressed our consultants to do the entire migration from the old analytical system to a new one.

During the project, the Customer’s business intelligence architects were cooperating closely with DataDesignCenter’s big data team. The former designed an idea, and the latter was responsible for its implementation.

For the new analytical system, the Customer’s architects selected the following frameworks:

Apache Hadoop – for data storage;

Apache Hive – for data aggregation, query and analysis;

Apache Spark – for data processing.

Amazon Web Services and Microsoft Azure were selected as cloud computing platforms.

Upon the Customer’s request, during the migration, the old system and the new one were operating in parallel.

Data preparation

Staging

Data warehouse 1

Desktop application

The system has been supplied with raw data taken from multiple sources, such as TV views, mobile devices browsing history, website visits data and surveys. To enable the system to process more than 1,000 different types of raw data (archives, XLS, TXT, etc.), data preparation included the following stages coded in Python:

Data transformation

Data parsing

Data merging

Data loading into the system.

Apache Hive formed the core of that module. At that stage, data structure was similar to raw data structure and had no established connections between respondents from different sources, for example, TV and internet.

Similar to the previous block, that one also based on Apache Hive. There, data mapping took place. For example, the system processed the respondents’ data for radio, TV, internet and newspaper sources and linked users’ ID from different data sources according to the mapping rules. ETL for that block was written in Python:

The new system enabled a cross analysis of almost 30,000 attributes and built intersection matrices allowing multi-angled data analytics for different markets. In addition to standard reports, such as Reach Pattern, Reach Ranking, Time Spent, Share of Time, etc., the Customer was able to create ad hoc reports. After the Customer selected several parameters of interest (for example, a particular TV channel, group of customers, time of day), the system returned a quick reply in the form of easy-to-understand charts. The Customer could also benefit from forecasting. For example, based on expected reach and planned advertising budget, the system would forecast the revenue.

At the project closing stage, the new system was able to process several queries up to 100 times faster than the outdated solution. With the valuable insights that the analysis of almost 30,000 attributes brought, the Customer was able to carry out comprehensive advertising channel analysis for different markets.