How to build a seamless data acquisition system for 400+ GB data/month across 35 countries and 2300+ partners?

From our experience working with multi-national retailers, we’ve learned that everyone understands the value of data. But where they all struggle is in acquiring quality data, completely and consistently.

The first step to any business analytics initiative is acquiring data. Not just any data—but data that is of good quality, complete and in context, and can be received consistently over a period of time. This is difficult due to various reasons: Sources are many, formats are disparate, availability is inconsistent and synchronization sub-optimal. Overcoming these challenges and setting up a seamless data acquisition mechanism that functions at scale is an enormous endeavor.

In this blog post, we’ll walk you through the work we did for one of our customers—multi-billion-dollar global consumer goods company—explore ways in which you can apply them to your business.

Step 1: Streamline your processes

If you’re going to scale your data acquisition program to your global footprint, it’s best to begin with a smaller pilot. It helps conduct assessments, build frameworks, test in the real world and learn quickly from mistakes. This is what we did for our client—we began with market assessment and pilot for one of their key markets.

We built the data acquisition process and the corresponding models to set up TradeEdge as the default consolidator of all their secondary data across emerging markets. TradeEdge also expanded its footprint in developed markets like Europe and Japan.

Step 2: Implement your data acquisition strategy

Technology for data acquisition

TradeEdge Market Connect can be customized to acquire information from across myriad sources and formats. For instance, for this client, we implemented TradeEdge Market Connect to acquire near real-time data from distributors, consolidators, retailers, customer DMS and online retailers. This includes data about secondary sales, inventory, retailer POS, retail execution and e-commerce analytics across formats such as spreadsheets, documents, xml, emails etc.

Checkpoints for data validation

Once the data is acquired, it’s important to validate that data. TradeEdge Market Connect enabled master data validation, mapping data over AS2 / sFTP / email, transforming external data from across partner ecosystems to seamlessly fit into our client’s context.

Hosted data exchange

Entirely on secure cloud, TradeEdge Market Connect served as the single-pane view—a hosted data exchange—enabling flexible data acquisition though a configurable rules engine, performing data cleansing, validation, and transformation.

Step 3: Monitoring and scale

Once the pilot program is successful comes your bigger challenge—scale. Getting all your global channel partners onboarded to your new system often ends up being a complex endeavor. Infosys’ award-winning managed services teams help retailers scale their data acquisition program by onboarding channel partners effectively.

For this client, Infosys’ managed services team offered 8×5 operations support for on-boarding distributors, issue resolution, and follow-ups with distributors for timely data submission.

Step 4: Enabling visibility

Once the data is acquired, you need dashboards that help you make sense of the data. TradeEdge extracted periodic reports—daily, weekly and monthly—for six different applications used by the client’s team, including those for business intelligence, sales and operations planning, demand planning, statistical forecasting, promotion, life-cycle planning and local market solutions

Today, TradeEdge Market Connect delivers near real-time visibility into data from 2300+ partners, across 35 countries. From 8 markets and 80GB of data per month in 2013-14, TradeEdge now supports five times the data with 96% SKU-to-data mapping. With near real-time visibility into secondary sales and inventory data, the supply chain teams optimized must have stock levels (MSL) and performed more effective demand planning.

You can read all details of this case study here.