Scaling in CPG and
Retail

A Spotlight on Scalability

While rapid change is routine for the retail and CPG industry, the last two years have dramatically increased the need for agile innovation. The rise of startups, supported by trends like healthy foods, organic produce, and a preference for environment-friendly products, has compounded the need for corporations to transform their operations. With sales channels diversifying from mom-and-pop stores and supermarkets to online marketplaces even for the basic items, enterprises must assimilate, unify, and analyze an explosion of data from disparate channels. The exponential complexity and sheer volume of data are matched only by the growth in another equally crucial factor – competition.

In an already competitive space, CPG and retail companies are under substantial pressure to transform their operations or risk being upended by any of the many agile startups at their heels. The ease of expansion to new markets is an example of this digital-driven opportunity, but, on the flip side, it has also removed the barriers to entry and opened up the space for new players. These entrants have the advantage of starting digital-first, sidestepping the many years of evolution that more established players are yet to build upon. These disruptions are forcing large corporations to look at rapid scaling, upwards or sideways. Agility is now a prerequisite to survive and thrive.

Constrained by the Traditional

The inherent limitations of conventional systems inhibit the performance of a variety of critical enterprise functions, especially responsiveness and flexibility. Functions constrained by these traditional systems include distributor onboarding and data processing, especially with multi-format data from disparate organizations. The status quo still sees manual and semi-automated processing of the order and invoicing data. If allowed to continue, this state of affairs can increase order fulfillment time, consequently inflating the time to market and time to reach the customer, both of which can cause a loss of market share.

Business managers need to quickly onboard more distributors and partners, which conventional and non-digital solutions do not support. Also, with the visibility essential to supply chain integrity, CIOs and data engineers must have a single source of truth for sales data across distribution channels. Additionally, to drive enterprise-wide cloud adoption at scale, CTOs need to provide business users agility that powers scale and ease in order and invoice processing operations. Each of these needs requires a scalable, cloud-based solution. Despite the urgency of this need, enterprise leaders remain indecisive for various reasons.

A Question of Confidence

Considering all the signs point to a more digital CPG enterprise, why do some leaders remain reluctant to implement this approach? In our experience, factors include:

• A lack of confidence in claims made by solutions
• The absence of awareness of solutions that offer enterprise-grade technology in SAAS offerings
• Concerns that a solution’s scalability may not match their rate of data ingestion and processing needs

By digitizing all information that flows in and out of the enterprise, transitioning to a digital-driven ecosystem also opens up many opportunities by extracting insights from data gathered. Consider the case of processing distributor orders and invoices from distributors. An appropriate automation platform could create more streamlined processes while lowering costs. It can also gather, integrate, and analyze data to develop agility in decision-making, quicken product launches, and improve the speed of responding to market changes. Choosing the right platform and implementation partner is integral to realizing these benefits. Let s look at the attributes of the ideal solution.

Doing it Right

The most important enabler of a reliable business solution is the proper tooling powered by scalable technology, combined with the right processes. Besides features capable of addressing every aspect of the value chain, the correct solution must also handle spikes in workload and volumes without compromising service levels. It is essential to consider several factors such as the agility of the solution, its track record of driving measurable results, scalability, modernity, with the least possible overheads of maintenance. We can categorize the salient features of the ideal solution under the following segments:

• Cloud and SaaS
  The best solutions must leverage everything that the latest technology offers. They are available through a SaaS model, hosted on the cloud, and make the best of its elasticity and other native services. A cloud-based solution offers flexibility, agility, and scalability. It also minimizes time to value, cutting down installation time and reducing IT procurement and maintenance overheads, yielding significant savings over the medium-long term. Also, by offering rapid expansion, these solutions can handle the periodic spikes typical of the retail and CPG industry, for example, the year-end holiday season, etc.
• Product Performance
  The architecture of the product, keeping in mind its ability to leverage the best of the cloud and its ability to drive scale and performance, ultimately determines whether it can handle consistently high workloads while maintaining service levels. Products must be multi-tiered, with each tier capable of scaling as required. Providers must tune them to perform under specific workloads typical of CPG and retail. These can include multi-file format transformations to more common formats, ingestion of large files, and quick record processing while providing the flexibility to generate customized output. Also, housekeeping activities such as logging, and audit tracing should not affect the performance of business processing. With traditional database servers compromising the flexibility to scale, increasing maintenance and administration overheads, cloud-native services such as DBaaS could help lower costs and offer agility. Just as a product requires sound design, these features need the suitable configuration to deliver at their best, which brings us to our final parameter.
• Optimum Configuration
  The underlying infrastructure and the design of cloud-native services are essential to the performance of the product. Critical configuration parameters include database pool settings, maximum connections, file descriptors, and maximum TCP connections. As workloads keep changing, their parameters require tuning. A cloud-based SaaS solution relieves the CPG company of such nuances while assuring similar service levels, scalability, agility, and speed that are alluded to throughout this piece.

What Makes a Solution Scalable?

The design principles of highly scalable applications include identifying and segregating models based on whether they are supporting services or process-heavy services. These principles enable a horizontal scale-out of the compute and data-intensive modules.

The ideal solution processing approach will allow applications to work consistently across multiple smaller files or even large standalone files. Another critical aspect of a super scalable solution is a workflow manager, an orchestrator that ensures services work in tandem with each other, asynchronously, eliminating bottlenecks. All instances of a given service and different services are then tied together through queuing systems, and each service, in turn, is a combination of a reader, processor, and writer, allowing each service to read from the queue, process data, and write the output back into the queue independent of other services. Since the architecture is staged and event-driven, it presents the flexibility to scale a particular processing engineer horizontally as required, depending on the workload increase or decrease. These application features facilitate a highly scalable structure integral to the rapidly changing workloads typical of a CPG company. With this framework in place, companies can adapt to spikes and dips in demand swiftly and also have the option to automate the scale.

Scalable solutions sidestep the pitfalls of traditional systems by consuming on-cloud native services such as DBaaS to leverage the elasticity of the cloud. In stark contrast with old generation on-premise systems, solutions on the cloud also provide more options such as the choice of the right kind of file storage infrastructure fit for file transfers and the optimal processing power, allowed by the flexibility of cloud infrastructure. Additionally, given the variations in data volumes and formats, enterprises should look for a cloud solution that can efficiently process data while scaling horizontally and vertically based on present and future needs.

Enterprise looking to pivot to a cloud-based solution must ensure that their choice ticks the right boxes. For instance, CTO’s must evaluate whether the solution has been tested through various benchmarks and scalability tests on popular cloud platforms such as Microsoft Azure or Google Cloud Platform. The solution provider should also have documented data sheets to aid the optimum configuration for pool sizes, operating system parameters, and network/TCP parameters to support high-speed processing and achieve scalability. As a result of this due diligence in choosing the right solution, enterprise clients can get the best of both worlds – a highly scalable software solution and the flexibility of the cloud while eliminating the overheads of management and administration of the environment.

Building a Digital CPG Organization

If chosen correctly, technology is not an expense but a force and growth multiplier for businesses. CPG and retail enterprises are sitting on a wealth of data that can be leveraged to draw insights for greater agility, better inventory management, promotions planning, product launches, and even for seasonal demand. The industry must start looking at order fulfillment and supply chain optimization with a laser focus on agility and scalability supported by data-driven insights. Over time, organizations quick to adopt this approach can develop an intuition for consumer trends and behavior, earning a massive competitive advantage.