Factory VS Cloud-Native

By Bradley Clerkin

Everyone who is shifting to build new technical solutions in the cloud comes upon the same fork in the road. Do we take how we operate today and shift it to work in the cloud, which will provide some incremental benefit? Or do we completely reimagine how to operate to take full advantage of cloud capabilities, which requires a complete shift in people, processes, and technology? At this point, only a small percentage of data groups are taking the road of reimagining their operations to utilize cloud-native capabilities. Unfortunately, this creates a major disparity between organizations with effective cloud-based data initiatives and companies that use a “factory approach,” which means implementing only minimal operating model changes and maintaining their existing way of working for data and machine learning (ML) projects. I’ve often found that the Seven Wastes of Software Development, described in the table below, are an excellent resource for understanding why status quo data systems and approaches, aka the “factory approach,” inhibit outcomes. Factory vs Cloud-Native If you compare the Factory approach and the Cloud-Native approach at a high level, you would observe the following.

Here’s how each approach looks in a visual comparison. The factory approach creates a big system to handle everything.

In contrast, the cloud-native approach creates a lot of smaller teams that are assigned to source data systems. Factory Approach Observations Going even further, here are some standard observations we receive when talking to leaders about their current factory approach, and which of the seven wastes of software development they fall under. “Our technical teams are working through technology architecture and selection because they only get one chance to get this right for all the potential integrations, storage scenarios, and reporting use-cases. They are telling us it’s going to be six to twelve months before we can start producing answers to our questions

Over Processing, Over Production “Getting information to flow effectively between stages in our process has been difficult. The technology vendors sold us a book of goods around ease of implementation, but we can’t seem to consistently get data flowing from one step to another.”

Transport, Defects, Waiting, Over Processing “Some of our teams, like the data warehouse team, are overwhelmed with work from all the data initiatives, and while other teams, like the reporting team, are left sitting on their hands waiting for the overworked teams.”

Waiting, Inventory, Motion “When we finally get a report or insight, it’s rarely accurate, and we are told troubleshooting is difficult because pinpointing problems is hard in a big system like ours. We are being told we need new data quality tooling to fix this. Meanwhile, we’ve got big questions that need answers.”

Defects, Inventory, Waiting “Our data and analytics teams have a substantial amount of delivery managers who have to help the system function and organize all the work. It’s very complex.”

Over Production, Transport, Waiting Executing a Cloud-Native Approach The following are key indicators to ensure you are executing the cloud-native approach correctly.

1. Stakeholders prioritize source data and then generate two or three insights they think the top source data provides. This acts as the target for a minimally viable output of our first cross-functional team. We scope it to ensure full end-to-end ownership of the output is possible. 2. A cloud platform team is formed to assist operationalizing cloud-based solutions needed for the effort. These solutions are turned on rapidly, and the platform team focuses on guardrails and automation that lower time to production and have minimal impact on velocity. 3. The cross-functional team starts to produce usable reports and APIs for its source data system. If possible, additional data sources are added that can help provide impactful output to the stakeholders. 4. Additional cross-functional teams are formed and use the patterns from the cloud platform team to make more source data insights available via APIs and reports. These teams can also pull information from other cross-functional teams’ APIs to generate more complex outputs and insights. 5. The capacity of cross-functional teams, time to production, the impact of insights, quality of outputs, and efficiency of platform operations are regularly measured and managed by leadership for long-term system operations.

With a cloud-native approach to data and analytic systems, we avoid the waste that plagues the factory approach. Value is produced rapidly, teams own their outputs end-to-end, more complex insights are discovered as the system expands, and with the right support from a cloud platform enablement team, patterns are used to avoid teams solving the same technical architectural problems over and over again. This is the road less traveled, but it produces substantially greater value, impact, and outputs.