Scaling Large Workflows Part 2: Data Management in Modular Workflows

Moses HardingProduct Owner
Scaling Large Workflows Part 2: Data Management in Modular Workflows
Scaling Large Workflows Part 2: Data Management in Modular Workflows

Data Management in Modular Workflows


Managing data management 

In a previous article, I delved into the advantages of adopting a modular approach to workflow construction over a monolithic one. If you haven’t explored those benefits yet, I recommend reviewing it. However, as we embrace modularity, a distinct challenge emerges – data management.


How things can get complicated 

Data management involves storing and accessing data. Unlike monolithic apps, where all data resides in a single location, a modular approach means that data could potentially reside across multiple apps. This creates complexities in saving and updating data, as a given app may hold data needed by other apps.

For instance, let’s say you have a workflow involving two parties across three apps. In the first app (‘App 1’), a requestor from the business generates a contract, in the second app (‘App 2’), the requestor signs a contract, and in the third app (‘App 3’), a third party signs the contract. In App 1, the user may specify the names of each signatory. That means that App 2 and App 3 need to reference App 1. App 3 may also need to reference App 2. The author now has to create and maintain two data operations to accomplish a single task. In this small example, this might not seem like a big issue, but imagine the complexity involved when we start to build workflows with ten, twenty, or even more apps. In a workflow that size, that level of complexity could introduce as many, if not more, possible points of failure, necessitating a precise understanding of data locations and order of operations.


Using ‘data containers’ 

To address this complexity, I recommend a strategy involving a centralized “data container.” This approach involves creating one app – the data container – that houses all the data needed throughout the workflow. Instead of navigating through multiple apps for data, each app can seamlessly access the singular data container, simplifying data management.

Returning to my above example, in which I mention a workflow with three apps, imagine I restructured it with this in mind. Instead of managing data in three distinct apps, a fourth app – the data container – holds all necessary data for every app. This streamlined approach requires only minimal interactions with the data (a single read and write in each app), eliminating the complexities associated with scattered data.


It’s crucial to differentiate between data used for storing information and those triggering events (or ‘actions’) in Neota. Data that triggers actions, such as emails or API calls, should remain in individual apps and not be stored in the data container. This separation ensures flexibility and avoids complications when modifying data structures.


Making the most of the modular approach to app development

Employing a data container in a modular approach not only simplifies data storage but also enhances overall workflow efficiency. By adopting this strategy, you not only mitigate potential points of failure but also create a more manageable and scalable system, enabling seamless adaptation to evolving business needs.

More Blog Articles
Unlocking Efficiency and Innovation with Workflow Automation

In today’s dynamic business landscape, companies are continually seeking avenues to streamline operations and drive innovation. At Neota, we recognise the pivotal role that workflow automation plays in achieving these goals. Our platform empowers organizations to streamline their processes, enhance productivity, and unlock unparalleled efficiency.   Workflow and Workflow Automation   Simply put, a workflow…

Harnessing AI for Practical Business Automation Solutions

Legal operations leaders face complex challenges in steering their teams towards efficiency and integration, while navigating a sea of technological possibilities including the emerging possibilities of Generative AI (GenAI). GenAI introduces new opportunities for innovation and automation but also brings unique challenges in assessing its implications and integrating its capabilities responsibly. Grasping the opportunity presented…