Scaling Large Workflows Part 2: Data Management in Modular Workflows

Moses HardingProduct Owner
Scaling Large Workflows Part 2: Data Management in Modular Workflows
Scaling Large Workflows Part 2: Data Management in Modular Workflows

Data Management in Modular Workflows

 

Managing data management 

In a previous article, I delved into the advantages of adopting a modular approach to workflow construction over a monolithic one. If you haven’t explored those benefits yet, I recommend reviewing it. However, as we embrace modularity, a distinct challenge emerges – data management.

 

How things can get complicated 

Data management involves storing and accessing data. Unlike monolithic apps, where all data resides in a single location, a modular approach means that data could potentially reside across multiple apps. This creates complexities in saving and updating data, as a given app may hold data needed by other apps.

For instance, let’s say you have a workflow involving two parties across three apps. In the first app (‘App 1’), a requestor from the business generates a contract, in the second app (‘App 2’), the requestor signs a contract, and in the third app (‘App 3’), a third party signs the contract. In App 1, the user may specify the names of each signatory. That means that App 2 and App 3 need to reference App 1. App 3 may also need to reference App 2. The author now has to create and maintain two data operations to accomplish a single task. In this small example, this might not seem like a big issue, but imagine the complexity involved when we start to build workflows with ten, twenty, or even more apps. In a workflow that size, that level of complexity could introduce as many, if not more, possible points of failure, necessitating a precise understanding of data locations and order of operations.

 

Using ‘data containers’ 

To address this complexity, I recommend a strategy involving a centralized “data container.” This approach involves creating one app – the data container – that houses all the data needed throughout the workflow. Instead of navigating through multiple apps for data, each app can seamlessly access the singular data container, simplifying data management.

Returning to my above example, in which I mention a workflow with three apps, imagine I restructured it with this in mind. Instead of managing data in three distinct apps, a fourth app – the data container – holds all necessary data for every app. This streamlined approach requires only minimal interactions with the data (a single read and write in each app), eliminating the complexities associated with scattered data.

 

It’s crucial to differentiate between data used for storing information and those triggering events (or ‘actions’) in Neota. Data that triggers actions, such as emails or API calls, should remain in individual apps and not be stored in the data container. This separation ensures flexibility and avoids complications when modifying data structures.

 

Making the most of the modular approach to app development

Employing a data container in a modular approach not only simplifies data storage but also enhances overall workflow efficiency. By adopting this strategy, you not only mitigate potential points of failure but also create a more manageable and scalable system, enabling seamless adaptation to evolving business needs.

More Blog Articles
Reshaping What’s Possible

Leveraging tailor-made legal automation.     No-code platforms - and a ‘custom’ approach to automation - are being adopted by law firms and legal teams worldwide due to the control and flexibility they offer. The practice involves using a drag-and-drop visual environment to build solutions without coding, enabling organizations to build tools for their exact…

Transforming Supplier Onboarding using No-code Technology

Driving internal efficiency   In the ever-changing landscape of global enterprises, managing supplier onboarding processes efficiently and effectively is paramount. Traditional methods often fall short, struggling to cope with the sheer volume of suppliers, diverse regulatory environments, and the need for seamless integration with internal processes and third-party systems. This is where the power of…