Event-Driven Architecture with MuleSoft
Introduction
In today's digital age, organizations seek real-time updates to stay ahead of the competition and deliver seamless experiences to their users. Traditional synchronous API patterns often fall short in providing real time updates driven by events. Event-Driven Architecture harnesses the power of events, associated states, and timestamps to deliver real-time updates. In this blog post, we will explore the world of Event-Driven Architecture and demonstrate how it integrates with the API-Led design, offering architects and developers a solution for their integration needs.
Problem Statement
When it comes to designing integrations, there is no one-size-fits-all solution. Sometimes, an Event-Driven architecture is the optimal choice, while at other times, fully RESTful synchronous patterns or scheduled batch jobs may be more suitable. Each pattern has its own set of advantages and disadvantages. To fully benefit from these integration methods, organizations need a solution that manages all patterns whether it is RESTful and Event-Driven architectures throughout their lifecycle.
Solution
Let's begin by exploring the API-Led design and how Queues or PubSub-Topics can seamlessly fit into the architecture. There are two primary options for Mule APIs to subscribe to Events: through an Experience API or a Process API.
Key Characteristics of Subscribing with an Experience API
One of the fundamental principles of API-Led design is that external systems should interact with Experience APIs rather than directly with Process APIs. While events may not fit the traditional definition of API calls, they do fall into a bit of a gray area. However, we recommend adhering to the API-Led principle by creating an Experience API that subscribes to the topics.
Using an Experience API allows you to attach an AsyncAPI Specification to the API for documentation and discoverability purposes. Additionally, you can perform the transformation to the Canonical Data Model at the Experience API level, ensuring architectural consistency.
Key Characteristics of Subscribing with a Process API
Subscribing to events directly with a Process API can lead to more efficient resource utilization, potentially using fewer vCores/Cores as logic can be aggregated into a single Process API. However, it's crucial to maintain the principle of loosely coupled services. Therefore, we recommend that a Process API, such as the Order Event Process API, directly calls another Process API, like the Checkout Process API in our example.
Similar to Experience APIs, you can attach an AsyncAPI Specification to a Process API. In this case, the transformation to the Canonical Data Model occurs immediately after subscribing to the Event. Additionally, implementing retry logic can make more sense within the Process API, where it can be tailored to specific requirements.
Retry Mechanism
Asynchronous API patterns, including Event-Driven design, require a robust retry mechanism. In our previous blog post on Asynchronous API Patterns, we distinguished between Connectivity Errors and Data Errors. Building on that knowledge, we will use the same for our Event-Driven Architecture. The following diagram outlines the expected behavior:
An Order Event Experience API subscribes to the Event of the OrderUpdate Topic / Queue.
The message is transformed into the Canonical Data Model and sent to the Checkout Process API, which, in turn, calls the SAP System API to update the Order in SAP.
Connectivity Error: In case of a connectivity error, the message should be retried within the same Topic / Queue.
Data Error: Data Errors do not need to be retried, as subsequent attempts are likely to fail as well. Instead, they should be directly published to the Error Topic or Dead Letter Queue, where operations teams can take manual actions as needed.
Additional requirements for the Retry Mechanism:
Always OOTB first: We always strive to utilize out-of-the-box (OOTB) functionalities from the Event Brokers such as Kafka, Google PubSub, AWS SQS, Azure Service Bus or Rabbit MQ as much as possible. This approach minimizes custom implementations in MuleSoft, ensuring a lean and maintainable integration layer.
Exponential increase of the timeouts between the retries: To prevent overloading the integration, we increase the timeout between the retries after each attempt.
Ordering: We prioritize maintaining the correct order of events, ensuring that a stuck order in the retry mechanism does not overwrite a new order. In the example shown below the OrderUpdate A is stuck in the retry mechanism, where as the OrderUpdate B is already successfully sent to SAP. In this case we want to avoid that the OrderUpdate A will overwrite the Update of B.
Use of Circuit Breakers: Martin Fowler described the advantage of Circuit Breakers already back in 2014. A circuit breaker serves as a mechanism that intervenes when a certain threshold of failures is reached, thereby interrupting a connection. This circuit breaker employs an error notification system to keep track of errors associated with an external service, referred to as "circuit failures." Various types of errors, such as HTTP timeouts or FTP service unavailability, can be attributed to circuit failures. We will dive into how to implement Circuit Breakers with MuleSoft in another blog post.
Conclusion
In this exploration of MuleSoft's Event-Driven Architecture, we've uncovered the seamless integration of events into the API-Led design. Architects and developers can now leverage the power of real-time updates, with options for Experience APIs and Process APIs. The robust retry mechanism, coupled with out-of-the-box configuration, ensures a reliable and efficient integration solution for the dynamic demands of today's digital landscape.
References
Check out the first blog post to learn about the API-Led principles and what's permitted and what's not.
Subscribe to stay up to date!
muletalks.dev