Introduction to Event-driven Architectures With RabbitMQ

Event-driven architectures (EDA) gather several useful patterns to deliver maintainable code, handle asynchronous tasks and build reliable applications.

As a developer at Theodo, I have been working with various companies to help them build great products. I frequently used event-driven architectures along the way, and I want to share my thoughts with you on this subject.

This article is a conceptual overview of some characteristics of event-driven architectures. For a hands-on tutorial on how to implement your own EDA with RabbitMQ, my advice is to go see RabbitMQ excellent tutorials.

What is an event-driven architecture?

Though it looks basic, this question is quite tricky. I recommend a very good article on this subject, by Martin Fowler: “What do you mean by event-driven?” (if you prefer videos, Martin Fowler also presents the content of the article in the following conference). Some key takeaways from this article:

• “Event-driven” is quite an ill-defined expression that gathers very different patterns
• There exist at least four useful design patterns related to events: Event Notification, Event-carried State Transfer, Event-Sourcing, and Command Query Responsibility Segregation (CQRS)

Each one of these patterns is of interest and deserve further reading! In the following parts of the article, we are going to present more in detail the Event Notification pattern.

Learning by example

We are going to illustrate the Event Notification pattern through an example. Let us pretend that I just bought a new item on an e-commerce website, also asking to save my credit card for further purchase.

When I click the “Buy” button, a payment request will be submitted, and hopefully will succeed. A payment success is a significant change in application state: in other words, an event (following Wikipedia’s definition). Following this event, the website should trigger several actions, for example:

• Create a new shipping order for the seller
• Save my credit card for further use
• Send me an order confirmation email
• Load the order confirmation page on the website

Each one of these actions is independent of the others and must be carried out by different parts of the code: let us say different microservices (abbreviated MS later in this article), as represented in the figure below. 

So the question is: How to notify every MS on the right part of the figure that a “payment_succeeded” event occurred in the Payment MS?

How to notify that the “payment_succeeded” event occurred?

We are going to present two different architectures for this task (spoiler: one of them is what we call the Event Notification pattern — will you guess which one ?): orchestration and choreography.

Event Notification: orchestration vs choreography

The distinction between orchestration and choreography is well captured by this quote by Sam Newman (taken from the very good book Building Microservices):

“With orchestration, we rely on a central brain to guide and drive the process, much like the conductor in an orchestra.

With choreography, we inform each part of the system of its job, and let it work out the details, like dancers all finding their way and reacting to others around them in a ballet”

Orchestration in action

In our example, the central brain would be the part of the code that senses the initial event, i.e. the Payment MS. He would then command hierarchically the other MS’s to carry out the required actions, by sending requests to endpoints specifically designed for this purpose. Therefore, this kind of architecture is called request-driven.

Orchestration pattern (request-driven architecture)

1: The front app sends a request to the Payment MS, informing it that I, customer, clicked the “Buy” button (with the “Save my credit card” checkbox also toggled);

2: The Payment MS handles the request and the payment succeeds;

3, 4, 5: the Payment MS successively sends requests to the Order MS, Messenger MS, and Customer MS to command them to carry out required actions, resp. create a new order, send an order confirmation email, and save the credit card;

6: The Payment MS can finally send a response to the front app request, to load the confirmation page.

Choreography in action

As an alternative, choreography is a way more decoupled way to pass the information from the Payment MS to the others.

The Payment MS knows that a payment succeeded. But is it its responsibility to know the list of all the tasks that must be triggered following a successful payment? Not really. It is rather the responsibility of the Order MS to know what to do on its side when a payment succeeds. The same thing holds for the Messenger and Customer MS’s. And here comes the… Event Notification pattern, a.k.a. choreography! 🎉

Following this pattern, the Payment MS will just notify that a “payment_succeeded” event occurred, using a messaging system. The messaging system will then deliver this message to all the MS’s that need to take action. Each MS will finally consume the message and trigger the corresponding actions.

The interesting part is that the Payment MS does not need to know which actions will be triggered in other parts of the code following the event notification. This ensures a high level of decoupling between the MS’s.

Choreography pattern (event-driven architecture)

1: The front app sends a request to the Payment MS, informing it that I, customer, clicked the “Buy” button (with a “Save my credit card” checkbox also toggled);

2: The Payment MS handles the request and the payment succeeds;

3: The Payment MS publishes a message “payment_succeeded” in a messaging system;

4: The Order, Messenger and Customer MS’s are notified asynchronously that the event occurred by receiving the message (= the “event notification”). In parallel, the Payment MS can directly send the response to the front app, without waiting for the three MS’s to carry out their tasks.

Diving into the Messaging System: RabbitMQ Concepts

The central element of the Event Notification pattern is the messaging system.

Several solutions exist to handle messages: we will not answer the question “How to choose your messaging system” in this article, but if you want some input on this question, the most used solutions as for today are: RabbitMQ, Kafka, or Amazon SQS. Here is also a story on a development team who tells how they made their choice. The one I know better is RabbitMQ, so let me give you a brief overview of how it works.

RabbitMQ follows a protocol called AMQP (Advanced Message Queueing Protocol), which defines a standard way for systems to communicate through messages. The main concepts are the following ones:

• Information in messaging systems is carried by messages, which contains attributes (like headers in a request) and a payload (the message content).
• Messaging systems receive messages from publishers (applications that produce the message) and route them to consumers (applications that process the message).
• Messages are published to an entity called an exchange: this is the mailbox inside the messaging system, where the publisher drops the message.
• Exchanges then distribute the messages to queues, following FIFO order (this applies to RabbitMQ but not all messaging systems).
• Consumption: messages that are stored in queues are then either delivered continually to consumers who subscribed to queues, or fetched from queues by consumers on demand.
• Routing: the rules for delivering the messages to the right queues are defined through bindings (links between exchanges and queues) and routing keys (a specific message attribute used for routing)

For a more detailed presentation of the inner working of RabbitMQ and other AMQP-compliant messaging systems, you can go have a look at this very good article!

Event notification implementation with RabbitMQ

Now that we know the basic concepts of RabbitMQ, let me show you how we implemented them in our previous example.

RabbitMQ configuration

The following figure describes how you can configure RabbitMQ:

• First, we created an exchange called “ms.payment”. Giving your exchange the name of the MS that publishes on it is not mandatory. However, this is a nice practice which makes it very simple to understand and maintain.
• On the right (green part of the figure), we defined one queue per task that should be carried out we events are consumed. We also prefixed each queue name by the name of the MS that will consume the message. Again, this is not mandatory but this is a good option.
• Finally, we declared bindings between the “ms.payment” exchange and the three queues, so that when a message with routing key “payment_succeeded” is published on the exchange, it gets routed to the three queues

Publishing and consuming messages

Finally, here is the full picture:

1. The Payment MS publishes a new message with the routing key “payment_succeeded” on the “ms.payment” exchange;
2. Thanks to the bindings we declared between the exchange and the queues, 3 copies of the message are created and distributed to the queues;
3. Consumers that registered to the queues get their copy of the message and trigger the execution of the task they must carry out.

Quite simple, isn’t it?

Final thoughts: advantages and limits of Event Notification

In conclusion, when should we implement the Event Notification pattern? Let me wrap some pros and cons to help you determine if your use case is a good candidate for it:

Pros

• High decoupling: using messages enables a high level of decoupling between publisher and consumers since the publisher does not have to know anything about who will consume the message nor what actions will be triggered.
• Asynchronous task handling: in our example, the actions following payment success are independent (from a business perspective; e.g. we do not want to check if the confirmation email was sent before saving the credit card). Event notification is a way to carry out all of these actions in parallel, asynchronously.
• Increased reliability: if for some reason a consumer falls temporarily, the queue acts as a buffer. It stores the messages until the consumer gets back and starts processing them again.
• Setting up retry queues: though not covered in this article, an interesting feature of messaging systems is to set up retry queues. They enable to automatically publish again a message in a given queue after a certain amount of time if processing failed. For example, if a consumer must get information from an unreliable API, this feature allows retrying several times in case the API is not responding.

Cons

• High decoupling: yes, decoupling can also be a limit! Sometimes, business logic just wants things to be coupled: did you notice in our article example that the front app communicates with the Payment MS through a request, and not a message? That is because the front app must know if the payment succeeded or not before loading the next page… So event notification is good only if the publisher does not care about the response!
• Implementation overhead: setting up RabbitMQ is a little bit of implementation work. However, the process is well documented, and I think if you see good use cases for event notification (generic events that trigger quite many different actions in separate parts of the code), setting up a messaging system is definitely worth the shot.

I hope this article was clear and useful for you, and that you will enjoy writing nice code using events in your app! Please leave comments if you have questions or remarks.

Want to use Event-Driven Architecture on your project back-end? Feel free to contact one of your Python back-end experts!