MassTransit and MediatR – Part 2

Introduction

In this second post on MassTransit and MediatR, we’re going to look at how we can add MassTransit Exponential Back-Off with an exponential back-off. First we’ll talk about what an exponential back-off is, why we should have this and then how we can implement this with MassTransit.

In our last post on MassTransit and MediatR, we went through how you can do the initial setup by adding MassTransit consumers and MediatR handlers then configuring the IoC container to work with both (and also integrating Service Bus). You can find part 1 here.

The full working code can be found here.

MassTransit Exponential Back-Off

What is an exponential back-off?

Let’s say you have a service like ours which is going to receive a BookingCreated message and then call a CarBooking API to actually book a car. If the API is down for some reason, we want to retry this call every so-often till it eventually works. Rather than having all the re-attempts to call the API happen on a standard interval, the exponential back-off increases that interval with every subsequent attempt so that rather than trying every second, we’d try after 1s then 2s then 4s then 8s for example.

Why use an exponential back-off?

So, we could just retry every second or so and that could work but let’s say there’s an issue and we have a lot of traffic. That means we have loads of calls queued up and they’re all retrying every second. The issue is that the downstream API may just be starting up or have a performance issue so as a result, we’re in fact making that worse and it could possibly go down again. The exponential back-off basically allows it time to recover.

Configure MassTransit Exponential Back-Off

If you remember back to part 1, we created some IServiceCollection extension methods to configure MassTransit. Here we’re just going to add the retry configuration.

MassTransit has the concept of an IRetryConfigurator which basically configures the retries and we use cfg.UseMessageRetry() on line 13 to add the IRetryConfigurator to our configuration. We’ll configure exponential retries but you can read more about the other options such as Interval or Immediate here.

    public static IServiceCollection AddBus(this IServiceCollection services, IConfiguration configuration)
    {
        services.AddMassTransit(o =>
        {
            o.SetKebabCaseEndpointNameFormatter();

            o.AddConsumers(typeof(BookingCreatedConsumer).Assembly);

            o.UsingAzureServiceBus((context, cfg) =>
            {
                cfg.Host(new Uri(configuration["MessageBus:ServiceBusUri"]));
                cfg.ConfigureEndpoints(context);
                cfg.UseMessageRetry(r => AddRetryConfiguration(r));
            });
        });

        return services;
    }

    private static IRetryConfigurator AddRetryConfiguration(IRetryConfigurator retryConfigurator)
    {
        retryConfigurator.Exponential(int.MaxValue, TimeSpan.FromMilliseconds(200), TimeSpan.FromMinutes(120),
                TimeSpan.FromMilliseconds(200));

        return retryConfigurator;
    }

If we look at the Exponential() method signature, this code basically means the retry count is int.MaxValue which may not be what you want so have a think about what it needs to be for production and then how you want to handle failed messages but that’s a whole new topic. The rest basically states that the initial retry is 200 milliseconds, the max interval is 2hrs and the amount which the interval increases by is 200 milliseconds.

It’s good to set sensible min, max and delta intervals otherwise the customer could we waiting days for their message to be retried 🙂

Testing MassTransit Exponential Back-Off

Okay, let’s try this out but with a larger interval so we can get more of an idea what’s going on. In order to force the message to fail, I’m going to use Debug.WriteLine() so I can find the times the message was received and simply throw an exception in the consumer after it receives the message:

using Contracts.Messages;
using MassTransit;
using MediatR;
using System.Diagnostics;

namespace CarBooking.Service.Consumers;

public class BookingCreatedConsumer : IConsumer<BookingCreated>
{
    private readonly IMediator _mediator;

    public BookingCreatedConsumer(IMediator mediator)
        => _mediator = mediator;

    public async Task Consume(ConsumeContext<BookingCreated> context)
    {
        Debug.WriteLine($"Message Received: {context.Message.BookingId}");
        throw new Exception();
    }
}

I’m also increasing the initial interval to 10 seconds and max interval to 40s so it’s more clear as there’s the overhead of the time taken to actually process a message:

private static IRetryConfigurator AddRetryConfiguration(IRetryConfigurator retryConfigurator)
{
    retryConfigurator.Exponential(int.MaxValue, TimeSpan.FromMilliseconds(200), TimeSpan.FromSeconds(40), TimeSpan.FromSeconds(10));

    return retryConfigurator;
}

Using the Visual Studio IntelliTrace, we can see the times the message was received. Initial interval is about 10s then 13s then 39s etc. It’s not exactly doubling each time but the interval is increasing at each attempt. Note how the interval then stops getting bigger and caps out at 40s as that’s our max.

Conclusion

In this post, we added an exponential retry to our MassTransit configuration. It’s a great feature that MassTransit has as it saves you having to write your own code library to do this. We also discussed how it’s important to have an exponential retry as opposed to just retrying on an interval and we also mentioned how the retry intervals need thought to make sure we strike the balance between responsiveness for the customer and putting too much load on backend systems.

The full working code can be found here.

In the next post, we’ll take a look at validation with MediatR and MassTransit in the picture and also how to prevent retries in case of invalid messages.