Building Resilient Service-to-Service Communications in ASP.NET Core with Polly, the .NET Resilience Framework

Time to read: 6 minutes

April 17, 2019
Written by
Bryan Hogan
Contributor
Opinions expressed by Twilio contributors are their own

rwNecMwoSUez0Ube7w9hsk4xKt_AVXOmIPDyu8qHhgvjZZUDl2t5hQyMp0YIclb6XpIdKdRSFRVnhg0Tf78mV7SmPFSWHGpkGMUzF68JTo2D40YC1VlT-auZj3qg-5Ub40shcDCu

Service-to-service communication is becoming more and more common, but we know the network has never been reliable and that other people's code has bugs. Packets are lost, exceptions are thrown, and requests fail; it will happen to your application—to all applications!
Using the Polly Resilience Framework you can easily retry a failed request, no messy for loops or try catches needed. Just a few lines of code and your applications will never be the same!

This post shows how to make reliable requests to an unreliable "remote" service with almost no overhead.

You’ll start with a “remote” Web API application that returns errors 75% of the time. Then you’ll build a “local“ Web API application that calls the remote one (of course receiving errors three quarters of the time).

Finally, you will add Polly to the “local” application to perform retires on failed requests. With a few minutes work you will be making reliable requests to a highly unreliable remote service.

Prerequisites

All you need to follow along with this post is .NET Core 2.0 or higher and you favorite IDE.

You are going to create two Web API projects representing two web services, I recommend doing this in two instances of Visual Studio 2017/2019 (or Visual Studio Code, if that is your preference) because it will make it easier to run the two services side by side, watch how they behave, and monitor breakpoints.

Building an Unreliable Service

Let's build our unreliable "remote" temperature service first. Start by creating a new .NET Core 2.x Web API application, name it TemperatureService. You’re going to make only a few changes to what you got out of the tin:

Rename the ValuesController.cs file in the Controllers folder to TemperatureController.cs.

In the TemperaturesController.cs file, rename the ValuesController class to TemperatureController.

Add the following using directive to the ValuesController.cs file:

using System.Net;

Replace the body of the TemperatureController class with the following C# code:

Route("[controller]")]
public class TemperatureController : ControllerBase
{
        static int _counter = 0;
        static readonly Random randomTemperature = new Random();

        [HttpGet("{locationId}")]
        public ActionResult Get(int locationId)
        {
            _counter++;

            if (_counter % 4 == 0) // only one out of four requests will succeed
            {
                return Ok(randomTemperature.Next(0

The action method of this new controller takes a locationId and returns failures for three out of every four requests. It also uses a random number generator to return values that look like temperatures in degrees Fahrenheit.

The next change is to the launchsetting.json file in the Properties directory. Remove the ”iisSettings” node and replace the ”profiles” node so it looks like the following:

{
  "$schema": "http://json.schemastore.org/launchsettings.json",
  "profiles": {
        "TemperatureService": {
          "commandName": "Project",
          "launchBrowser": true,
          "applicationUrl": "http://localhost:6001",
          "environmentVariables": {
               "ASPNETCORE_ENVIRONMENT": "Development"
           }
     }
  }
}

Note that this will change the Kestrel configuration to start the application listening on HTTP port 6001.

That's all you need to do to make a highly unreliable service.

If you want to test it, run the project and navigate to http://localhost:6001/temperature/222 in your favorite web browser. (You can use any integer value in place of “222” for the locationid.) Keep reloading the page until you get a successful response. Only every fourth request will return a temperature.

Calling the Unreliable Service from Another Service

In your other instance of Visual Studio (or Visual Studio Code) create a weather service that calls the temperature service. But you’re not going to add Polly just yet; instead, the weather service will make requests to the temperature service and hope for the best.

Create a .NET Core 2.x Web api application and call it WeatherService.

Add the Microsoft.AspNet.WebApi.Client NuGet package to the WeatherService project.

Edit the Startup class to add HttpClient to the service collection: you’re going to use dependency injection to pass the HttpClient to your controller. (There are pluses and minuses to this, but that is a different story for a different day.)

Add the following using directive to the Startup.cs file:

using System.Net.Http;

Modify the ConfigureServices method of the Startup class so it looks like the following:

public void ConfigureServices(IServiceCollection services)
{
    HttpClient httpClient = new HttpClient()
    {
        BaseAddress = new Uri("http://localhost:6001/") // this is the address of the temperature service
    };
    services.AddSingleton<HttpClient>(httpClient);

    services.AddMvc();
}

As with the temperature service, you will change the port on which the application runs. Open launchsetting.json inside the properties directory, replace the content with the below. Note that the application is listening on port 5001.

{
  "$schema": "http://json.schemastore.org/launchsettings.json",
  "profiles": {
        "WeatherService": {
          "commandName": "Project",
          "launchBrowser": true,
          "applicationUrl": "http://localhost:5001",
          "environmentVariables": {
               "ASPNETCORE_ENVIRONMENT": "Development"
           }
     }
  }
}

Let's replace the ValuesController with a WeatherController by renaming the ValuesController.cs file and the class. In the constructor, take the  HttpClient as a parameter: it will be used to make the outbound request to the temperature service. The GET method takes a locationId and calls the temperature service.

Add the following using directive to the WeatherController.cs file:

using System.Net.Http;

Replace the body of the WeatherController class with the following C# code:

Route("[controller]")]
public class WeatherController : ControllerBase
{
    private readonly HttpClient _httpClient;

    public WeatherController(HttpClient httpClient)
    {
        _httpClient = httpClient;
    }

    [HttpGet("{locationId}")]
    public async Task<ActionResult> Get(int locationId)
    {
        HttpResponseMessage httpResponseMessage = await _httpClient.GetAsync($"temperature/{locationId}");

        if (httpResponseMessage.IsSuccessStatusCode)
        {
            int temperature = await httpResponseMessage.Content.ReadAsAsync<int>();
            return Ok(temperature);
        }

        return StatusCode((int)httpResponseMessage.StatusCode

Now you’re ready to send requests from the weather service to the temperature service.

Start both applications. Open your browser and navigate to this address: http://localhost:5001/weather/222.

The weather service will call the temperature service, which returns an error, and the weather service in turn returns an error to the browser.

Refresh the page three times: on the fourth request you will get a successful response.

This is not the type of retry you or any of your customers want to use.

Making Reliable Calls to the Temperature Service with Polly

Let's add Polly to our weather service and get everything working reliably.

The first step is to add the Polly NuGet package to the WeatherService project. As of this writing, the current version is 7.1.0.

Add the following using directive to the Startup.cs file:

using Polly;

Replace the contents of the ConfigureServices method of the Startup class with the following C# code:

public void ConfigureServices(IServiceCollection services)
{
    IAsyncPolicy<HttpResponseMessage> retryPolicy =
        Policy.HandleResult<HttpResponseMessage>(r => !r.IsSuccessStatusCode)
            .RetryAsync(3);
    services.AddSingleton<IAsyncPolicy<HttpResponseMessage>>(retryPolicy);

    HttpClient httpClient = new HttpClient()
    {
        BaseAddress = new Uri("http://localhost:6001/") // this is the address of the temperature service
    };
    services.AddSingleton<HttpClient>(httpClient);

    services.AddMvc();
}

A Polly policy is made up of two parts. The first is the handles clause; this specifies the conditions under which the policy becomes active. The second part is the behavior clause; this specifies what the policy should do when it is active. In this example, the handles clause says that the policy becomes active if the HTTP response code is not in the 200 range, and the behavior clause specifies up to three retries will be attempted. (The retry policy retries immediately after a failure response is received, this is fine in many scenarios. If you want to insert a delay before retrying, use the WaitAndRetry policy; this lets you specify an exact time or a delegate to calculate the delay.)

The first statement in the revised ConfigureServices method creates the Polly policy to retry HTTP requests if the HTTP response indicates a failure, as shown below:

IAsyncPolicy<HttpResponseMessage> retryPolicy =
    Policy.HandleResult<HttpResponseMessage>(r => !r.IsSuccessStatusCode)
        .RetryAsync(3);

To keep things simple, this project adds the policy to the service collection directly, as shown below. (You could use a Polly policy registry, but let's leave that aside for now.)

services.AddSingleton<IAsyncPolicy<HttpResponseMessage>>(retryPolicy);

The final lines you inserted add the HTTP client to the service collection as a singleton:

HttpClient httpClient = new HttpClient()
{
    BaseAddress = new Uri("http://localhost:6001/") // this is the address of the temperature service
};
services.AddSingleton<HttpClient>(httpClient);

With just these few lines of code you have the Polly policy and the HTTP client inside the dependency injection container.

Back to the controller.

Add the following using statement to the WeatherController.cs file:

using Polly;

Remember, you are going to use dependency injection to pass the policy to the constructor as a parameter.

Modify the beginning of the WeatherController class in the WeatherService.cs file to match the following:

namespace WeatherService.Controllers
{
    [Route("[controller]")]
    public class WeatherController : ControllerBase
    {
        private readonly HttpClient _httpClient;
        private readonly IAsyncPolicy<HttpResponseMessage> _retryPolicy;

        public WeatherController(HttpClient httpClient, IAsyncPolicy<HttpResponseMessage> retryPolicy)
        {
            _httpClient = httpClient;
            _retryPolicy = retryPolicy;
        }
...

Ellipsis ("...") in code indicates a section redacted for brevity.

Inside the action method is where the magic happens, you make the call by the HTTP client inside the retry policy.

Change the first statement in the Get method to match the following:

HttpResponseMessage httpResponseMessage = await _retryPolicy.ExecuteAsync(() =>
    _httpClient.GetAsync($"temperature/{locationId}"));

You’re done! Nothing else changes.

Start up the two applications, open your browser, and navigate to http://localhost:5001/weather/222. This time you will get successful result on the first attempt.

Woo-hoo! Polly called the HTTP client, received an error response, retried, received an error response, retried, received an error response, retried—and this time got an OK and the temperature.

That’s all there is to using simple retries with Polly. How have you done without it?

Summary of Building Resilient Service-to-Service Communications in ASP.NET Core with Polly, the .NET Resilience Framework

This post introduced you to Polly, the resilience framework for .NET. You saw how quickly and easily you can add fault tolerance to an ASP.NET Core Web API project that calls an unreliable service.

If you are familiar with .NET Core 2.1 and above, you might be wondering if you can use Polly with the HttpClientFactory. You can, and that will be the topic of a future post.

Additional Resources

Companion Repository – The complete code for the projects in this post is available on GitHub and can be reused under the MIT license.

The Polly Project – The project homepage is an essential resource for new feature announcements and other Polly news. Check out the elevator pitch while you’re there.

The Polly repo on GitHub – The source, issues, and essential usage instructions.

Polly in the NuGet Gallery – All the installation goodness.

Bryan Hogan is a software architect, podcaster, blogger, Pluralisight author and speaker who has been working in .NET since 2004. He lives in the Boston area and is involved with the community there through meetups and other events. Last year Bryan authored a Pluralsight course on Polly, he also blogs at the No Dogma Blog and even hosts a podcast!

Related Posts

Related Resources

A newspaper article

Twilio Docs

From APIs to SDKs to sample apps

API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.
An Open book

Resource Center

The latest ebooks, industry reports, and webinars

Learn from customer engagement experts to improve your own communication.
User group reactions

Ahoy

Twilio's developer community hub

Best practices, code samples, and inspiration to build communications and digital engagement experiences.