Polly Fallbacks: The Last Line of Defense in .NET Service-to-Service Communication

Polly Fallbacks: The Last Line of Defense in .NET Service-to-Service Communication

Using Polly, the resilience framework for .NET, you can gracefully handle lost packets, thrown exceptions, and failed requests which inevitably make their way into service-to-service communications on the web. Polly provides resilience strategies for your apps through policies such as Retry, WaitAndRetry, and CircuitBreaker, enabling you to implement fault tolerance in your distributed systems in a fluent fashion.

This post will introduce you to the Fallback policy and demonstrate its use in a straightforward ASP.NET Core 2.1 example. It will guide you in using the Fallback policy to handle failures you cannot recover from: when all else fails, fallback to this last line of defense.

If you’re not already familiar with Polly, the first post in this series on the Twilio blog introduced Polly, the .NET resilience framework. The second post build on the first and introduced the HttpClientFactory.

Prerequisites

To follow along with this post you need:

.NET Core 2.1+ SDK (The SDK includes the runtime.)

Git (Used to clone the repository for this post.)

curl, Fiddler, Postman, or Insomnia (The Git installer for Windows includes a curl executable.)

Visual Studio 2017/2019, Visual Studio Code, or your favorite development environment.

The companion repository for this post is available on GitHub. You'll use it as the basis for the coding you'll do in the tutorial that follows.

Getting Started

The case study software in the companion repository consists of two Visual Studio ASP.NET Core 2.1 Web API solutions, one implementing a Weather Service and the other a Temperature Service. The Weather Service makes HTTP requests to the Temperature service. In previous posts the Temperature Service was setup to fail some of the time but would eventually return a valid response. The Polly Retry policy in the Weather Service was responsible for retrying a request when a failure was returned from the Temperature Service.

In this post, the Temperature Service will return failures 100% of the time. You will use the Wait and Retry policy in combination with Fallback policy. The Wait and Retry lets you retry a request a specified number of times with a defined delay between retries.

The Fallback policy allows you to perform any action when a request fails, this could be restarting a service, messaging an admin, scaling out a cluster, etc. It is usually used as at the last policy to be called inside a policy wrap.

Policy Wraps

A policy wrap allows multiple policies to be combined and executed in series. Any number of policies can be combined. A request that is executed by the policy wrap will pass through each policy, and the response will return through each in reverse order. The behavior clause of each policy is executed only if the response matches the handles clause of the policy. See this post for more on the handles and behavior clauses of policies.

Here is an example of a policy wrap with two policies:

When an HTTP request is made, the request first passes through the fallbackPolicy which passes it to the waitAndRetryPolicy, which executes the HttpClient request.

When the response is received by the HttpClient it then is passed to the waitAndRetryPolicy which might perform a retry depending on the response received. If any retries are to be performed the waitAndRetryPolicy will do this. If all retries have been performed, or none were needed, the response passes to the fallbackPolicy. Its handles clause inspects the response and determines if its behavior clause should execute.

If you had more policies in the wrap the response would then pass to these policies, but as mentioned earlier, the Fallback policy is generally the last policy to use.

Adding a Fallback policy to a web service 

The companion project uses HttpClientFactory in conjunction with Polly to create instances of HttpClient in the Weather Service controller. The previous post explained how to use the HttpClientFactory with Polly policies and dependency injection if you need more information on that technique.

Clone the accompanying repository:

Go to the PollyFallbackWeatherService directory and open the following solution files in separate instances of Visual Studio:

TemperatureService/TemperatureService.sln
WeatherService/WeatherService.sln

In the Temperature Service, take a look at the TemperatureController, it has a single Get method that returns a 500 Internal Server Response every time it is executed. You don’t have to make any changes to this solution.

From the Weather Service, you are going to make HTTP requests to the Temperature Service using the Wait and Retry and Fallback policies wrapped together. You will use the HttpClientFactory to pass the HttpClient to WeatherController and then use a HttpClient to execute the requests that will hit the TemperatureController in the Temperature Service.

If you don’t want to code along, execute the following command in the PollyFallbackWeatherService directory to checkout the finished code:

If you are coding along, add the NuGet package Microsoft.Extensions.Http.Polly to the WeatherService project, being sure to pick the version that works with the version of .NET Core you are using. The source code provided in the companion repository uses .NET Core 2.1, so the appropriate version of the Polly NuGet package is version 2.1.1.

In Startup.cs add a using Polly; statement at the top of the file.

In the ConfigureServices method add the Wait and Retry policy and the Fallback policies.

Add the FallbackAction and OnFallbackAsync methods:

The Wait and Retry policy’s handles clause specifies that it becomes active if the HTTP response status code is anything other than a success code, i.e. not in the 200 range. It retries up to three times with a delay between each retry. The length of the delay before retrying starts at 1 second for the first retry, then increase to 2 seconds for the second retry, the last retry is delayed by 3 seconds.

The Fallback policy’s handles clause is the same as the Wait and Retry, the Fallback’s behavior clause becomes active if the response is anything other than a success code. The behavior clause specifies the action to take before the action is performed the onFallback delegate executes.

In this example, when the Fallback policy receives the HttpResponseMessage and sees that it is a 500, the onFallback delegate is executed. This is commonly used for logging or tracking metrics. Then the fallback action is executed; the provided example calls a method, but you can do this with a lambda expression. The method is passed three parameters including the HttpResponseMessage from the failed request to the Temperature Service.

The fallback action method returns a HttpResponseMessage that is passed back to the original caller in place of the HttpResponseMessage received from the Temperature Service. In this example the action to take is to send a dummy message to an admin, but it could be anything you want.

Create the policy wrap by adding the following code just after the previous block of new code:

To add the HttpClientFactory add the following C# code immediately after the wrap policy:

The HttpClientFactory adds the wrapOfRetryAndFallback as its policy handler.

Move to the WeatherController.cs file in the Controllers folder and add the following code:

This passes the HttpClientFactory into the controller and assigns it to a local variable for use in the action method below.

Directly below the constructor add the following block of code:

Those are all the changes needed.

This action method responds to API requests sent in from a client, such as a curl command. It gets a HttpClient from the HttpClientFactory, which makes a request to the Temperature Service. Because you are using the HttpClientFactory the wrapOfRetryAndFallback policy executes with the request. The policy behaves as described above, ending with the Fallback policy generating a new HttpResponseMessage.

This HttpResponseMessage is then checked inside the Get method, if it is in the 200 range (it won’t be) an OK message with the temperature is sent back to the caller.

But, as you know, it will be the HttpResponseMessage created by the Fallback policy. The last line of the method creates returns a StatusCode with the 500 and the message from inside the Fallback policy.

Take it for a spin

The Temperature Service will fail 100% of the time and the Weather Service will retry three times before executing the fallback.

Start the two services from their separate instances of Visual Studio (or from the .NET CLI if you're working with another IDE or an editor). The Weather Service opens on HTTP port 5001 and the Temperature Service opens on port 6001.

The following instructions for making the HTTP calls to the Weather Service use curl, but you can use Fiddler, Postman, Insomnia, your web browser or PowerShell Invoke-WebRequest if you prefer.

Execute the following command-line instruction in your console window:

Your request will first hit the Weather Service, which then makes a request to the Temperature Service. The call to the Temperature Service will result in an HTTP 500 Internal Server Error response being returned to the Weather Service. The Wait and Retry policy will execute and perform the request again. This process repeats itself until the third retry has occurred and the Wait and Retry policy gives up.

The Fallback then steps in, examines the HttpResponseMessage and executes its onFallback delegate to perform some logging. After the onFallback completes, the FallbackAction is executed, the admin is notified, the original HttpResponseMessage is examined and a new one is created. This new HttpResponseMessage is then returned to the original calling code, the HttpClient inside the WeatheController.

Like most Polly policies, the Fallback policy has many overloads; you need to look at auto generated documentation inside your IDE or view the Polly source code to see them all.

Summary

This post provides an introduction to the Polly Fallback policy and demonstrated how to use it with an ASP.NET Core 2.1 Web API application that uses the HttpClientFactory pattern to create instances of HttpClient. It’s the third in a series of posts providing a practical introduction to using Polly with ASP.NET Core.

Additional Resources

Companion Repository – The complete code for the projects in this post is available on GitHub and can be reused under the MIT license.

Bryan’s blog posts on Polly – Check out some of Bryan’s posts on Polly on his own blog.

The Polly Project – The project homepage is an essential resource for new feature announcements and other Polly news. Check out the elevator pitch while you’re there.

The Polly repo on GitHub – The source, issues, and essential usage instructions.

Polly in the NuGet Gallery – All the installation goodness.

Bryan Hogan is a software architect, podcaster, blogger, Pluralisight author, and speaker who has been working in .NET since 2004. He lives in the Boston area and is involved with the community there through meetups and other events. Last year Bryan authored a Pluralsight course on Polly, he also blogs at the No Dogma Blog and even hosts a podcast!