Using Polly Circuit Breakers for Resilient .NET Web Service Consumers

If you have a bit of familiarity with Polly, the resilience framework for .NET, you will know how useful the Retry and Wait and Retry policies are for handling transient faults and longer outages. But there are times when no amount of retries will solve the problem, the remote system is unresponsive, has been for a while and will likely continue to be so. In this scenario, cutting the connection to the failing system might be the best approach.

Polly lets you do this with its Circuit Breaker policies. The policies get their name from the circuit breaker in your home that cuts electricity to an outlet or fixture. Just like the real-world circuit breaker, the Polly Circuit Breakers cut your connection to a faulting system.

This post shows you how to incorporate Polly Circuit Breakers into your resilience strategy. You’ll build working examples with ASP.NET Core 2.1 and see how Circuit Breakers work with other Polly policies.

Prerequisites

To follow along with this post you need:

• .NET Core 2.1+ SDK (The SDK includes the runtime.)
• Git (Used to clone the repository for this post.)
• curl, Fiddler, Postman, or Insomnia (The Git installer for Windows includes a curl executable.)
• Visual Studio 2017/2019, Visual Studio Code, or your favorite development environment.

The companion repository for this post is available on GitHub. You'll use it as the basis for the coding you'll do in the tutorial that follows.

If you’re new to Polly, or want to refresh your understanding of the Retry and Wait and Retry policies before proceeding, the previous posts in this series will give you an introduction to the framework and these foundational elements:

Building Resilient Service-to-Service Communications in ASP.NET Core with Polly, the .NET Resilience Framework – Introduces Polly and the Retry policy.

More Resilient Service-to-Service Communication with Polly and .NET HttpClientFactory – Demonstrates the Retry policy with the ASP.NET 2.1 HttpClientFactory class.

Polly Fallbacks: The Last Line of Defense in .NET Service-to-Service Communication – Demonstrates the Wait and Retry Policy.

Circuit Breakers

The circuit breaker can be viewed as a state machine that starts in the closed state; this is its normal state and allows the flow of requests across it. When a problem is detected the circuit breaker moves to the open state, blocking all requests for specified period. After that period elapses the circuit breaker moves to a half-open state where the first request is treated as a test request. If this request succeeds the circuit closes and normal operation resumes, but if it fails the circuit moves back to open and remains there for a specified period before once again moving to half-open.

If the naming “closed” and “open” sounds counterintuitive, consider how electricity flows across a circuit: it can only flow when the circuit is closed and cannot flow if the circuit is open.

The circuit breaker can move between the following states:

• Closed to open
• Open to half-open
• Half-open back to open
• Half-open to closed

Polly offers two variations of the policy: the basic circuit breaker that cuts the connection if a specified number of consecutive failures occur, and the advanced circuit breaker that cuts the connection when a specified percentage of errors occur over a specified period and when a minimum number of requests have occurred in that period.

Here is an example of the Basic Circuit Breaker policy, if 2 consecutive errors occur, the circuit is cut for 30 seconds:

This is the Advanced Circuit Breaker policy: the circuit will be cut if 25% of requests fail in a 60 second window, with a minimum of 7 requests in the 60 second window, then the circuit should be cut for 30 seconds:

In the previous posts in this series each policy could be used in multiple locations at the same time and each usage of the policy did NOT share state or interfere with each other. For example, a retry policy executing request A was NOT affected by the same retry policy executing request B. Each policy execution was independent.

For the circuit breaker to work correctly you want the policy to share state while executing requests.

Let's say you are connecting to endpoints X and Y on a remote service. The circuit breaker counts the failures on both X and Y, and if the circuit breaks your connection to both X and Y will be cut.

Getting started

The companion repository contains two Visual Studio ASP.NET Core 2.1 Web API solutions, one implementing a Weather Service and the other a Temperature Service. The Weather Service has two (somewhat contrived) actions methods, both get the weather information for a location, which includes the temperature, one in Celsius and one in Fahrenheit. The Weather Service in turn calls corresponding methods in the Temperature Service. The Temperature Service has a single controller with two actions, one for Celsius and one for Fahrenheit; the Celsius action method fails 25% of the time and the Fahrenheit action method fails 100% of the time.

You can imagine that the Weather Service would also call other backend services like a humidity service, a storm tracker service, etc., but only the Temperature Service is shown in this example.  

You don’t need to make any changes to the TemperatureService project, but you will make changes to the WeatherService project in the Startup.cs file. The full implementation of the WeatherController is provided.

Under the PollyCircuitBreakers directory, open the following solution files in separate instances of Visual Studio:

TemperatureService/TemperatureService.sln
WeatherService/WeatherService.sln

Using separate instances of Visual Studio (or separate console windows if you're running from the .NET CLI) will enable you to conveniently run both services simultaneously and watch the console output from each application.

If you don’t want to code along, execute the following command in the WeatherService  directory to checkout the finished code:

Implementing basic Polly Circuit Breaker policies

If you are coding along, add the NuGet package Microsoft.Extensions.Http.Polly to the WeatherService project, being sure to pick the version that works with the version of .NET Core you are using. The source code provided in the companion repository uses .NET Core 2.1, so the appropriate version of the Polly NuGet package is version 2.1.1. The NuGet Package Manager will also install Polly 6.0.1 and Polly.Extensions.Http 2.0.1 as dependencies.

In the WeatherService project, open the Startup.cs file.

In Startup.cs add a using Polly; statement at the top of the file.

Add the Basic Circuit Breaker policy to the ConfigureServices method:

In the policy you might have noticed three delegates, OnBreak, OnReset, and OnHalfOpen that are linted. These are optional delegates that are called as the circuit breakers move between states and they print out logging information to the console.

Add the following three methods to Startup.cs:

To use basicCircuitBreakerPolicyadd it to the HttpClientFactory policy handler. Insert the following code in the ConfigureServices method below the basicCircuitBreakerPolicy definition you previously added:

That is all that is needed to use the Basic Circuit Breaker.

If you are not familiar with the HttpClientFactory take a look at an earlier post in this series - More Resilient Service-to-Service Communication with Polly and .NET HttpClientFactory. The Circuit Breaker will apply to all requests made from HttpClients created by the HttpClientFactory.

In the WeatherController add a local variable for the HttpClientFactory and a constructor taking the HttpClientFactory:

Add two action methods that take requests and in turn call the TemperatureService, the calls using the HttpClient are wrapped the circuit breaker policy:

Break some circuits

Remember: the Celsius controller will fail 25% of the time and the Fahrenheit controller will fail 100% of the time.

Start the two services from their separate instances of Visual Studio (or from the .NET CLI if you're working with another IDE or an editor). The Weather Service opens on HTTP port 5001 and the Temperature Service opens on port 6001.

Execute the API calls in the following code block with your browser, curl, or your preferred tool. Keep in mind that the number of calls to each endpoint and the timing of the calls is significant. If you make a mistake, it’s a good idea to start over to ensure everything proceeds as expected.

As you do, keep an eye on the .NET CLI (dotnet.exe) window for the Weather Service project to see the results of the circuit breaker operation.  There will only be status messages when the circuit breaker is in operation. Each request command will also return results from the Weather Service based on the success or failure of the call to the Temperature Service.

Here are the responses you will receive for each request:

As mentioned earlier, the defined Basic Circuit Breaker breaks when there are two consecutive failures. It doesn’t matter if they occur a second, a minute, an hour or a day apart. Also, the circuit would NOT break if you had a pattern like this: failure, success, failure, success, failure, etc.

Advanced Circuit Breakers

If you want a little more fine-grained control over the circuit, try the Advanced Circuit Breaker policy.

Add the following into the ConfigureServices method in Startup.cs:

The currentHttpClientFactory block uses the basicCircuitBreakerPolicy:

Replace it with this: 

Use the same process to query the Weather Service. You will need to make more requests. The following set of requests will cut the circuit when run quickly enough:

Watch the log of the Weather Service to see the state of the circuit breaker as it moves between states. The circuit will break when there are 25% failures over a 60-second window with a minimum of 7 requests. If you can’t make the required number of requests quickly enough, increase the sampling window to a bigger number.

Summary

This post introduced the two Polly circuit breaker policies and showed how to use them with an ASP.NET Core 2.1 Web API application that uses the HttpClientFactory pattern to create instances of HttpClient.

Additional Resources

Companion Repository – The complete code for the projects in this post is available on GitHub and can be reused under the MIT license.

Bryan’s blog posts on Polly – Check out some of Bryan’s posts on Polly on his own blog.

The Polly Project – The project homepage is an essential resource for new feature announcements and other Polly news. Check out the elevator pitch while you’re there.

The Polly repo on GitHub – The source, issues, and essential usage instructions.

Polly in the NuGet Gallery – All the installation goodness.

Bryan Hogan is a Microsoft Most Valuable Professional, software architect, podcaster, blogger, Pluralisight author, and speaker who has been working in .NET since 2004. He lives in the Boston area and is involved with the community there through meetups and other events. Last year Bryan authored a Pluralsight course on Polly, he also blogs at the No Dogma Blog and even hosts a podcast!