Polly + Resilience Pipeline: Stop Letting Transient Failures Kill Your System

Most production failures are temporary:

Network blip
SQL timeout
Redis restart
Downstream service overload

Without resilience, one short failure cascades.

Request
   │
   ▼
HTTP / SQL / Message
   │
   X  ← transient error
   │
   ▼
User sees 500

With a resilience pipeline:

Request
   │
   ▼
Resilience Pipeline
   │  retry / circuit / timeout / bulkhead
   ▼
HTTP / SQL / Message
   │
   ✓ or controlled failure

---

What Problem Does It Solve?

Resilience is not about never failing. It’s about failing in a controlled way.

---

Classic Polly vs Resilience Pipeline

Old style (Polly v7)
────────────────────
Policy.Handle<HttpRequestException>()
      .WaitAndRetryAsync(...)
      .ExecuteAsync(...)

New style (.NET 8+)
───────────────────
ResiliencePipeline pipeline = new ResiliencePipelineBuilder()
    .AddRetry(...)
    .AddCircuitBreaker(...)
    .AddTimeout(...)
    .Build();

await pipeline.ExecuteAsync(async ct => ...);

The Resilience Pipeline is the modern, composable way.

---

Core Building Blocks

┌──────────────────────────────────────┐
│         Resilience Pipeline          │
│                                      │
│  ┌─────────┐  ┌─────────┐  ┌───────┐ │
│  │  Retry  │→ │ Circuit │→ │Timeout│ │
│  └─────────┘  └─────────┘  └───────┘ │
│         ↓                            │
│  ┌─────────┐                         │
│  │ Bulkhead│                         │
│  └─────────┘                         │
└──────────────────────────────────────┘

1. Retry

Attempt 1 → fail
Attempt 2 → fail
Attempt 3 → success

Use for: transient network, SQL deadlocks, brief overloads.

2. Circuit Breaker

Closed ──(failures)──▶ Open ──(wait)──▶ Half-Open
                              │
                              ▼
                         fail fast

Use for: protecting yourself when the dependency is truly down.

3. Timeout

Call starts
   │
   │  3 seconds...
   │
   X  ← cancelled

Use for: never wait forever.

4. Bulkhead

Max 10 concurrent
   │
   ├── 10 running
   └── rest rejected immediately

Use for: isolating resources so one slow dependency doesn’t starve the whole app.

---

How to Use It (3 Practical Ways)

---

1. Named HttpClient (Best Default)

Register once in Program.cs - resilience becomes automatic.

// Program.cs
builder.Services.AddHttpClient("payment-api", client =>
{
    client.BaseAddress = new Uri("https://payment.example.com/");
})
.AddResilienceHandler("payment-pipeline", builder =>
{
    builder
        .AddRetry(new HttpRetryStrategyOptions
        {
            MaxRetryAttempts = 3,
            Delay = TimeSpan.FromMilliseconds(200),
            BackoffType = DelayBackoffType.Exponential,
            UseJitter = true
        })
        .AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions
        {
            FailureRatio = 0.5,
            SamplingDuration = TimeSpan.FromSeconds(10),
            MinimumThroughput = 8,
            BreakDuration = TimeSpan.FromSeconds(30)
        })
        .AddTimeout(TimeSpan.FromSeconds(5));
});

Usage - clean, no extra code:

public class PaymentService
{
    private readonly HttpClient _http;

    public PaymentService(IHttpClientFactory factory)
    {
        _http = factory.CreateClient("payment-api"); // already resilient
    }

    public async Task PayAsync(...)
    {
        var response = await _http.PostAsync("/charge", content);
    }
}

Visual flow:

Request
   │
   ▼
Timeout (5s)
   │
   ▼
Circuit Breaker
   │  (open? → fail fast)
   ▼
Retry (3x + jitter)
   │
   ▼
Actual HTTP call

---

2. Shared Pipeline for non-HTTP (SQL, Redis, etc.)

// Program.cs
builder.Services.AddResiliencePipeline("default", builder =>
{
    builder
        .AddTimeout(TimeSpan.FromSeconds(3))
        .AddRetry(new RetryStrategyOptions
        {
            MaxRetryAttempts = 3,
            Delay = TimeSpan.FromMilliseconds(150),
            BackoffType = DelayBackoffType.Exponential,
            UseJitter = true
        })
        .AddCircuitBreaker(new CircuitBreakerStrategyOptions
        {
            FailureRatio = 0.5,
            BreakDuration = TimeSpan.FromSeconds(20)
        });
});

// Multiple named pipelines
builder.Services.AddResiliencePipeline("critical", builder => { ... });
builder.Services.AddResiliencePipeline("background", builder => { ... });

Usage:

public class OrderService
{
    private readonly ResiliencePipelineProvider<string> _pipelines;

    public OrderService(ResiliencePipelineProvider<string> pipelines)
    {
        _pipelines = pipelines;
    }

    public async Task ProcessAsync()
    {
        var pipeline = _pipelines.GetPipeline("default");

        await pipeline.ExecuteAsync(async ct =>
        {
            await _db.SaveChangesAsync(ct);
        });
    }
}

---

3. Inline (Rare)

var pipeline = new ResiliencePipelineBuilder()
    .AddTimeout(TimeSpan.FromSeconds(2))
    .AddRetry(new RetryStrategyOptions { MaxRetryAttempts = 2 })
    .Build();

await pipeline.ExecuteAsync(async ct =>
{
    // one-off call
});

Use only for special cases. Prefer the DI versions above.

---

What about Attribute / ActionFilter style?

Rate Limiting works well with attributes ([EnableRateLimiting("policy")]) because it is a request-level decision.

Resilience is usually a dependency-level decision (this HttpClient, this database call).

You can build a custom [Resilient("pipeline-name")] filter, but it adds complexity and is less flexible than putting resilience on the outgoing call itself.

Recommendation: Keep resilience on the HttpClient or service, not on the controller action.

---

Quick Decision

Calling external HTTP API?
        │
        ▼
Named HttpClient + AddResilienceHandler   ← best

Calling SQL / Redis / internal service?
        │
        ▼
Named ResiliencePipeline from DI

One special case only?
        │
        ▼
Inline pipeline (rare)

Suggested Project Structure

Program.cs
├── AddHttpClient("payment").AddResilienceHandler(...)
├── AddHttpClient("shipping").AddResilienceHandler(...)
├── AddResiliencePipeline("default", ...)
└── AddResiliencePipeline("critical", ...)

Services
├── PaymentService  → uses "payment" HttpClient
├── OrderService    → uses "default" pipeline for DB
└── ...

---

Recommended Order of Strategies

Always compose in this order:

1. Timeout        ← hard limit
2. Rate Limiter   ← optional
3. Circuit Breaker
4. Retry
5. Bulkhead       ← outermost isolation

Wrong order creates surprising behavior.

---

Decision Guide

Is the failure temporary?
        │
   ┌────┴────┐
   │         │
  Yes       No
   │         │
   ▼         ▼
 Retry    Circuit Breaker
   │         │
   ▼         ▼
Timeout   Fail fast

---

Common Mistakes

❌ Retry forever
❌ Retry on 400 / 401 / 403 (client errors)
❌ No jitter → thundering herd
❌ Circuit breaker with too low threshold
❌ Timeout longer than the user is willing to wait
❌ Different policies scattered in every service

---

Minimal Working Example

var pipeline = new ResiliencePipelineBuilder()
    .AddTimeout(TimeSpan.FromSeconds(3))
    .AddRetry(new RetryStrategyOptions
    {
        MaxRetryAttempts = 3,
        Delay = TimeSpan.FromMilliseconds(200),
        BackoffType = DelayBackoffType.Exponential,
        UseJitter = true
    })
    .AddCircuitBreaker(new CircuitBreakerStrategyOptions
    {
        FailureRatio = 0.5,
        BreakDuration = TimeSpan.FromSeconds(20)
    })
    .Build();

await pipeline.ExecuteAsync(async token =>
{
    // your HTTP / SQL / gRPC call
}, cancellationToken);

---

When You Don’t Need It

• Pure in-memory operations
• Calls that must never be retried (non-idempotent without keys)
• Extremely low-latency internal calls where overhead matters more than resilience

---

Final Takeaway

Without resilience:
  One blip → user sees error

With Resilience Pipeline:
  One blip → retry → success
  Real outage → circuit opens → fail fast
  Slow call  → timeout → user gets response

Resilience is not optional infrastructure.
It’s how production systems stay calm when the network is not.

---

Packages

dotnet add package Microsoft.Extensions.Http.Resilience
# or
dotnet add package Polly.Core

For HttpClient the Microsoft package is the recommended path in modern .NET.