Circuit Breaker: Learning When to Stop

A retry policy assumes that trying again will eventually work. Sometimes it will not. If the database is down, retrying every 200 milliseconds for the next 30 seconds just generates 150 doomed requests. The service needs time to recover, and hammering it with traffic makes recovery harder.

A circuit breaker solves this by tracking recent failures and, when enough accumulate, stopping calls entirely for a cooling-off period. After the period expires, it cautiously allows a single probe request through. If the probe succeeds, normal traffic resumes. If it fails, the circuit re-opens.

The State Machine

stateDiagram-v2
    direction LR
    [*] --> CLOSED
    CLOSED --> OPEN : consecutive failures reach failureThreshold
    OPEN --> HALF_OPEN : openDuration expires
    HALF_OPEN --> CLOSED : successThreshold probes succeed
    HALF_OPEN --> OPEN : any probe fails
    CLOSED --> CLOSED : a success resets the failure count

    note right of OPEN
        calls rejected immediately
        with CircuitOpenException
    end note

    classDef closed fill:#a6d189,stroke:#40a02b,color:#232634
    classDef open fill:#e78284,stroke:#d20f39,color:#232634
    classDef probing fill:#e5c890,stroke:#df8e1d,color:#232634
    class CLOSED closed
    class OPEN open
    class HALF_OPEN probing

Configuration

CircuitBreakerConfig config = CircuitBreakerConfig.builder()
    .failureThreshold(5)                      // 5 failures before opening
    .successThreshold(3)                      // 3 probes must succeed to close
    .openDuration(Duration.ofSeconds(30))     // Stay open for 30 seconds
    .callTimeout(Duration.ofSeconds(5))       // Each call times out after 5s
    .recordFailure(ex ->                      // Only count certain exceptions
        !(ex instanceof BusinessValidationException))
    .build();

The recordFailure predicate is important: not every exception means the service is unhealthy. A 400 Bad Request or a business validation error reflects a problem with the request, not the service. Only count failures that indicate the service itself is struggling.

Creating a Circuit Breaker

// With custom configuration
CircuitBreaker breaker = CircuitBreaker.create(config);

// With sensible defaults
CircuitBreaker breaker = CircuitBreaker.withDefaults();

Protecting VTask Operations

The protect() method is generic. A single circuit breaker instance can protect calls that return different types:

CircuitBreaker paymentBreaker = CircuitBreaker.create(
    CircuitBreakerConfig.builder()
        .failureThreshold(3)
        .openDuration(Duration.ofSeconds(30))
        .build());

// Protects a call returning String
VTask<String> getStatus = paymentBreaker.protect(
    VTask.of(() -> paymentService.getStatus(orderId)));

// Same breaker protects a call returning BigDecimal
VTask<BigDecimal> getBalance = paymentBreaker.protect(
    VTask.of(() -> paymentService.getBalance(accountId)));

// Both share state: failures from either call count towards the threshold

This is the correct design. A circuit breaker protects a service endpoint, not a specific return type.

Fallbacks

When the circuit is open, protect() throws CircuitOpenException. Use protectWithFallback() to provide a default value instead:

VTask<String> withFallback = paymentBreaker.protectWithFallback(
    VTask.of(() -> paymentService.getStatus(orderId)),
    ex -> "status-unavailable");

Or compose with recover() for more control:

VTask<String> resilient = paymentBreaker.protect(
        VTask.of(() -> paymentService.getStatus(orderId)))
    .recover(ex -> {
        if (ex instanceof CircuitOpenException coe) {
            log.warn("Payment service down, retry after {}", coe.retryAfter());
            return cachedStatus(orderId);
        }
        return "unknown";
    });

Path-Native Circuit Breakers

The lazy Path carriers chain breaker protection directly, with the same shareable breaker:

IOPath<String> guarded = Path.io(() -> paymentService.getStatus(orderId))
    .withCircuitBreaker(paymentBreaker);

VTaskPath<String> guardedAsync = Path.vtask(() -> paymentService.getStatus(orderId))
    .withCircuitBreaker(paymentBreaker);

On the typed-error carriers the breaker is railway-aware: a Left is a successfully computed value (the service answered, just not with a Right), so it does not count as a breaker failure. Only thrown exceptions (defects) trip the circuit. A stream of "customer not found" responses will never open the circuit; a stream of connection resets will.

The typed overloads keep an open-circuit rejection on the typed channel instead of surfacing CircuitOpenException:

// VResultPath: instance combinator, rejection lands as a Left
VResultPath<OrderError, Reservation> guarded =
    reserveInventory(order)
        .withCircuitBreaker(
            inventoryBreaker,
            open -> OrderError.SystemError.circuitBreakerOpen("inventory"));

// EitherPath is eager, so the combinator is static and takes the step as a Supplier
EitherPath<OrderError, Reservation> reserved = EitherPath.withCircuitBreaker(
    () -> reserveInventory(order),
    inventoryBreaker,
    open -> OrderError.SystemError.circuitBreakerOpen("inventory"));

Without the onOpen argument, the rejection propagates as-is: a thrown CircuitOpenException on EitherPath, a defect on the VTask failure channel for VResultPath.

Metrics

CircuitBreakerMetrics m = breaker.metrics();

log.info("Circuit breaker: total={}, success={}, failed={}, rejected={}, transitions={}",
    m.totalCalls(), m.successfulCalls(), m.failedCalls(),
    m.rejectedCalls(), m.stateTransitions());

Manual Control

// Reset to CLOSED with zeroed counters
breaker.reset();

// Manually trip to OPEN (e.g., during maintenance)
breaker.tripOpen();

// Inspect current state
CircuitBreaker.Status status = breaker.currentStatus();

Combining with Retry

A common pattern is to combine circuit breaker with retry. The order matters:

// Circuit breaker inside retry: each retry attempt checks the circuit
VTask<String> resilient = Retry.retryTask(
    paymentBreaker.protect(VTask.of(() -> paymentService.get(url))),
    RetryPolicy.exponentialBackoff(3, Duration.ofMillis(200))
        .retryIf(ex -> !(ex instanceof CircuitOpenException)));

Note the retry predicate: CircuitOpenException should not be retried, because the circuit breaker has already determined the service is unhealthy. Combined Patterns introduces ResilienceBuilder, which applies this ordering for you without manual wiring.

Previous: Retry Next: Bulkhead