Handling circuit breakers between gRPC services directly at the application layer without a service mesh

These days, I often meet teams that adopted Istio, got burned by memory usage and CRD complexity, and went back to the application layer. I too initially thought "just slap on a sidecar and everything's solved," but once you're actually operating it, you find yourself digging through Istio CRD docs just to change one circuit breaker policy. As a result, controlling circuit breakers directly in code is getting attention again, and especially in gRPC-based systems, this approach fits surprisingly cleanly.

This post covers implementing circuit breakers by layering sony/gobreaker (Go) and Resilience4j (Java) onto gRPC client interceptors. We'll look at why the grpc-go team doesn't provide a built-in implementation and instead recommends "external library + client interceptor," the gRPC-specific status code mapping problem, and the trap of state being distributed across multiple Pods.

Where Circuit Breakers Get Tricky in gRPC

Understanding the Three States as a Finite State Machine

The circuit breaker is a pattern systematized by Michael Nygard in Release It!, a finite state machine (FSM) that transitions among three states. The concept is simple, but applying it to gRPC has a different texture than HTTP.

In the Closed state, all requests are forwarded downstream and failure counts accumulate. Once the threshold is crossed, it transitions to Open, where subsequent requests return an error immediately without calling downstream at all. After a configured timeout, it enters Half-Open, allowing only a small number of probe requests — if the service has recovered, it returns to Closed; if it's still down, it goes back to Open.

Why gRPC Differs from Plain HTTP

Plugging a circuit breaker designed for HTTP/1.1 services directly into gRPC doesn't fit well. There are a few reasons.

First, gRPC runs on HTTP/2 multiplexing, so multiple RPCs flow concurrently over a single TCP connection. Counting failures at the connection level is meaningless — you need to track at the individual RPC call level.

Second, the criteria for judging failure are different. In the gRPC status code system, codes like UNAVAILABLE, INTERNAL, and UNKNOWN should count as failures, but NOT_FOUND or INVALID_ARGUMENT are responses returned by a normally functioning downstream and should not count as failures. DEADLINE_EXCEEDED is where teams disagree — I prefer counting timeouts as failures too (since from the client's perspective, there's no result).

Third, the grpc-go team has explicitly stated in GitHub Issue #5672 that they will not provide a built-in circuit breaker. This issue is frequently cited as a design reference when discussing circuit breakers in gRPC Go.

Combining sony/gobreaker with a Unary Interceptor in Go

Managing Independent Instances per Service

If you have multiple downstreams, it's important to maintain independent circuit breaker instances per service. You don't want the Payment service going down to also open the circuit for the Inventory service.

// Conceptual example — based on sony/gobreaker, import path is github.com/sony/gobreaker
package circuitbreaker
 
import (
    "sync"
 
    "github.com/sony/gobreaker"
)
 
type MultiServiceBreaker struct {
    mu       sync.RWMutex
    breakers map[string]*gobreaker.CircuitBreaker
    settings gobreaker.Settings
}
 
func NewMultiServiceBreaker(settings gobreaker.Settings) *MultiServiceBreaker {
    return &MultiServiceBreaker{
        breakers: make(map[string]*gobreaker.CircuitBreaker),
        settings: settings,
    }
}
 
func (m *MultiServiceBreaker) Get(service string) *gobreaker.CircuitBreaker {
    m.mu.RLock()
    cb, ok := m.breakers[service]
    m.mu.RUnlock()
    if ok {
        return cb
    }
 
    m.mu.Lock()
    defer m.mu.Unlock()
    if cb, ok = m.breakers[service]; ok {
        return cb
    }
    s := m.settings
    s.Name = service
    cb = gobreaker.NewCircuitBreaker(s)
    m.breakers[service] = cb
    return cb
}

By customizing the ReadyToTrip callback in gobreaker.Settings, you receive Counts aggregates and can define flexible thresholds like "6 or more failures out of 10" instead of "5 consecutive failures." The important point here is that ReadyToTrip only decides whether to trip. Whether an individual error counts as a failure is determined by the return value of the Execute closure covered in the next section. Confusing these two points leads to failure filtering ending up in the wrong place.

gRPC Status Code Filter for Failure Determination

package circuitbreaker
 
import (
    "google.golang.org/grpc/codes"
    "google.golang.org/grpc/status"
)
 
func isFailure(err error) bool {
    if err == nil {
        return false
    }
    s, ok := status.FromError(err)
    if !ok {
        return true
    }
    switch s.Code() {
    case codes.Unavailable, codes.Internal, codes.Unknown:
        return true
    case codes.DeadlineExceeded:
        return true // Can be changed to false depending on team policy
    default:
        return false
    }
}

gRPC Unary Client Interceptor

package circuitbreaker
 
import (
    "context"
    "errors"
    "strings"
 
    "github.com/sony/gobreaker"
    "google.golang.org/grpc"
    "google.golang.org/grpc/codes"
    "google.golang.org/grpc/status"
)
 
// The method parameter is passed in the form "/package.ServiceName/MethodName".
// We extract only the service name to use as the circuit breaker key.
func extractServiceName(fullMethod string) string {
    trimmed := strings.TrimPrefix(fullMethod, "/")
    if idx := strings.Index(trimmed, "/"); idx > 0 {
        return trimmed[:idx]
    }
    return trimmed
}
 
func CircuitBreakerInterceptor(msb *MultiServiceBreaker) grpc.UnaryClientInterceptor {
    return func(
        ctx context.Context,
        method string,
        req, reply any,
        cc *grpc.ClientConn,
        invoker grpc.UnaryInvoker,
        opts ...grpc.CallOption,
    ) error {
        svcName := extractServiceName(method)
        cb := msb.Get(svcName)
 
        var callErr error
        _, cbErr := cb.Execute(func() (any, error) {
            callErr = invoker(ctx, method, req, reply, cc, opts...)
            if isFailure(callErr) {
                // Return the original error as-is so gobreaker counts it as a failure
                return nil, callErr
            }
            // Business errors like NOT_FOUND are reported to gobreaker as success,
            // and the original error is returned as-is to the caller outside.
            return nil, nil
        })
 
        switch {
        case errors.Is(cbErr, gobreaker.ErrOpenState):
            return status.Errorf(codes.Unavailable, "circuit breaker open for %s", svcName)
        case errors.Is(cbErr, gobreaker.ErrTooManyRequests):
            // Exceeded the allowed limit (MaxRequests) in Half-Open state
            return status.Errorf(codes.Unavailable, "circuit breaker probing limit for %s", svcName)
        }
        return callErr
    }
}

Here we've also fixed two traps from an earlier draft. First, gobreaker counts any non-nil error returned by the Execute closure as a failure, so business errors like NOT_FOUND must be closed with nil, nil in the closure, with the actual error extracted into a variable outside the closure (callErr). Second, when MaxRequests is exceeded in Half-Open, ErrTooManyRequests is returned rather than ErrOpenState — if you don't branch on this case, the raw error leaks upward.

This interceptor is injected at grpc.NewClient time via the grpc.WithUnaryInterceptor(...) option.

Combining Resilience4j with GlobalClientInterceptor in Java

In a Spring Boot environment, you can use @GrpcGlobalClientInterceptor to apply a circuit breaker to all gRPC client calls at once.

// Conceptual example — grpc-spring-boot-starter + Resilience4j
@Component
@GrpcGlobalClientInterceptor
public class CircuitBreakerClientInterceptor implements ClientInterceptor {
 
    private final CircuitBreakerRegistry registry;
 
    public CircuitBreakerClientInterceptor(CircuitBreakerRegistry registry) {
        this.registry = registry;
    }
 
    @Override
    public <ReqT, RespT> ClientCall<ReqT, RespT> interceptCall(
            MethodDescriptor<ReqT, RespT> method,
            CallOptions callOptions,
            Channel next) {
 
        String cbName = method.getServiceName();
        CircuitBreaker cb = registry.circuitBreaker(cbName);
 
        return new ForwardingClientCall.SimpleForwardingClientCall<>(
                next.newCall(method, callOptions)) {
 
            private long startNanos;
 
            @Override
            public void start(Listener<RespT> responseListener, Metadata headers) {
                cb.acquirePermission(); // Throws CallNotPermittedException when Open
                startNanos = System.nanoTime();
 
                super.start(new ForwardingClientCallListener
                        .SimpleForwardingClientCallListener<>(responseListener) {
                    @Override
                    public void onClose(Status status, Metadata trailers) {
                        long elapsed = System.nanoTime() - startNanos;
                        if (isGrpcFailure(status)) {
                            cb.onError(elapsed, TimeUnit.NANOSECONDS, toException(status));
                        } else {
                            cb.onSuccess(elapsed, TimeUnit.NANOSECONDS);
                        }
                        super.onClose(status, trailers);
                    }
                }, headers);
            }
        };
    }
 
    private boolean isGrpcFailure(Status status) {
        return status.getCode() == Status.Code.UNAVAILABLE
            || status.getCode() == Status.Code.INTERNAL
            || status.getCode() == Status.Code.UNKNOWN
            || status.getCode() == Status.Code.DEADLINE_EXCEEDED;
    }
 
    // A helper for the reader to implement according to their organization's conventions.
    // The simplest approach is to just return status.asRuntimeException() directly.
    private Throwable toException(Status status) {
        return status.asRuntimeException();
    }
}

Passing 0 as the duration argument to onError/onSuccess kills Resilience4j's time-based sliding window statistics. You must capture the call start time with System.nanoTime() and pass the elapsed duration so that not only count-based but also time-based window configurations ("error rate over 50% in the last 30 seconds") work correctly.

Resilience4j supports two sliding window types — Count-based and Time-based — so you can set ratio-based thresholds like "50% or more failures out of the last 100 calls." For a reference implementation of gRPC client integration, Deep Network GmbH's Resilience4j-gRPC example is widely cited.

Request Flow and Behavior Sequence by Circuit State

It's safer to explicitly define the number of probe requests allowed in Half-Open and the success criteria. For example, you can set gobreaker's MaxRequests to 2 and require both to succeed before returning to Closed.

Trade-offs and Common Pitfalls in Practice

Pros and Cons at a Glance

State Not Shared Due to Pod Distribution

This is the most commonly overlooked pitfall in practice. If you have 3 Pods, you have 3 circuit breaker instances each living independently. Even if the circuit opens in one Pod, the other two Pods still forward failing requests to downstream.

There are two ways to mitigate this. First, set conservative, low thresholds so each Pod can independently reach Open quickly. Second, instrument circuit state with OpenTelemetry metrics and integrate the circuit state of all Pods visually in Prometheus/Grafana, so the operations team can detect early when a circuit is repeatedly opening in a particular Pod.

Collision with gRPC Built-in Retry

Using retryPolicy in the gRPC service configuration together with circuit breakers causes UNAVAILABLE errors returned from the Open state to be classified as retryable codes, resulting in retries being attached. If you intend to fail-fast by opening the circuit but retries keep coming in, the effect disappears. You need to either remove UNAVAILABLE from retryableStatusCodes, or attach separate metadata to errors returned by the circuit breaker so the retry logic can distinguish them.

Streaming RPC Requires Separate Design

The interceptors above are based on Unary RPC. For bidirectional streaming, you need a separate design for how to count failures that occur mid-stream after the stream is opened. Implementation-wise, you'd need to create a new StreamClientInterceptor, but the safest starting point from a circuit breaker perspective is intercepting the stream creation attempt itself. This is conceptually the same strategy as wrapping the invoker call in the Unary interceptor, and only reflects UNAVAILABLE errors that occur at the stream creation stage into the circuit. Meanwhile, individual message failures within the stream are better handled as separate metrics (e.g., stream duration, reconnection frequency) based on team policy — this helps reduce false positives.

Library Selection Criteria: gobreaker vs mercari/go-circuitbreaker

These are two libraries frequently compared in the Go ecosystem. sony/gobreaker has a minimal API and lightweight dependencies, making it a good fit for cases like this post where you wrap everything inside an interceptor with a single Execute closure. mercari/go-circuitbreaker treats context.Context as a first-class citizen and exposes explicit result-marking APIs like Ignore/MarkAsSuccess, making it advantageous for codebases where you need to finely separate context cancellation from success/failure determination logic. There's rarely a reason to mix both libraries in one project, so it's better to establish a single organizational standard.

Design Decision Tree

A flow to reference when deciding how to implement.

Closing

What I really wanted to emphasize in this post isn't a flashy architecture — it's a very narrow point. If you implement the gRPC status code filtering inside the gobreaker Execute closure incorrectly, the circuit breaker silently malfunctions. It might count NOT_FOUND as a failure and open the circuit even though downstream is healthy, or miss ErrTooManyRequests and let raw errors leak up through the Half-Open threshold. The core of this pattern is clearly separating what to count as failure and what not to inside the closure before handing it to gobreaker, and extracting the real application error into a variable outside the closure to return as-is to the caller.

A service mesh isn't the right answer for every team. Fault isolation can start with just one library and one interceptor, and what makes that start safe is these two narrow lines of return values.

References

• Circuit Breaker recommendations · grpc/grpc-go Issue #5672 (GitHub)
• sony/gobreaker — Circuit Breaker implemented in Go (GitHub)
• mercari/go-circuitbreaker — Context-aware circuit breaker (GitHub)
• Setting up Resilience4j Circuit Breaker for gRPC Java Client (Deep Network GmbH)
• Resilience4j CircuitBreaker Official Docs
• Adding Circuit Breaker and Bulkheading Interceptor · go-grpc-middleware Issue #575 (GitHub)
• gRPC codes package Official Docs (Go)
• Michael Nygard, Release It! — Circuit Breaker Pattern Original