Why an Innocent Logging Filter Crashed Our Spring WebClient: Streaming, Netty, and the 256 KB Trap

Table of contents

The Incident: An Unexpected HTTP 500 on pageSize=1000

Earlier this week, we were testing a Backend-for-Frontend (BFF) service that fetches paginated product orders from downstream TMForum (TMF) microservices.

For standard small requests (pageSize=10, pageSize=20), everything worked like a charm. But the moment we tested a bulk query with pageSize=1000, the BFF instantly blew up with an HTTP 500 Internal Server Error:

org.springframework.core.io.buffer.DataBufferLimitException: 
Exceeded limit on max bytes to buffer : 262144
	at org.springframework.core.io.buffer.LimitedDataBufferList.raiseLimitException(LimitedDataBufferList.java:99)
	at org.springframework.core.io.buffer.LimitedDataBufferList.updateCount(LimitedDataBufferList.java:92)
	at org.springframework.core.io.buffer.LimitedDataBufferList.add(LimitedDataBufferList.java:59)
	at org.springframework.core.io.buffer.DataBufferUtils.lambda$join$4(DataBufferUtils.java:304)
    ...

At first glance, the stack trace looked bizarre:

• Why is our microservice failing at exactly 256 KB (262,144 bytes)?
• Our JVM had gigabytes of heap space free.
• Why did our outbound HTTP client care about buffering when it was just passing JSON along?

The culprit turned out to be an innocent-looking line of code registered on our outbound WebClient:

// What seemed harmless:
Logbook logbook = logbookProvider.getIfAvailable();
if (logbook != null) {
  builder.filter(new LogbookExchangeFilterFunction(logbook));
}

Here is the deep dive into what actually happened under the hood—from TCP packet streaming and Netty ByteBuf allocation to Spring’s reactive codecs and how to safely log HTTP traffic without crashing production.

1. The Core Conflict: Streaming vs. Logging

To understand the bug, we have to look at how modern non-blocking HTTP clients like Spring’s WebClient process network traffic compared to traditional blocking clients (like RestTemplate).

TCP Doesn’t Send Files; It Streams Packets

No network card sends a 500 KB JSON document in one single solid block.

• Standard Ethernet MTU (Maximum Transmission Unit) limits packets to 1,500 bytes.
• When the backend microservice sends a 500 KB response, the operating system kernel slices it into hundreds of small TCP segments over the wire.
• The receiving machine collects these segments in OS socket buffers and delivers them in small, streaming slices (typically 4 KB to 8 KB chunks).

How Spring WebClient Handles Data

Spring’s WebClient is built on top of Project Reactor and Netty.

Netty receives these 4 KB network chunks as ByteBuf objects. Spring wraps them in a reactive abstraction called DataBuffer, producing a stream:

$$\text{TCP Socket} \longrightarrow \text{Netty ByteBuf} \longrightarrow \text{Flux<DataBuffer>} \longrightarrow \text{Jackson Parser}$$

In normal reactive streaming, Jackson’s non-blocking JSON parser reads tokens (START_OBJECT, "orderId", "12345", etc.) directly off each DataBuffer as it arrives off the wire.

Where Do the Previous Chunks Go? (The Assembly Line Analogy)

A common question when thinking about streaming is: If the response arrives in chunks, don’t the previous chunks still have to sit in RAM until the request finishes?

The answer is no for raw bytes, yes for Java objects:

[Incoming TCP Packet: 4 KB]
         │
         ▼
 ┌───────────────┐
 │ Netty ByteBuf │ ──(Read by Jackson)──► Assembles Java Domain Object
 └───────────────┘                        (List<ProductOrder> in JVM Heap)
         │
         ▼
 [BUFFER RELEASED / RECYCLED] ◄── Memory freed immediately!

Think of an assembly line building bicycles:

1. A delivery worker brings Box 1 (4 KB).
2. The mechanic opens Box 1, takes out the pedal, screws it onto the bicycle frame (Java Object in JVM Heap), and immediately flattens Box 1 and tosses it into the recycling bin.
3. Worker brings Box 2. The mechanic takes out the wheel and discards Box 2.

At any single millisecond, only one small cardboard box exists in memory. The raw network buffers are consumed and recycled on the fly!

2. Enter Logbook: The Memory Trap

Zalando’s Logbook is a fantastic library for logging HTTP requests and responses (status codes, headers, and bodies). We used it extensively on our inbound Servlet controllers.

However, logging has a fundamental requirement that breaks streaming:

You cannot print an HTTP body to a log line until you have the ENTIRE body in your hands.

Furthermore, reactive byte streams can only be read once. If Logbook reads the bytes to log them, Jackson would find an empty stream.

To solve this, LogbookExchangeFilterFunction intercepts the ClientResponse:

1. It halts the reactive stream.
2. It invokes Spring’s DataBufferUtils.join(), collecting all incoming chunks into an in-memory list.
3. Once all chunks arrive, it stitches them together into one giant byte buffer.
4. It logs the full payload as a string.
5. It re-wraps the cached bytes into a new reactive Publisher so Jackson can read it.

Back to our bicycle analogy:

The supervisor walks into the workshop and shouts: “Nobody throw away any cardboard boxes! Stack every single box inside my tiny 256 sq ft office because I want a photograph of all 1,000 boxes together!“

3. Spring’s Safety Fuse: maxInMemorySize = 256 KB

To protect applications from Denial-of-Service (DoS) attacks and Out-Of-Memory (OOM) crashes caused by malicious or unbounded HTTP payloads, Spring WebFlux introduced a default safety guardrail in its codec layer:

$$\text{Default maxInMemorySize} = 256 \times 1,024 = 262,144 \text{ bytes (256 KB)}$$

Whenever any component calls DataBufferUtils.join() to turn a streaming reactive flow into an aggregated in-memory buffer, Spring attaches a strict counter (LimitedDataBufferList):

// Spring Framework DataBuffer internal logic:
this.byteCount += buffer.readableByteCount();
if (this.maxByteCount != -1 && this.byteCount > this.maxByteCount) {
    raiseLimitException(this.maxByteCount);
}

private void raiseLimitException(int maxByteCount) {
    throw new DataBufferLimitException("Exceeded limit on max bytes to buffer : " + maxByteCount);
}

Why pageSize=10 Worked, but pageSize=1000 Exploded

• With pageSize=10, the response was ~25 KB. Logbook buffered it, logged it, and passed it to Jackson. No rules were broken.
• With pageSize=1000, the backend returned ~160 rich order structures with addresses, line items, and product characteristics (~450 KB of uncompressed JSON).
• At byte 262,145, Spring’s internal safety tripwire snapped.
• Instead of waiting for more chunks, Spring immediately aborted the stream with DataBufferLimitException, failing the client call and returning a 500 Internal Server Error to the user!

4. The Solution: How to Log Safely Without Buffering

Step 1: Remove Logbook from Outbound WebClient

In TmfClientBaseConfig.java, we stripped out LogbookExchangeFilterFunction:

- Logbook logbook = logbookProvider.getIfAvailable();
- if (logbook != null) {
-   builder.filter(new LogbookExchangeFilterFunction(logbook));
- }

Inbound GUI $\leftrightarrow$ BFF logging still remains 100% active via the standard Spring MVC Servlet LogbookFilter. (Because Servlet requests use standard stream wrappers on Tomcat, they aren’t subject to WebFlux’s 256 KB reactive codec limit).

Step 2: For Outbound Visibility, Use Netty Wiretap!

If you still need full wire-level request and response logs when calling backend microservices during local development or debugging, what should you use?

The answer is Netty Wiretap, which is built directly into Netty’s HttpClient:

@Bean
public WebClient webClient(WebClient.Builder webClientBuilder, ObjectMapper mapper) {
  HttpClient httpClient = HttpClient.create();

  if (httpWiretapEnabled) {
    // Sniffs chunks directly on the TCP wire without buffering!
    httpClient = httpClient.wiretap(
        "reactor.netty.http.client.HttpClient",
        LogLevel.DEBUG,
        AdvancedByteBufFormat.TEXTUAL);
  }

  return webClientBuilder
      .clientConnector(new ReactorClientHttpConnector(httpClient))
      .build();
}

Then in application.yaml:

logging:
  level:
    reactor.netty.http.client: DEBUG

Why Netty Wiretap Doesn’t Crash:

Unlike Logbook, Netty Wiretap does not call DataBufferUtils.join() to assemble chunks in memory. It observes individual ByteBuf slices as they stream across the socket and dumps them to the console.

Payloads of 500 KB, 10 MB, or 100 MB stream through cleanly without ever violating Spring’s buffer limit.

Key Takeaways for Enterprise Spring Boot Developers

1. Streaming and Full-Body Logging are Fundamentally at Odds: Any filter that logs full response bodies must buffer the stream into memory, completely undoing the memory efficiency of reactive programming.
2. Spring’s 256 KB Limit is a Feature, Not a Bug: It prevents runaway network payloads from blowing up your JVM heap with direct byte buffers. Never blindly bump max-in-memory-size to 100 MB just to make an aggressive logging filter happy.
3. Use the Right Tool for the Right Layer:
   • Inbound HTTP (Client $\leftrightarrow$ BFF): Use Servlet filters like Zalando Logbook where request/response auditing is required.
   • Outbound Internal HTTP (BFF $\leftrightarrow$ Microservices): Rely on structured application logs (log.info()), tracing headers (TraceId / OpenTelemetry), or non-buffering wiretaps (reactor.netty.http.client).