feat(api): bidirectional chunk stream protocol by martinconic · Pull Request #5618 · ethersphere/bee

martinconic · 2026-09-16T07:13:34Z

Checklist

I have read the coding guide.
My change requires a documentation update, and I have done it.
I have added tests to cover my changes.
I have filled out the description and linked the related issues.

Description

Replaces the upload-only GET /chunks/stream with a request/response protocol: a client opens one websocket and pipelines chunk downloads and uploads over it, instead of paying for an HTTP request per chunk.

Closes #5417, closes #5599.

The first version of this PR added a download-only mode with hand-rolled ['D'][32-byte address]... framing. Following review (@acud, @aloknerurkar) it now uses protobuf, with typed Get and Put messages and a status per request, so one bad chunk no longer closes the connection and a client can recover after an error.

Breaking change: the legacy upload-only stream (raw [8-byte span][data] messages acknowledged with empty frames) is removed, as is the mode query parameter. A client still sending raw chunk messages has its connection closed with 1003. bee-js does not use the endpoint. OpenAPI is bumped to 9.0.0.

Protocol

Messages are defined in pkg/api/pb/chunkstream.proto, one protobuf message per websocket binary message. There is no subprotocol or mode to negotiate.

Request is a oneof of GetRequest (32-byte Address, optional CacheOption) and PutRequest (Address, Data, optional Stamp, optional BatchId).
Response is a oneof of GetResponse (Address, Status, Data, Error) and PutResponse (Address, Status, Error). Every request gets exactly one response, in completion order.
Correlation: requests carry no id; the chunk address is the key. Each response echoes the address of its request, and its type tells a download reply from an upload reply for the same chunk. A client must not have two requests of the same type for the same address in flight on one connection.
Status: OK, NOT_FOUND, ERROR, BAD_REQUEST, BUSY when a queue is full (retry with backoff), and UNAVAILABLE when no peer could serve a chunk, because none was usable or every peer was tried (the chunk may or may not exist; retry a limited number of times). STATUS_UNSPECIFIED = 0 is never sent.
Reading replies: clients should read replies while they send requests. A client that stops reading fills the socket, and the node then stops reading its requests until it does.

Uploads

Chunk type is inferred. The node rebuilds the chunk from Data and accepts it only if it produces the requested Address, as a CAC or as a SOC with a valid signature. Small SOC data also parses as a valid CAC; the address settles which one it is. A mismatch is answered with BAD_REQUEST.
Stamping, in order of precedence: a pre-signed Stamp on the request, the request's own BatchId, then the connection's Swarm-Postage-Batch-Id. Without a connection batch, one connection can upload to several batches. A connection with Swarm-Postage-Batch-Id is pinned to that batch, so a proxy that sets the header also decides the batch of every upload; a BatchId or pre-signed Stamp from a different batch is rejected with BAD_REQUEST. A bad connection batch fails the handshake; a bad per-request batch fails only that request.
Batches are checked for every chunk, as HTTP uploads do, so a batch that expires while the connection is open is no longer used. Pre-signed stamps are checked with Stamp.Valid, including their bucket and index, so a stamp the network would reject is not pushed.
Uploads are always direct. OK means the chunk has been pushed to the network, and is never sent before that. Swarm-Tag / ?swarm-tag= is rejected with 400 at the handshake, so a client expecting deferred uploads finds out immediately instead of having the tag ignored.

Downloads

Swarm-Cache / ?cache= set the connection default; CacheOption overrides it per request.
storage.ErrNotFound maps to NOT_FOUND, and topology.ErrNotFound (no peer could serve the chunk) to UNAVAILABLE, matching how GET /chunks and /bzz answer 503 for it.
Encrypted references (64 bytes) are not supported: a client fetches an encrypted chunk by its address and decrypts it.

Errors and closing

A failing request gets its own status and does not close the connection. The connection is closed only for a message that cannot be decoded or has no body (1003), a message over 8 KiB (1009), or node shutdown (1001, now that #5671 closes the API service on shutdown). A connection dropped without a close frame, for example after 15 minutes without a request, a pong or a ping, shows as 1006. Responses received before a close are final; a client should reconnect and resend every request that got no response. This is documented in the OpenAPI spec.

Notes for review

Structure: each connection is a chunkStream run under an errgroup: the reader, 8 download and 8 upload workers, a pinger, and a watcher that closes the socket when the stream ends (closing the socket is what releases a blocked read or write). The first error ends the stream, including a clean close by the client; request handlers return only reply write errors, so a failed request is still answered.
Encoding: gogo protobuf, following the p2p protocols' pb/ convention. No new dependency.
Fairness: downloads and uploads have separate queues (1024 each) and worker pools, so a backlog of slow uploads can't starve downloads. A full queue answers BUSY rather than queuing without bound.
Timeouts: a download is bounded by getter.DefaultFetchTimeout, an upload by 30s. The response write deadline is 5 minutes, so a client that pauses reading while its buffer drains doesn't get disconnected.
Keepalive: the node pings the client every WsPingPeriod, and a pong or a client ping extends the 15-minute read deadline just like a request does. gorilla treats a control frame write that times out as fatal, so pings and pongs get the same 5-minute deadline as responses: a keepalive cannot end the stream of a client that has paused reading.
Memory: a client message is limited to 8 KiB, under twice the largest valid request (a full SOC put with a stamp and batch id, about 4.4 KiB), so a full request queue holds at most a few MiB.
Work after the stream ends: requests still queued when the stream ends are dropped before they are validated, stamped or retrieved.
Metrics: open connections, deliveries by outcome (download success/not found/unavailable/error, upload success/error, and busy for requests turned away by a full queue), and fetch duration.

Testing

27 tests cover the stream: upload and download, interleaving, a Get and a Put for the same address in flight, per-request errors, a failed push, fairness under blocked uploads, a full queue, pre-signed stamps (including one for the wrong bucket) and per-request-batch stamping, a feed-sized SOC, a rejected tag, cache options, protocol violations, shutdown with an idle stream, with pending writes and with a client that has stopped reading, a protocol violation from a client that has stopped reading, a response write timeout followed by shutdown, and a client close ending the stream. Further tests cover the frame size limit, pongs and client pings keeping a stream open, keepalives in both directions while the client has paused reading, download error statuses including UNAVAILABLE, batch pinning, a batch expiring mid-stream (node stamping and pre-signed), the cache flag reaching the storer for every header, query and request combination, and no stamps or retrievals for requests still queued when the stream ends. Each of these was checked against code with its fix reverted, to confirm it fails when it should. The keepalive test depends on timing: with its fix reverted it fails in most runs (4 of 5 in each direction), and it never fails on correct code.

Two fuzz targets in chunk_stream_fuzz_test.go cover the untrusted input: FuzzChunkStreamRequest (protobuf decode plus a marshal round trip) and FuzzChunkForAddress (an accepted chunk must have the requested address and be a valid CAC or SOC).

make build, make lint, make test, the race detector and make fuzz FUZZPKG=./pkg/api/... all pass; the redocly problem count is unchanged from master.

Live test

Run against a Sepolia testnet full node built from this branch, with a Go client that uses pkg/api/pb:

handshake rejections and Origin checks;
200 CAC uploads pushed to the network (26–30 chunks/s with 8 upload workers) and downloaded back byte-identical (54–103 chunks/s, varying with the network);
full-size and feed-sized SOCs;
per-request batch, pinning, and pre-signed stamps from /envelope;
protocol violations, the 8 KiB limit, and a raw legacy upload message closed with 1003;
BUSY under a flood;
a client that stops reading for 70s (longer than the 60s ping period) and then gets all 1000 replies;
a 130s idle stream that receives the node's pings;
SIGTERM with requests in flight: the client gets 1001 "node shutting down".

Missing chunks are answered STATUS_UNAVAILABLE in about 2s, never NOT_FOUND, which matches the 503 that GET /chunks returns for them.

AI Disclosure

This PR contains code that has been generated by an LLM.
I have reviewed the AI generated code thoroughly.
I possess the technical expertise to responsibly review the code generated in this PR.

acud · 2026-09-17T22:38:42Z

+	s.metrics.ChunkStreamDeliveryCount.WithLabelValues("success").Inc()
+
+	chunkData := chunk.Data()
+	resp := make([]byte, 1+swarm.HashSize+len(chunkData))


not sure about this custom serialization by hand thing... it is just specced out in the openapi spec and assumed that implementers should implement it by hand. it is fragile and breakable. why not use some sort of standard serialization format to both decode the request and encode the response?

aloknerurkar

Since we are making breaking changes and also building on the serialization thread — I'd suggest going further than reworking the download framing: use one connection for both directions, with request/response semantics and a request id.

Two things this fixes beyond tidiness:

Upload is round-trip bound. The upload loop is strictly sequential — read → put → ack → read, one chunk in flight. Each chunk costs a full RTT, so on a 50ms link that's ~20 chunks/sec regardless of bandwidth. Download already has a 16-worker pool; upload has none. stamper.Stamp already takes issuer.mtx (pkg/postage/stamper.go:43), so concurrent stamping is safe — the ceiling is the protocol, not the storage layer.
Neither direction is recoverable. Download replies carry no id, and the upload ack is successWsMsg = []byte{} — an empty frame with no address at all. Clients correlate purely by ordering. On any error both paths call sendErrorClose and drop the connection, so a client that had N requests outstanding cannot tell which completed. For bulk upload that means restarting from zero.

A shared envelope — [type][8-byte request-id][payload] for requests, [type][8-byte request-id][status][payload] for responses — would give:

one read loop feeding one typed job channel, with a worker pool serving both Get and Put (removes the duplicated read/deadline/close-handler logic between the two handlers)
pipelined uploads instead of one-at-a-time
per-request error status instead of connection teardown, so a single bad chunk no longer kills the stream
correlation for recovery after an error close

I'd keep this as raw binary rather than JSON-RPC or protobuf. JSON-RPC costs more on the wire with base64 and ~15x encode/decode — self-defeating for an endpoint that exists to cut per-chunk overhead. Protobuf is bee's p2p convention but has never appeared in pkg/api; requiring a schema compiler would be a new burden on bee-js. The binary envelope stays a four-line DataView parse in the browser with no dependency.

The upload stream endpoint was not written with care. I was going through the code and there is a lot of scope to cleanup. Multiple putters defined, deferred/direct upload semantics, stamped and unstamped chunks etc. I feel that this would make it much more usable for clients and also close to what elad mentioned about having grpc style chunk get/put API.

aloknerurkar · 2026-09-22T20:53:52Z

+}
+
+// fetchAndSendChunk retrieves a single chunk and writes exactly one response
+// frame for it. That one-frame-per-requested-address invariant is what lets a


Exactly one reply per requested address, and "a dropped frame is indistinguishable from a slow one" because replies carry no request id. But the protocol-error paths in the read loop (sendErrorClose + return on bad message type, bad length, unknown opcode, oversized batch) void every in-flight request silently.

A client that sends 256 addresses and then trips CloseMessageTooBig on its next frame has no way to determine which of the first 256 were answered. With no request id and no per-batch boundary marker in the reply stream, the only recovery is to discard everything and re-request.

The OpenAPI spec documents the close codes, but not that pending replies are lost when they fire. At minimum that consequence belongs in the spec. A framing with a request/batch id — which is largely what @acud point about a standard serialization format would give you for free — would make it recoverable instead.

martinconic · 2026-09-23T10:39:18Z

Since we are making breaking changes and also building on the serialization thread — I'd suggest going further than reworking the download framing: use one connection for both directions, with request/response semantics and a request id.

Two things this fixes beyond tidiness:

Upload is round-trip bound. The upload loop is strictly sequential — read → put → ack → read, one chunk in flight. Each chunk costs a full RTT, so on a 50ms link that's ~20 chunks/sec regardless of bandwidth. Download already has a 16-worker pool; upload has none. stamper.Stamp already takes issuer.mtx (pkg/postage/stamper.go:43), so concurrent stamping is safe — the ceiling is the protocol, not the storage layer.

Neither direction is recoverable. Download replies carry no id, and the upload ack is successWsMsg = []byte{} — an empty frame with no address at all. Clients correlate purely by ordering. On any error both paths call sendErrorClose and drop the connection, so a client that had N requests outstanding cannot tell which completed. For bulk upload that means restarting from zero.

A shared envelope — [type][8-byte request-id][payload] for requests, [type][8-byte request-id][status][payload] for responses — would give:

one read loop feeding one typed job channel, with a worker pool serving both Get and Put (removes the duplicated read/deadline/close-handler logic between the two handlers)

pipelined uploads instead of one-at-a-time

per-request error status instead of connection teardown, so a single bad chunk no longer kills the stream

correlation for recovery after an error close

I'd keep this as raw binary rather than JSON-RPC or protobuf. JSON-RPC costs more on the wire with base64 and ~15x encode/decode — self-defeating for an endpoint that exists to cut per-chunk overhead. Protobuf is bee's p2p convention but has never appeared in pkg/api; requiring a schema compiler would be a new burden on bee-js. The binary envelope stays a four-line DataView parse in the browser with no dependency.

The upload stream endpoint was not written with care. I was going through the code and there is a lot of scope to cleanup. Multiple putters defined, deferred/direct upload semantics, stamped and unstamped chunks etc. I feel that this would make it much more usable for clients and also close to what elad mentioned about having grpc style chunk get/put API.

Thank you for this, sounds like a good change. @acud do you also agree?

acud · 2026-09-23T18:00:36Z

@martinconic i generally encourage the initiative from @aloknerurkar, but i'm still not 100% convinced a custom encoding is right.
@aloknerurkar, to answer some of your points:

the per-chunk overhead we are reducing has to do more with TCP socket open/close overhead and all sorts of stamper serialization overhead, not message encoding
i agree that it would be better to have things as you suggested
i'd still suggest to have a standard encoding library, ideally protobuf, because it has tooling out there and you don't have to send people (or agents) to do endianness or the likes in a browser js library
also not sure, if all is managed on one websocket, how much we have to deal with partial reads/flushes/buffering

i think we can progress with this proposal, and make serialization a pluggable implementation. but in general i'd encourage to move away from hand written serialization at least on wire-formats (persistence is a different story).

aloknerurkar · 2026-09-23T18:08:06Z

@martinconic i generally encourage the initiative from @aloknerurkar, but i'm still not 100% convinced a custom encoding is right. @aloknerurkar, to answer some of your points:

the per-chunk overhead we are reducing has to do more with TCP socket open/close overhead and all sorts of stamper serialization overhead, not message encoding

i agree that it would be better to have things as you suggested

i'd still suggest to have a standard encoding library, ideally protobuf, because it has tooling out there and you don't have to send people (or agents) to do endianness or the likes in a browser js library

also not sure, if all is managed on one websocket, how much we have to deal with partial reads/flushes/buffering

i think we can progress with this proposal, and make serialization a pluggable implementation. but in general i'd encourage to move away from hand written serialization at least on wire-formats (persistence is a different story).

I am fine with protobuf. The only reason I didn't suggest it is that bee-js/javascript will be primary consumer if this is successful and I remember there were issues with javascript ecosystem when working with protobuf. I spoke to claude and I see that there is a new typescript compiler which makes things easier. So I am totally on board!

…ownload

…tream

…ds on chunk stream

…ownload # Conflicts: # openapi/Swarm.yaml

Add fuzz targets for decoding chunk stream requests and for chunkForAddress, which validates uploaded chunks against the requested address. Add a test for a stream that ends on a protocol violation while its workers are blocked writing to a client that has stopped reading. The shutdown watcher has exited by then, so only closing the socket before waiting on the workers releases them.

Close the socket when the stream ends on a failed response write, so the read loop is released and node shutdown does not find the stream still open. Ping the client and let its pongs extend the read deadline, so a client that is only waiting for replies is not dropped. Lower the frame size limit to 8 KiB, which bounds the memory a full request queue can hold. Validate an uploaded SOC in a single pass, and do not stamp requests still queued when the stream ends.

…ownload

Report a chunk that no peer could serve as STATUS_UNAVAILABLE instead of STATUS_NOT_FOUND, as GET /chunks and /bzz now answer 503 for it. Pin a connection with Swarm-Postage-Batch-Id to that batch: a request batch id or pre-signed stamp from another batch is rejected. Send pings from their own goroutine and answer client pings with the response write deadline. gorilla treats a control frame write that times out as fatal, so a short deadline could end the stream of a client that had paused reading. Do not start a retrieval for a request still queued when the stream ends, document close code 1006, and test that the cache option reaches the storer.

A node shutdown closes the API's http.Server only, so a chunk stream is dropped without a close frame (1006). 1001 is sent only when the API service itself is closed.

…ownload # Conflicts: # pkg/api/router.go

Extend the read deadline on client pings as well as pongs, so a client that keeps the stream alive with its own pings is not dropped. Skip validating an upload request that is still queued when the stream ends, and keep the check before stamping, since the stream can end while the chunk is validated. Count STATUS_BUSY replies in the chunk stream delivery metric.

…ownload

/chunks/stream now always speaks the protobuf request/response protocol. The upload-only mode, the mode query parameter and the swarm-chunk-stream subprotocol are removed; a client still sending raw chunk messages has its connection closed with 1003. OpenAPI is bumped to 9.0.0. Run each stream as a chunkStream under an errgroup: the reader, the download and upload workers, the pinger and a watcher that closes the socket when the stream ends. The first error ends the stream, and request handlers return only reply write errors, so a failed request is still answered. Look up the postage batch for every uploaded chunk instead of caching it per connection, so a batch that expires while the stream is open is no longer used, and validate pre-signed stamps with Stamp.Valid, which also checks the bucket and index. Document that encrypted references are not supported, that clients should read replies while sending, and that node shutdown sends 1001. BREAKING CHANGE: the upload-only /chunks/stream mode is removed.

feat: download mode for the chunk stream endpoint

064ec06

github-actions Bot assigned martinconic Sep 16, 2026

martinconic marked this pull request as draft September 16, 2026 12:44

martinconic marked this pull request as ready for review September 16, 2026 15:30

martinconic requested review from acud, akrem-chabchoub, aloknerurkar, gacevicljubisa, janos and sbackend123 September 16, 2026 15:30

acud reviewed Sep 17, 2026

View reviewed changes

aloknerurkar reviewed Sep 22, 2026

View reviewed changes

martinconic added 2 commits September 23, 2026 11:21

refactor: address review feedback on the chunk download stream

676c748

fix: close the socket before waiting on stalled download workers

daf2fb9

martinconic added 2 commits September 24, 2026 20:11

feat: protobuf request/response protocol for the chunk stream

1e79780

Merge remote-tracking branch 'origin/master' into feat/chunk-stream-d…

4bdf24d

…ownload

acud reviewed Sep 29, 2026

View reviewed changes

Comment thread pkg/api/pb/chunkstream.proto

Comment thread pkg/api/pb/chunkstream.proto Outdated

aloknerurkar reviewed Sep 29, 2026

View reviewed changes

Comment thread pkg/api/pb/chunkstream.proto Outdated

Comment thread pkg/api/pb/chunkstream.proto Outdated

Comment thread pkg/api/pb/chunkstream.proto

martinconic added 3 commits September 29, 2026 14:03

refactor(api): use address correlation and typed responses in chunk s…

972c344

…tream

refactor(api): infer chunk type, per-request batch, direct-only uploa…

d31c7a6

…ds on chunk stream

test(api): wait for the pushed chunk in the chunk stream upload test

bef0d8e

martinconic requested review from acud and aloknerurkar October 6, 2026 08:26

martinconic added 4 commits October 15, 2026 14:02

Merge remote-tracking branch 'origin/master' into feat/chunk-stream-d…

ba4c6be

…ownload # Conflicts: # openapi/Swarm.yaml

Merge remote-tracking branch 'origin/master' into feat/chunk-stream-d…

8a49dad

…ownload

martinconic added 2 commits October 21, 2026 15:10

docs(api): correct the chunk stream close code on node shutdown

5930061

A node shutdown closes the API's http.Server only, so a chunk stream is dropped without a close frame (1006). 1001 is sent only when the API service itself is closed.

This was referenced Oct 21, 2026

Node shutdown never calls api.Service.Close, so websocket clients get 1006 instead of 1001 #5670

Closed

fix(node): close the api service on shutdown #5671

Merged

martinconic added 2 commits October 21, 2026 16:53

Merge remote-tracking branch 'origin/master' into feat/chunk-stream-d…

80eb2fd

…ownload # Conflicts: # pkg/api/router.go

martinconic changed the title ~~feat: download mode for the chunk stream endpoint~~ feat(api): bidirectional chunk stream protocol Oct 21, 2026

aloknerurkar reviewed Oct 22, 2026

View reviewed changes

martinconic added 2 commits October 23, 2026 09:24

Merge remote-tracking branch 'origin/master' into feat/chunk-stream-d…

7346e2a

…ownload

martinconic requested a review from aloknerurkar October 23, 2026 07:36

Conversation

martinconic commented Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Description

Protocol

Uploads

Downloads

Errors and closing

Notes for review

Testing

Live test

AI Disclosure

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

acud Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

aloknerurkar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

aloknerurkar Sep 22, 2026

Choose a reason for hiding this comment

Uh oh!

martinconic commented Sep 23, 2026

Uh oh!

acud commented Sep 23, 2026

Uh oh!

aloknerurkar commented Sep 23, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

martinconic commented Sep 16, 2026 •

edited

Loading