Deliver upstream feedback to remote tasks and stop spurious "No remote destination available" warnings by ErnestasVaiciukeviciusAdform · Pull Request #9175 · apache/storm

ErnestasVaiciukeviciusAdform · 2026-10-16T13:12:13Z

What is the purpose of the change

To fix #9174.

When topology.upstream.feedback.enable=true, workers flood their logs with
WARN o.a.s.utils.TransferDrainer - No remote destination available for task <id>,
and — more importantly — the upstream-feedback tuples that trigger those warnings are
dropped. Upstream feedback is therefore only delivered to tasks on the same
worker; feedback destined for upstream tasks on other workers never arrives.

Root cause. Upstream feedback is routed directly to upstream (source) component task ids
via Task.sendUnanchoredFeedback, bypassing stream grouping
(Executor.sendUpstreamFeedback / Executor.computeUpstreamTaskIds). But a worker's routable
remote destinations (WorkerState.cachedTaskToNodePort) and its connections are built only
from outboundTasks, which WorkerState.workerOutboundTasks() derives purely from the
downstream-target components of local tasks. addUpstreamFeedback declares
FEEDBACK_STREAM_ID only as an output stream (no grouping/input edge), so feedback recipients
are never added to outboundTasks. For a remote recipient, TransferDrainer.send finds no
node, logs the warning, and drops the tuple.

Why it looks intermittent / misleading. outboundTasks membership is decided per
component: if a component is a downstream target of any local task, all of its tasks become
routable. So a worker emits no warning for feedback to upstream component X whenever X
also happens to be a downstream target of some co-located task — the feedback rides the
existing connection and looks healthy. The same feedback edge warns and drops on a worker that
hosts upstream tasks of X but none of X's downstream consumers. Consequently "no
warnings" does not mean feedback is delivered; it only reflects task placement (e.g. the
warnings vanish entirely if every worker happens to host a task of every component), which can
mislead operators into thinking the feature works.

Fix. When upstream feedback is enabled, also include the non-system source (upstream)
components of the locally-hosted components when computing outboundTasks, mirroring
Executor.computeUpstreamTaskIds() at worker granularity. This makes every feedback recipient
(bolt or spout) a routable remote destination with a connection, so feedback is delivered
correctly and the spurious warning disappears — independent of task placement. The change is
confined to WorkerState.workerOutboundTasks() and only takes effect when
topology.upstream.feedback.enable=true (off by default).

How was the change tested

Added WorkerOutboundTasksTest (storm-client) covering a spout -> bolt1 -> bolt2 topology:
- feedback disabled → only downstream-target tasks are routable (upstream spout tasks excluded);
- feedback enabled → upstream source (spout) tasks are additionally routable, downstream tasks preserved;
- the misleading placement-dependent case → when a worker co-locates a spout and a bolt2
  task, bolt1 is already routable as the spout's downstream target even with feedback off,
  documenting that absence of warnings does not imply correct delivery.
Commands run (Maven 3.10, Java 25):
- mvn -pl storm-client -am install -DskipTests — BUILD SUCCESS
- mvn -pl storm-client test -Dtest=WorkerOutboundTasksTest — Tests run: 3, Failures: 0, Errors: 0
- mvn -pl storm-client test -Dtest=WorkerStateTest (existing tests, no regression) — Tests run: 3, Failures: 0, Errors: 0
- mvn -pl storm-client checkstyle:check@validate — 0 Checkstyle violations

reiabreu

This review was generated with the help of an LLM (Claude).

Inline notes below.

GGraziadei · 2026-10-17T19:18:00Z

Hi @ErnestasVaiciukeviciusAdform thanks a lot for reporting and addressing this bug!

If you have bandwidth, can you execute with your changes this topology in a two supervisors cluster with flag on/off?

https://github.com/apache/storm/blob/master/examples%2Fstorm-perf%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fstorm%2Fperf%2FJitterAwareGroupingTopology.java

ErnestasVaiciukeviciusAdform · 2026-10-19T13:09:40Z

Hi @ErnestasVaiciukeviciusAdform thanks a lot for reporting and addressing this bug!

If you have bandwidth, can you execute with your changes this topology in a two supervisors cluster with flag on/off?

https://github.com/apache/storm/blob/master/examples%2Fstorm-perf%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fstorm%2Fperf%2FJitterAwareGroupingTopology.java

Sure.
Below what i've got running JitterAwareGroupingTopology on codebase with PR changes (2 runs for each case):

4 workers, 2 supervisors - feedback disabled (-c topology.upstream.feedback.enable=false)
120,4,12,12,38640,644,1,2680,2300,44,11400.035,11400.035
second run
120,4,12,12,33380,556,1,2040,2020,34,13016.901,13016.901

4 workers, 2 supervisors - feedback enabled (-c topology.upstream.feedback.enable=true)
120,4,12,12,49680,828,1,3580,3060,59,8996.098,8996.098
second run
120,4,12,12,46240,770,1,3340,2760,55,9766.036,9766.036

Also for comparison on unmodified 3.1.0:

4 workers - feedback disabled
120,4,12,12,39620,660,1,3000,2300,50,11436.565,11436.565

4 workers - feedback enabled
120,4,12,12,38220,637,1,2660,2300,44,11365.287,11365.287

GGraziadei · 2026-10-19T17:18:18Z

Hi @ErnestasVaiciukeviciusAdform thanks!
What about repeating the benchmark with this configuration and running for 240s?

topology.executor.receive.control.queue.enable: true/false
topology.max.spout.pending: 500
topology.upstream.feedback.enable: true/false

Reducing p99 latency of 4s(~30%) is a lot, it requires additional validation.

GGraziadei

A few requests before merging:

Please rebase on master
Document the connection fan-in in docs/Performance.md (the only place describing the flag) and/or the TOPOLOGY_UPSTREAM_FEEDBACK_ENABLE javadoc: with the flag on, every worker hosting a user bolt now opens a Netty client connection to every worker hosting a task of its upstream components
A few small things on the tests, inline.

…e destination available" warnings When topology.upstream.feedback.enable is true, feedback tuples are routed directly to upstream (source) component tasks, but a worker's routable remote destinations (cachedTaskToNodePort) and connections are built only from outboundTasks, which contained just the downstream-target components of local tasks. Feedback recipients were therefore missing, so TransferDrainer logged "No remote destination available for task" and silently dropped the tuple; cross-worker feedback was never delivered. Whether the warning appeared depended on task placement, which was misleading. Include the non-system source (upstream) components of local tasks in outboundTasks when feedback is enabled, mirroring Executor.computeUpstreamTaskIds at worker granularity, so feedback recipients are routable and connected. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…cker task behavior

…d add feedback configuration tests

…back to WorkerOutboundTasksTest

ErnestasVaiciukeviciusAdform · 2026-10-20T09:06:38Z

Hi @ErnestasVaiciukeviciusAdform thanks! What about repeating the benchmark with this configuration and running for 240s?
topology.executor.receive.control.queue.enable: true/false
topology.max.spout.pending: 500
topology.upstream.feedback.enable: true/false
Reducing p99 latency of 4s(~30%) is a lot, it requires additional validation.

Checked all 4 variations (on code rebased on master).

-c topology.executor.receive.control.queue.enable=false -c topology.upstream.feedback.enable=false

240,4,12,12,28300,471,1,2060,1880,34,14326.030,14326.030

-c topology.executor.receive.control.queue.enable=false -c topology.upstream.feedback.enable=true

240,4,12,12,45760,762,1,3020,3040,50,9497.082,9497.082

-c topology.executor.receive.control.queue.enable=true -c topology.upstream.feedback.enable=false

240,4,12,12,30100,501,1,1880,2020,31,14314.659,14314.659

-c topology.executor.receive.control.queue.enable=true -c topology.upstream.feedback.enable=true

240,4,12,12,47820,797,1,3080,3120,51,8984.315,8984.315

… implementations and simplify method signatures

…ead and security implications in Config.java and Performance.md

ErnestasVaiciukeviciusAdform · 2026-10-20T10:15:14Z

A few requests before merging:

Please rebase on master

Document the connection fan-in in docs/Performance.md (the only place describing the flag) and/or the TOPOLOGY_UPSTREAM_FEEDBACK_ENABLE javadoc: with the flag on, every worker hosting a user bolt now opens a Netty client connection to every worker hosting a task of its upstream components

A few small things on the tests, inline.

Addressed all the points.
Documentation extended in both Performance.md and javadoc.

GGraziadei requested review from GGraziadei, reiabreu and rzo1 October 16, 2026 17:29

reiabreu reviewed Oct 17, 2026

View reviewed changes

reiabreu added the bug label Oct 17, 2026

reiabreu added this to the 3.2.0 milestone Oct 17, 2026

GGraziadei reviewed Oct 19, 2026

View reviewed changes

Comment thread storm-client/test/jvm/org/apache/storm/daemon/worker/WorkerOutboundTasksTest.java

Comment thread storm-client/test/jvm/org/apache/storm/daemon/worker/WorkerOutboundTasksTest.java Outdated

ErnestasVaiciukeviciusAdform and others added 7 commits October 20, 2026 10:20

Remove unnecessary test, cleanup comments

9f9728d

Upstream feedback setup: use topologyConf and exclude system components

743b6c9

Enhance WorkerOutboundTasksTest: add tests for feedback routing and a…

f9c6ade

…cker task behavior

Refactor WorkerState test utilities: enhance getWorkerState method an…

84d985b

…d add feedback configuration tests

Apply suggestions from compiler warnings

637c604

Move testTopologyConfigEnabledFeedback/testTopologyConfigDisabledFeed…

0f60ebe

…back to WorkerOutboundTasksTest

ErnestasVaiciukeviciusAdform force-pushed the fix/upstream-feedback-remote-destination branch from c18e109 to 0f60ebe Compare October 20, 2026 07:22

ErnestasVaiciukeviciusAdform added 4 commits October 20, 2026 12:35

Improve testTopologyConfigDisabledFeedback testcase

d0b8446

Refactor NoopSpout and NoopBolt classes: remove unnecessary interface…

ab849e8

… implementations and simplify method signatures

Enhance documentation for upstream feedback: clarify connection overh…

87e1af1

…ead and security implications in Config.java and Performance.md

WorkerOutboundTasksTest: made test naming consistent

3ce24e8

Navigation Menu

Repository navigation

Deliver upstream feedback to remote tasks and stop spurious "No remote destination available" warnings - #9175

Deliver upstream feedback to remote tasks and stop spurious "No remote destination available" warnings#9175
ErnestasVaiciukeviciusAdform wants to merge 11 commits into
apache:masterfrom
ErnestasVaiciukeviciusAdform:fix/upstream-feedback-remote-destination

ErnestasVaiciukeviciusAdform commented Oct 16, 2026

Uh oh!

reiabreu left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

GGraziadei commented Oct 17, 2026 •

edited

Loading

Uh oh!

ErnestasVaiciukeviciusAdform commented Oct 19, 2026

Uh oh!

GGraziadei commented Oct 19, 2026 •

edited

Loading

Uh oh!

GGraziadei left a comment

Uh oh!

Uh oh!

Uh oh!

ErnestasVaiciukeviciusAdform commented Oct 20, 2026

Uh oh!

ErnestasVaiciukeviciusAdform commented Oct 20, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

ErnestasVaiciukeviciusAdform commented Oct 16, 2026

What is the purpose of the change

How was the change tested

Uh oh!

reiabreu left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

GGraziadei commented Oct 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ErnestasVaiciukeviciusAdform commented Oct 19, 2026

Uh oh!

GGraziadei commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GGraziadei left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ErnestasVaiciukeviciusAdform commented Oct 20, 2026

Uh oh!

ErnestasVaiciukeviciusAdform commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

reiabreu left a comment •

edited

Loading

GGraziadei commented Oct 17, 2026 •

edited

Loading

GGraziadei commented Oct 19, 2026 •

edited

Loading

ErnestasVaiciukeviciusAdform commented Oct 20, 2026 •

edited

Loading