Skip to content

ir/timers: never stop epoch ticking - #4274

Merged
carpawell merged 1 commit into
masterfrom
fix/ir-do-not-try-epoch-tick-after-stuck
Nov 17, 2026
Merged

carpawell merged 1 commit into
masterfrom
fix/ir-do-not-try-epoch-tick-after-stuck

Conversation

@carpawell

Copy link
Copy Markdown
Member

If anything blocks epoch tick, do not simply try it once and wait for the next chain "NewEpoch" notification, instead, try to predict the next tick time and never stop trying. An epoch stuck is a critical error in the system, IRs must try their best.

@carpawell carpawell self-assigned this Nov 5, 2026
@carpawell
carpawell force-pushed the fix/ir-do-not-try-epoch-tick-after-stuck branch 4 times, most recently from bd9565d to eed742b Compare November 5, 2026 15:24
@codecov

codecov Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 30.76923% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 33.28%. Comparing base (79a04b4) to head (6c5113b).

Files with missing lines Patch % Lines
pkg/innerring/innerring.go 0.00% 9 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4274      +/-   ##
==========================================
+ Coverage   33.24%   33.28%   +0.04%     
==========================================
  Files         673      673              
  Lines       41881    41888       +7     
==========================================
+ Hits        13924    13944      +20     
+ Misses      27957    27944      -13     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@roman-khimov roman-khimov left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not resetting in error handler? This seems dangerous by default (some other timers can have problems with this).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The timer can submit the epoch transaction on every block when no valid duration is available, so the active moderate-severity issue must be fixed.

1 open issue
What changed in this PR

Updates epoch timers to keep retrying epoch ticks using the configured epoch duration when NewEpoch notifications are delayed or unavailable.

Changes:

  • Replaces one-shot completion state with persistent epoch-duration scheduling.
  • Reschedules the next epoch tick after each attempted handler execution.
  • Stores the epoch duration during timer resets.
  • A moderate issue remains when epochDur is zero: handlers may be invoked once per block instead of preserving one-shot behavior until a valid duration is available.
File Description
pkg/​timers/​timer.go Implements recurring epoch-tick scheduling and reset duration tracking.

Review effort: Lite

Comment thread pkg/timers/timer.go Resolved
@End-rey

End-rey commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

This looks like the timer commit from #4272, with both issues I pointed out there fixed.

@carpawell

Copy link
Copy Markdown
Member Author

Why not resetting in error handler?

What is "error handler"? Current IR does not wait for new epoch tx inclusion, it just sends notary request, do you want it to wait for NewEpoch notification (or vub expiration)?

This seems dangerous by default (some other timers can have problems with this).

There are no other timers, only the new epoch will have this exclusive opportunity. If everything is ok, it will be reset likely almost immediately by the NewEpoch event. But if smth is wrong, the next tick will, at least, happen.

@roman-khimov

@carpawell
carpawell force-pushed the fix/ir-do-not-try-epoch-tick-after-stuck branch from eed742b to b3ceaa5 Compare November 11, 2026 15:16
@carpawell

Copy link
Copy Markdown
Member Author

@roman-khimov, treat this PR like time.Timer -> time.Ticker evolution. Reset will always be called after a real NewEpoch event is handled. What exactly bothers you?

Comment thread CHANGELOG.md Resolved
@carpawell
carpawell force-pushed the fix/ir-do-not-try-epoch-tick-after-stuck branch from b3ceaa5 to b6f06e6 Compare November 17, 2026 12:19
@carpawell

Copy link
Copy Markdown
Member Author

/hal review

@hal0x9000 hal0x9000 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The re-arm works for a non-zero epoch duration and does not change storage-node behaviour. Two things need fixing before merge: a zero epoch duration now fires the epoch tick on every block, and InitialEpochDuration gets overwritten before anything reads it. There is also still no test.

On the concern about other timers: the only other timers.NewTimers user is the SN size-load reporter (cmd/neofs-node/container.go:155-172). It registers only DeltaTicks, and their done handling is unchanged, so the new eHandlers re-arm loop runs over an empty slice there. The IR epoch tick is the only handler affected.

What I checked

  • UpdateTime/Reset in full, the IR wiring (New → initTimers, Start → initConfigFromBlockchain → resetEpochTimer → Reset, the header handler at innerring.go:203-208), processNewEpoch/processNewEpochTick, and the SN caller.
  • I ran a throwaway test against a copy of pkg/timers (nothing committed): Reset(1000, 500) followed by UpdateTime at 1400, 1500, 1600, 1999 and 2000.
    • Head: two fires (1500 and 2000).
    • Base: one fire.
    • Reset(1000, 0) followed by three blocks: three fires on head, one on base.

Findings (details inline)

  1. Zero duration: handlers fire on every block. Before this PR they fired once.
  2. InitialEpochDuration is never read, so it, the extra RPC and the new New() error path can go.
  3. No test for the new behaviour.
  4. CHANGELOG wording.

Not introduced here, for the record. Suppose processNewEpoch sets the counter (process_epoch.go:30) and then ResetEpochTimer fails (:39-41), and this IR had not fired yet. The stale nextTickAt then fires NewEpoch(counter+1) on the next block. That already happened before this PR. The PR only lets it repeat once per epoch afterwards instead of stopping.

Not determined: whether the netmap contract rejects a zero EpochDuration (it is in neofs-contracts, not checked). pkg/morph/client/netmap/config.go:47 passes the value through unvalidated.


Generated review by hal0x9000, covering b6f06e6.

Comment thread pkg/timers/timer.go
// prevent epoch stuck: even if smth is wrong with the epochs,
// the next tick will be done in time; if everything is ok,
// the following Reset call will tune the next tick more precisely
if et.epochDur != 0 {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With epochDur == 0, nextTickAt is never advanced past curr, so nextTickAt <= curr holds on every later block. Each block then calls HandleNewEpochTick → netmapClient.NewEpoch(...). Before this PR, done limited that to one call. Reproduced: after Reset(1000, 0), three UpdateTime calls fire three times on head and once on base.

A zero duration from the contract is not hypothetical in this tree. cmd/neofs-node/netmap.go:75-82 (b0b1270, "Zero value is not acceptable ... panic was observed at least once") falls back to a default for exactly this case.

Suggestion: keep one-shot semantics for a zero duration. For example, push nextTickAt out of reach when epochDur == 0, or reject or replace zero in resetEpochTimer, where the duration is read. Either way, a zero value can't turn into a per-block transaction storm.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a zero value can't turn into a per-block transaction storm

why not? correct way to me, contract should never return 0 epoch time

Comment thread pkg/timers/timer.go Resolved
epochH, stopEpochHandlerFunc := util.SingleAsyncExecutingInstance(server.netmapProcessor.HandleNewEpochTick)
server.closers = append(server.closers, func() error { stopEpochHandlerFunc(); return nil })

epochDuration, err := server.netmapClient.EpochDuration()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This value can't be observed. Server.Start calls initConfigFromBlockchain (innerring.go:159) → resetEpochTimer(0) (:1022) → epochTimers.Reset, which sets epochDur (timer.go:85). Only after that is the header handler that calls UpdateTime registered (:203-208). If that reset fails, Start returns before the handler exists. So Reset always overwrites InitialEpochDuration before the first UpdateTime.

Dropping EpochTicks.InitialEpochDuration would also remove:

  • this extra RPC
  • the new error return of initTimers and the failure path in New()
  • the time import
  • the second seconds→ms conversion style (uint64(time.Second/time.Millisecond) here vs const msInS = 1000 at :264)

timer.go then only needs epochDur = dur in Reset.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropping EpochTicks.InitialEpochDuration would also remove:

...

None of these is a problem

So Reset always overwrites InitialEpochDuration before the first UpdateTime

it is safer to have some prediction of the next ticks before actual reset is called later

Comment thread pkg/timers/timer.go
return
}

if et.nextTickAt <= curr {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pkg/timers has no tests, and this changes when the epoch tick fires. The profile asks for a bug fix to come with a test that fails without it. A small require table test over Reset/UpdateTime that counts Tick calls needs no dependencies. It should cover:

  • re-fire after epochDur without Reset (fails on base: 1 fire instead of 2)
  • Reset moving the next tick
  • dur == 0
  • delta ticks unaffected

Comment thread CHANGELOG.md Resolved
@carpawell

Copy link
Copy Markdown
Member Author

Long story short:

The IR epoch tick is the only handler affected.

@carpawell

Copy link
Copy Markdown
Member Author

/hal can you add a unit test for it in this PR?

@hal0x9000

hal0x9000 Bot commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

I can't add it in this PR. A /hal review run only reads and comments, and this branch is yours, so I can't push to it. Here is a test you can add as pkg/timers/timer_test.go (the package has no test file yet):

package timers

import (
	"testing"

	"github.com/stretchr/testify/require"
)

func TestEpochTimers_UpdateTime(t *testing.T) {
	newTimers := func(initDur uint64) (*EpochTimers, *int) {
		var n int
		return NewTimers(EpochTicks{
			InitialEpochDuration: initDur,
			NewEpochTicks:        []Tick{func() { n++ }},
		}), &n
	}

	t.Run("keeps ticking without reset", func(t *testing.T) {
		et, n := newTimers(0)
		et.Reset(1000, 500)

		et.UpdateTime(1400)
		require.Equal(t, 0, *n)
		et.UpdateTime(1500)
		require.Equal(t, 1, *n)
		et.UpdateTime(1600)
		et.UpdateTime(1999)
		require.Equal(t, 1, *n)
		et.UpdateTime(2000)
		require.Equal(t, 2, *n)
	})

	t.Run("initial duration before first reset", func(t *testing.T) {
		et, n := newTimers(500)

		et.UpdateTime(100)
		require.Equal(t, 1, *n)
		et.UpdateTime(599)
		require.Equal(t, 1, *n)
		et.UpdateTime(600)
		require.Equal(t, 2, *n)
	})

	t.Run("reset reschedules", func(t *testing.T) {
		et, n := newTimers(0)
		et.Reset(1000, 500)

		et.UpdateTime(1500)
		require.Equal(t, 1, *n)
		et.Reset(1700, 500)
		et.UpdateTime(2000)
		require.Equal(t, 1, *n)
		et.UpdateTime(2200)
		require.Equal(t, 2, *n)
	})
}

What I ran

  • On the head (b6f06e6): go test -count=1 ./pkg/timers/ passes.
  • On the base (79a04b4), with the InitialEpochDuration line removed because that field doesn't exist there: keeps ticking without reset and initial duration before first reset both fail with expected: 2, actual: 1. So the test covers the re-arm this PR adds.

The test doesn't cover a zero duration, because the head still fires on every block in that case (see my earlier review). Once you decide what zero should do, add a subtest for it.

To have me commit it instead, open your own branch and push it there. My runs can only push to branches I opened.


Generated by hal0x9000, covering b6f06e6.

If anything blocks epoch tick, do not simply try it once and wait for the next
chain "NewEpoch" notification, instead, try to predict the next tick time and
never stop trying. An epoch stuck is a critical error in the system, IRs must
try their best.

Signed-off-by: Pavel Karpy <carpawell@nspcc.io>
@carpawell
carpawell force-pushed the fix/ir-do-not-try-epoch-tick-after-stuck branch from b6f06e6 to 6c5113b Compare November 17, 2026 14:35
@carpawell
carpawell merged commit baf4d43 into master Nov 17, 2026
21 of 22 checks passed
@carpawell
carpawell deleted the fix/ir-do-not-try-epoch-tick-after-stuck branch November 17, 2026 14:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants