Repository navigation
ir/timers: never stop epoch ticking - #4274
Conversation
bd9565d to
eed742b
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #4274 +/- ##
==========================================
+ Coverage 33.24% 33.28% +0.04%
==========================================
Files 673 673
Lines 41881 41888 +7
==========================================
+ Hits 13924 13944 +20
+ Misses 27957 27944 -13 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
roman-khimov
left a comment
There was a problem hiding this comment.
Why not resetting in error handler? This seems dangerous by default (some other timers can have problems with this).
There was a problem hiding this comment.
🟡 Changes recommended
The timer can submit the epoch transaction on every block when no valid duration is available, so the active moderate-severity issue must be fixed.
1 open issue
What changed in this PR
Updates epoch timers to keep retrying epoch ticks using the configured epoch duration when NewEpoch notifications are delayed or unavailable.
Changes:
- Replaces one-shot completion state with persistent epoch-duration scheduling.
- Reschedules the next epoch tick after each attempted handler execution.
- Stores the epoch duration during timer resets.
- A moderate issue remains when
epochDuris zero: handlers may be invoked once per block instead of preserving one-shot behavior until a valid duration is available.
| File | Description |
|---|---|
pkg/timers/timer.go |
Implements recurring epoch-tick scheduling and reset duration tracking. |
Review effort: Lite
|
This looks like the timer commit from #4272, with both issues I pointed out there fixed. |
What is "error handler"? Current IR does not wait for new epoch tx inclusion, it just sends notary request, do you want it to wait for NewEpoch notification (or vub expiration)?
There are no other timers, only the new epoch will have this exclusive opportunity. If everything is ok, it will be reset likely almost immediately by the NewEpoch event. But if smth is wrong, the next tick will, at least, happen. |
eed742b to
b3ceaa5
Compare
|
@roman-khimov, treat this PR like |
b3ceaa5 to
b6f06e6
Compare
|
/hal review |
There was a problem hiding this comment.
The re-arm works for a non-zero epoch duration and does not change storage-node behaviour. Two things need fixing before merge: a zero epoch duration now fires the epoch tick on every block, and InitialEpochDuration gets overwritten before anything reads it. There is also still no test.
On the concern about other timers: the only other timers.NewTimers user is the SN size-load reporter (cmd/neofs-node/container.go:155-172). It registers only DeltaTicks, and their done handling is unchanged, so the new eHandlers re-arm loop runs over an empty slice there. The IR epoch tick is the only handler affected.
What I checked
UpdateTime/Resetin full, the IR wiring (New→initTimers,Start→initConfigFromBlockchain→resetEpochTimer→Reset, the header handler atinnerring.go:203-208),processNewEpoch/processNewEpochTick, and the SN caller.- I ran a throwaway test against a copy of
pkg/timers(nothing committed):Reset(1000, 500)followed byUpdateTimeat 1400, 1500, 1600, 1999 and 2000.- Head: two fires (1500 and 2000).
- Base: one fire.
Reset(1000, 0)followed by three blocks: three fires on head, one on base.
Findings (details inline)
- Zero duration: handlers fire on every block. Before this PR they fired once.
InitialEpochDurationis never read, so it, the extra RPC and the newNew()error path can go.- No test for the new behaviour.
- CHANGELOG wording.
Not introduced here, for the record. Suppose processNewEpoch sets the counter (process_epoch.go:30) and then ResetEpochTimer fails (:39-41), and this IR had not fired yet. The stale nextTickAt then fires NewEpoch(counter+1) on the next block. That already happened before this PR. The PR only lets it repeat once per epoch afterwards instead of stopping.
Not determined: whether the netmap contract rejects a zero EpochDuration (it is in neofs-contracts, not checked). pkg/morph/client/netmap/config.go:47 passes the value through unvalidated.
Generated review by hal0x9000, covering b6f06e6.
| // prevent epoch stuck: even if smth is wrong with the epochs, | ||
| // the next tick will be done in time; if everything is ok, | ||
| // the following Reset call will tune the next tick more precisely | ||
| if et.epochDur != 0 { |
There was a problem hiding this comment.
With epochDur == 0, nextTickAt is never advanced past curr, so nextTickAt <= curr holds on every later block. Each block then calls HandleNewEpochTick → netmapClient.NewEpoch(...). Before this PR, done limited that to one call. Reproduced: after Reset(1000, 0), three UpdateTime calls fire three times on head and once on base.
A zero duration from the contract is not hypothetical in this tree. cmd/neofs-node/netmap.go:75-82 (b0b1270, "Zero value is not acceptable ... panic was observed at least once") falls back to a default for exactly this case.
Suggestion: keep one-shot semantics for a zero duration. For example, push nextTickAt out of reach when epochDur == 0, or reject or replace zero in resetEpochTimer, where the duration is read. Either way, a zero value can't turn into a per-block transaction storm.
There was a problem hiding this comment.
a zero value can't turn into a per-block transaction storm
why not? correct way to me, contract should never return 0 epoch time
| epochH, stopEpochHandlerFunc := util.SingleAsyncExecutingInstance(server.netmapProcessor.HandleNewEpochTick) | ||
| server.closers = append(server.closers, func() error { stopEpochHandlerFunc(); return nil }) | ||
|
|
||
| epochDuration, err := server.netmapClient.EpochDuration() |
There was a problem hiding this comment.
This value can't be observed. Server.Start calls initConfigFromBlockchain (innerring.go:159) → resetEpochTimer(0) (:1022) → epochTimers.Reset, which sets epochDur (timer.go:85). Only after that is the header handler that calls UpdateTime registered (:203-208). If that reset fails, Start returns before the handler exists. So Reset always overwrites InitialEpochDuration before the first UpdateTime.
Dropping EpochTicks.InitialEpochDuration would also remove:
- this extra RPC
- the new error return of
initTimersand the failure path inNew() - the
timeimport - the second seconds→ms conversion style (
uint64(time.Second/time.Millisecond)here vsconst msInS = 1000at:264)
timer.go then only needs epochDur = dur in Reset.
There was a problem hiding this comment.
Dropping EpochTicks.InitialEpochDuration would also remove:
...
None of these is a problem
So Reset always overwrites InitialEpochDuration before the first UpdateTime
it is safer to have some prediction of the next ticks before actual reset is called later
| return | ||
| } | ||
|
|
||
| if et.nextTickAt <= curr { |
There was a problem hiding this comment.
pkg/timers has no tests, and this changes when the epoch tick fires. The profile asks for a bug fix to come with a test that fails without it. A small require table test over Reset/UpdateTime that counts Tick calls needs no dependencies. It should cover:
- re-fire after
epochDurwithoutReset(fails on base: 1 fire instead of 2) Resetmoving the next tickdur == 0- delta ticks unaffected
|
Long story short:
|
|
/hal can you add a unit test for it in this PR? |
|
I can't add it in this PR. A package timers
import (
"testing"
"github.com/stretchr/testify/require"
)
func TestEpochTimers_UpdateTime(t *testing.T) {
newTimers := func(initDur uint64) (*EpochTimers, *int) {
var n int
return NewTimers(EpochTicks{
InitialEpochDuration: initDur,
NewEpochTicks: []Tick{func() { n++ }},
}), &n
}
t.Run("keeps ticking without reset", func(t *testing.T) {
et, n := newTimers(0)
et.Reset(1000, 500)
et.UpdateTime(1400)
require.Equal(t, 0, *n)
et.UpdateTime(1500)
require.Equal(t, 1, *n)
et.UpdateTime(1600)
et.UpdateTime(1999)
require.Equal(t, 1, *n)
et.UpdateTime(2000)
require.Equal(t, 2, *n)
})
t.Run("initial duration before first reset", func(t *testing.T) {
et, n := newTimers(500)
et.UpdateTime(100)
require.Equal(t, 1, *n)
et.UpdateTime(599)
require.Equal(t, 1, *n)
et.UpdateTime(600)
require.Equal(t, 2, *n)
})
t.Run("reset reschedules", func(t *testing.T) {
et, n := newTimers(0)
et.Reset(1000, 500)
et.UpdateTime(1500)
require.Equal(t, 1, *n)
et.Reset(1700, 500)
et.UpdateTime(2000)
require.Equal(t, 1, *n)
et.UpdateTime(2200)
require.Equal(t, 2, *n)
})
}What I ran
The test doesn't cover a zero duration, because the head still fires on every block in that case (see my earlier review). Once you decide what zero should do, add a subtest for it. To have me commit it instead, open your own branch and push it there. My runs can only push to branches I opened. Generated by hal0x9000, covering b6f06e6. |
If anything blocks epoch tick, do not simply try it once and wait for the next chain "NewEpoch" notification, instead, try to predict the next tick time and never stop trying. An epoch stuck is a critical error in the system, IRs must try their best. Signed-off-by: Pavel Karpy <carpawell@nspcc.io>
b6f06e6 to
6c5113b
Compare

If anything blocks epoch tick, do not simply try it once and wait for the next chain "NewEpoch" notification, instead, try to predict the next tick time and never stop trying. An epoch stuck is a critical error in the system, IRs must try their best.