From a162d7eefac06e790090f9612b77bf87c5b00174 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Wed, 23 Sep 2026 07:35:47 -0600 Subject: [PATCH 01/12] docs: explain superseded subscription event policy --- reference/resources/resource-api.md | 53 ++++++++++++++++++++++------- 1 file changed, 40 insertions(+), 13 deletions(-) diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index 36e7a8449..f975a9f8e 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -238,15 +238,42 @@ Called for MQTT subscribe commands. Returns a `Subscription` — an `AsyncIterab All properties are optional: -| Property | Description | -| -------------------- | --------------------------------------------------------------------------------------------------------- | -| `includeDescendants` | Include all updates with an id prefixed by the subscribed id (e.g. `sub/*`) | -| `startTime` | Start from a past time (catch-up of historical messages). Cannot be used with `previousCount`. | -| `databaseGeneration` | Resume from `startTime` as a checked position. See [Resuming from a position](#resuming-from-a-position). | -| `previousCount` | Return the last N updates/messages. Cannot be used with `startTime`. | -| `omitCurrent` | Do not send the current/retained record as the first update. | -| `rowFilter` | Synchronous JavaScript predicate applied to authoritative row values. | -| `eventFilter` | Synchronous JavaScript predicate for events that may not carry an authoritative row. | +| Property | Description | +| -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `includeDescendants` | Include all updates with an id prefixed by the subscribed id (e.g. `sub/*`) | +| `startTime` | Resume after a local audit-log timestamp, such as a previously received event's `localTime`. Cannot be used with `previousCount`. | +| `databaseGeneration` | Resume from `startTime` as a checked position. See [Resuming from a position](#resuming-from-a-position). | +| `previousCount` | Return up to the last N eligible updates/messages. Cannot be used with `startTime`. | +| `includeSuperseded` | Include record mutations superseded by a later version. Defaults to `false`, or `true` when `rawEvents` is enabled. An explicit value overrides that default. Added in v5.3.0. | +| `rawEvents` | Return raw audit events rather than reconstructing full records. Defaults to `false`. | +| `omitCurrent` | Do not send the current/retained record as the first update. | +| `rowFilter` | Synchronous JavaScript predicate applied to authoritative row values. | +| `eventFilter` | Synchronous JavaScript predicate for events that may not carry an authoritative row. | + +#### Superseded record updates + + + +By default, table subscriptions skip record mutations whose version differs from the current stored record's version. This applies both to catch-up with `startTime` or `previousCount` and to live delivery. If a record was updated at versions 2, 3, and 4 before catch-up begins, and version 4 is still current, the subscription delivers only version 4. Current deletion tombstones are eligible for delivery; older updates or deletions superseded by a later write are skipped. Published messages are independent events and are not suppressed by this version check. + +`startTime` is a local audit-log cursor, not a record version. Use an event's `localTime` to resume after that event. Catch-up reads retained audit history; `previousCount` may return fewer events because of superseded versions, filters, or the bounded history scan. + +Previously, catch-up could include superseded mutations even though live delivery suppressed them. Consumers that need historical transitions can opt in: + +```javascript +const subscription = await Product.subscribe({ + startTime: lastLocalTime, + includeSuperseded: true, +}); + +for await (const event of subscription) { + processEvent(event); +} +``` + +With `includeSuperseded: true`, record values are reconstructed for each historical version, unless `rawEvents` is enabled. This option applies to live delivery too. Delivery still depends on retained audit history and subscription filters; it does not provide an exactly-once guarantee. + +`rawEvents: true` preserves its existing default of including superseded events. Set `includeSuperseded: false` explicitly to apply the same version check while retaining raw event payloads. #### Resuming from a position @@ -260,10 +287,10 @@ As of v5.4.0, a subscription on a RocksDB database can resume from a position th Harper refuses a position it cannot replay completely: -| Error | Status | Code | When | -| -------------------------------- | ------ | ----------------------------- | -------------------------------------------------------------------------------------------------------- | -| `DatabaseGenerationChangedError` | 409 | `DATABASE_GENERATION_CHANGED` | The database was replaced by a restored or copied state after the position was recorded, or it is LMDB. | -| `ResumeHistoryUnavailableError` | 410 | `RESUME_HISTORY_UNAVAILABLE` | Audit retention pruned history after the position, or one record has more than 10,000 versions after it. | +| Error | Status | Code | When | +| -------------------------------- | ------ | ----------------------------- | ---------------------------------------------------------------------------------------------------------- | +| `DatabaseGenerationChangedError` | 409 | `DATABASE_GENERATION_CHANGED` | The database was replaced by a restored or copied state after the position was recorded, or it is LMDB. | +| `ResumeHistoryUnavailableError` | 410 | `RESUME_HISTORY_UNAVAILABLE` | Audit retention pruned history after the position, or one record has more than 10,000 versions after it. | A subscription to one record is checked against that record's own history: it resumes as long as the record's retained version history still reaches back to the position, even when retention has pruned other history past it. A record first written after the position is checked against retention like a table subscription. From c8c1a5a92302bbbd6f497a1c7bcd1b1f8ce896fa Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Wed, 23 Sep 2026 07:43:04 -0600 Subject: [PATCH 02/12] docs: call out subscription replay migration and MQTT impact --- reference/mqtt/overview.md | 8 ++++++-- reference/resources/resource-api.md | 26 +++++++++++++------------- release-notes/v5-lincoln/5.3.md | 5 +++++ 3 files changed, 24 insertions(+), 15 deletions(-) diff --git a/reference/mqtt/overview.md b/reference/mqtt/overview.md index c09157fbb..3a6b1f527 100644 --- a/reference/mqtt/overview.md +++ b/reference/mqtt/overview.md @@ -55,7 +55,7 @@ Harper supports multi-level topics for both publishing and subscribing: -A durable session retains a client's subscription list and any unacknowledged messages across disconnects. When the client reconnects with the same client ID, it picks up from where it left off — including any messages published while it was offline. +A durable session retains a client's subscription list and any unacknowledged messages across disconnects. When the client reconnects with the same client ID, Harper catches up subscribed topics from retained audit history, picking up from where it left off. Durable sessions in Harper are persisted as records in the `hdb_durable_session` system table, indexed by client ID. The session record holds the list of subscriptions (topic + QoS) and, per topic, the position in the transaction log from which a reconnect catches up. Because durable sessions are records rather than in-memory state, an abandoned session sits idle with no runtime cost until the client reconnects or the record is deleted. @@ -73,7 +73,7 @@ mqtt.connect('mqtts://harper.example.com:8883', { }); ``` -**Catch-up on reconnect** — When the client reconnects, Harper replays missed messages on subscribed topics by reading the audit log. For this to work, audit logging must be enabled on the tables backing the subscribed topics. See [Transaction Logging](../database/transaction.md) and [`logging.auditLog`](../logging/configuration.md#loggingauditlog). Catch-up applies to QoS 1 and 2 subscriptions. A QoS 0 subscription is kept with the session and resumes live, without the messages published while the client was away. +**Catch-up on reconnect** — When the client reconnects, Harper replays missed messages on subscribed topics by reading retained audit history. For this to work, audit logging must be enabled on the tables backing the subscribed topics. See [Transaction Logging](../database/transaction.md) and [`logging.auditLog`](../logging/configuration.md#loggingauditlog). Catch-up applies to QoS 1 and 2 subscriptions. A QoS 0 subscription is kept with the session and resumes live, without the messages published while the client was away. **When a session can no longer catch up** — As of v5.4.0, Harper checks each subscription's saved position against the database's history before replaying from it. When audit retention has since removed history the replay would need, Harper resets the session instead of replaying only part of what was missed: @@ -85,6 +85,10 @@ A position saved before the database was restored or copied, or saved on another Harper keeps a quiet topic's position current while the client is connected, and again when it disconnects, so a subscription that receives nothing is not reset merely because other tables' retention advanced. A position only moves past a message once that message, and every other message from the same transaction, has been acknowledged; an out-of-order acknowledgement can mean some messages are delivered again after a reconnect. A client that sends SUBSCRIBE again for a topic its session already holds, as many clients do on every connect, continues from the saved position, so resubscribing loses nothing. A new connection with the same client ID on the same Harper thread takes the session over, and the older connection is closed (MQTT v5 reason code `0x8E`); a clean-start connect closes it the same way. These checks apply to RocksDB databases; LMDB sessions catch up as before. + + +Retained publications (`retain: true`) write records, so reconnect catch-up now skips superseded record versions, matching live delivery. If a topic was updated three times while a client was offline, catch-up delivers its current state rather than each intermediate state. Non-retained publications remain independent messages and are not suppressed by this version check. Catch-up is limited by audit-log retention. See [Superseded record updates](../resources/resource-api.md#superseded-record-updates). + **Session expiry** — In MQTT v5, the `sessionExpiryInterval` property on `CONNECT` controls how long the session is retained after the client disconnects. With `sessionExpiryInterval: 0` (or a clean session connect), Harper deletes the session record at disconnect. Connecting with the same client ID and `clean: true` also explicitly deletes any existing durable session. **Distributed durable sessions** — Harper does not currently maintain a single logical durable session across multiple cluster nodes; a durable session record lives on the node where it was created. Clients that may reconnect to different nodes (e.g., behind a load balancer with round-robin DNS) should use sticky routing or terminate on a fixed node. diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index f975a9f8e..e1212d0cd 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -238,17 +238,17 @@ Called for MQTT subscribe commands. Returns a `Subscription` — an `AsyncIterab All properties are optional: -| Property | Description | -| -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `includeDescendants` | Include all updates with an id prefixed by the subscribed id (e.g. `sub/*`) | -| `startTime` | Resume after a local audit-log timestamp, such as a previously received event's `localTime`. Cannot be used with `previousCount`. | -| `databaseGeneration` | Resume from `startTime` as a checked position. See [Resuming from a position](#resuming-from-a-position). | -| `previousCount` | Return up to the last N eligible updates/messages. Cannot be used with `startTime`. | -| `includeSuperseded` | Include record mutations superseded by a later version. Defaults to `false`, or `true` when `rawEvents` is enabled. An explicit value overrides that default. Added in v5.3.0. | -| `rawEvents` | Return raw audit events rather than reconstructing full records. Defaults to `false`. | -| `omitCurrent` | Do not send the current/retained record as the first update. | -| `rowFilter` | Synchronous JavaScript predicate applied to authoritative row values. | -| `eventFilter` | Synchronous JavaScript predicate for events that may not carry an authoritative row. | +| Property | Description | +| -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `includeDescendants` | Include all updates with an id prefixed by the subscribed id (e.g. `sub/*`) | +| `startTime` | Resume after a local audit-log timestamp, such as a previously received event's `localTime`. Cannot be used with `previousCount`. | +| `databaseGeneration` | Resume from `startTime` as a checked position. See [Resuming from a position](#resuming-from-a-position). | +| `previousCount` | Return up to the last N eligible updates/messages. Cannot be used with `startTime`. | +| `includeSuperseded` | Include record mutations superseded by a later version. Defaults to `false`, or `true` when `rawEvents` is enabled. An explicit value overrides that default. | +| `rawEvents` | Return raw audit events rather than reconstructing full records. Defaults to `false`. | +| `omitCurrent` | Do not send the current/retained record as the first update. | +| `rowFilter` | Synchronous JavaScript predicate applied to authoritative row values. | +| `eventFilter` | Synchronous JavaScript predicate for events that may not carry an authoritative row. | #### Superseded record updates @@ -258,7 +258,7 @@ By default, table subscriptions skip record mutations whose version differs from `startTime` is a local audit-log cursor, not a record version. Use an event's `localTime` to resume after that event. Catch-up reads retained audit history; `previousCount` may return fewer events because of superseded versions, filters, or the bounded history scan. -Previously, catch-up could include superseded mutations even though live delivery suppressed them. Consumers that need historical transitions can opt in: +Before v5.3.0, non-raw catch-up could include superseded mutations even though non-raw live delivery suppressed them. Consumers that need historical transitions can opt in: ```javascript const subscription = await Product.subscribe({ @@ -267,7 +267,7 @@ const subscription = await Product.subscribe({ }); for await (const event of subscription) { - processEvent(event); + await processEvent(event); } ``` diff --git a/release-notes/v5-lincoln/5.3.md b/release-notes/v5-lincoln/5.3.md index d7186c99f..d5f743af2 100644 --- a/release-notes/v5-lincoln/5.3.md +++ b/release-notes/v5-lincoln/5.3.md @@ -77,6 +77,11 @@ Every replication retry now follows one jittered backoff schedule. Subscription ### Nodes and Clones `update_node` now works; the documented operation was never registered. A request that changes only `revoked_certificates` or `shard` updates the node record locally without resetting replication ([harper-pro#916](https://github.com/HarperFast/harper-pro/pull/916)). See [Update Node](/reference/v5/replication/clustering#update-node). A clone's wait for the initial sync is now bounded by the data size the leader reports and resumes across restarts, so a large clone no longer times out on a fixed budget or starts over ([harper-pro#661](https://github.com/HarperFast/harper-pro/pull/661)). +## Subscription Catch-up + +Table subscriptions now skip superseded record updates during catch-up with `startTime` or `previousCount`, matching live delivery. A record updated at versions 2, 3, and 4 is delivered only at version 4 if that version is still current. **Consumers that relied on replaying intermediate record versions must set `includeSuperseded: true`.** Raw-event subscriptions retain their existing default; `rawEvents: true` includes superseded events unless `includeSuperseded: false` is explicit. + +MQTT durable-session reconnects use the same default: retained publications update records, so catch-up delivers current record state. Non-retained published messages remain independent and are not suppressed by this version check. See [Subscription options and migration](/reference/v5/resources/resource-api#superseded-record-updates) and [MQTT durable sessions](/reference/v5/mqtt/overview#durable-sessions). ## Node Identity From 0c5a7dc5c6c70ecb133de081e91515153d7ab440 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Wed, 23 Sep 2026 07:43:45 -0600 Subject: [PATCH 03/12] docs: clarify subscription cursor and history count bounds --- reference/resources/resource-api.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index e1212d0cd..7f4587791 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -241,9 +241,9 @@ All properties are optional: | Property | Description | | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `includeDescendants` | Include all updates with an id prefixed by the subscribed id (e.g. `sub/*`) | -| `startTime` | Resume after a local audit-log timestamp, such as a previously received event's `localTime`. Cannot be used with `previousCount`. | +| `startTime` | Resume after a local audit-log cursor; all events at that cursor are excluded. Cannot be used with `previousCount`. | | `databaseGeneration` | Resume from `startTime` as a checked position. See [Resuming from a position](#resuming-from-a-position). | -| `previousCount` | Return up to the last N eligible updates/messages. Cannot be used with `startTime`. | +| `previousCount` | Return up to the last N accepted updates/messages, capped at 1,000. Cannot be used with `startTime`. | | `includeSuperseded` | Include record mutations superseded by a later version. Defaults to `false`, or `true` when `rawEvents` is enabled. An explicit value overrides that default. | | `rawEvents` | Return raw audit events rather than reconstructing full records. Defaults to `false`. | | `omitCurrent` | Do not send the current/retained record as the first update. | @@ -256,7 +256,9 @@ All properties are optional: By default, table subscriptions skip record mutations whose version differs from the current stored record's version. This applies both to catch-up with `startTime` or `previousCount` and to live delivery. If a record was updated at versions 2, 3, and 4 before catch-up begins, and version 4 is still current, the subscription delivers only version 4. Current deletion tombstones are eligible for delivery; older updates or deletions superseded by a later write are skipped. Published messages are independent events and are not suppressed by this version check. -`startTime` is a local audit-log cursor, not a record version. Use an event's `localTime` to resume after that event. Catch-up reads retained audit history; `previousCount` may return fewer events because of superseded versions, filters, or the bounded history scan. +`startTime` is a local audit-log cursor, not a record version. Audit events expose this cursor as `localTime`, which multiple events in a transaction can share. Resuming with `startTime` excludes all events at that cursor, so callers must account for shared cursors before checkpointing. + +`previousCount` counts accepted events: superseded or filtered events do not consume the requested count. The request is capped at 1,000 events, and scanning stops after 10,000 in-scope audit entries or the end of retained history, so fewer events may be returned. Before v5.3.0, non-raw catch-up could include superseded mutations even though non-raw live delivery suppressed them. Consumers that need historical transitions can opt in: From 18c34335dc232bc22d199583215bf19cb9918994 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Wed, 23 Sep 2026 07:46:04 -0600 Subject: [PATCH 04/12] docs: clarify subscriptions with missing primary records --- reference/resources/resource-api.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index 7f4587791..4ad626cd1 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -254,7 +254,7 @@ All properties are optional: -By default, table subscriptions skip record mutations whose version differs from the current stored record's version. This applies both to catch-up with `startTime` or `previousCount` and to live delivery. If a record was updated at versions 2, 3, and 4 before catch-up begins, and version 4 is still current, the subscription delivers only version 4. Current deletion tombstones are eligible for delivery; older updates or deletions superseded by a later write are skipped. Published messages are independent events and are not suppressed by this version check. +By default, table subscriptions skip record mutations whose version differs from the current stored record's version. This applies both to catch-up with `startTime` or `previousCount` and to live delivery. If a record was updated at versions 2, 3, and 4 before catch-up begins, and version 4 is still current, the subscription delivers only version 4. Current deletion tombstones are eligible for delivery; older updates or deletions superseded by a later write are skipped. When no stored record exists, only actual delete audit events are delivered; older puts, patches, invalidations, and relocations are skipped, so eviction is not reported as deletion. Published messages are independent events and are not suppressed by this version check. `startTime` is a local audit-log cursor, not a record version. Audit events expose this cursor as `localTime`, which multiple events in a transaction can share. Resuming with `startTime` excludes all events at that cursor, so callers must account for shared cursors before checkpointing. @@ -275,7 +275,7 @@ for await (const event of subscription) { With `includeSuperseded: true`, record values are reconstructed for each historical version, unless `rawEvents` is enabled. This option applies to live delivery too. Delivery still depends on retained audit history and subscription filters; it does not provide an exactly-once guarantee. -`rawEvents: true` preserves its existing default of including superseded events. Set `includeSuperseded: false` explicitly to apply the same version check while retaining raw event payloads. +`rawEvents: true` preserves its existing default of including superseded events. Set `includeSuperseded: false` explicitly to apply the same version and missing-record checks while retaining raw event payloads. #### Resuming from a position From 6fa01524165224d6b63609827f6c31280a3e8deb Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Wed, 23 Sep 2026 07:49:11 -0600 Subject: [PATCH 05/12] docs: require current entries for subscription mutations --- reference/resources/resource-api.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index 4ad626cd1..e00f733ef 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -254,7 +254,7 @@ All properties are optional: -By default, table subscriptions skip record mutations whose version differs from the current stored record's version. This applies both to catch-up with `startTime` or `previousCount` and to live delivery. If a record was updated at versions 2, 3, and 4 before catch-up begins, and version 4 is still current, the subscription delivers only version 4. Current deletion tombstones are eligible for delivery; older updates or deletions superseded by a later write are skipped. When no stored record exists, only actual delete audit events are delivered; older puts, patches, invalidations, and relocations are skipped, so eviction is not reported as deletion. Published messages are independent events and are not suppressed by this version check. +By default, table subscriptions skip record mutations whose version differs from the current stored record's version. This applies both to catch-up with `startTime` or `previousCount` and to live delivery. If a record was updated at versions 2, 3, and 4 before catch-up begins, and version 4 is still current, the subscription delivers only version 4. Deletion events require a retained tombstone with a matching version; older updates or deletions superseded by a later write are skipped. If the record or its tombstone has been removed, record mutations are skipped because no current version remains to compare. Published messages are independent events and are not suppressed by this version check. `startTime` is a local audit-log cursor, not a record version. Audit events expose this cursor as `localTime`, which multiple events in a transaction can share. Resuming with `startTime` excludes all events at that cursor, so callers must account for shared cursors before checkpointing. From 0772c9ac17080cfe804f78a3e66634fc03541133 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Wed, 23 Sep 2026 07:58:07 -0600 Subject: [PATCH 06/12] docs: preserve durable MQTT publication delivery --- reference/mqtt/overview.md | 2 +- reference/resources/resource-api.md | 2 +- release-notes/v5-lincoln/5.3.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/reference/mqtt/overview.md b/reference/mqtt/overview.md index 3a6b1f527..4845156a5 100644 --- a/reference/mqtt/overview.md +++ b/reference/mqtt/overview.md @@ -87,7 +87,7 @@ Harper keeps a quiet topic's position current while the client is connected, and -Retained publications (`retain: true`) write records, so reconnect catch-up now skips superseded record versions, matching live delivery. If a topic was updated three times while a client was offline, catch-up delivers its current state rather than each intermediate state. Non-retained publications remain independent messages and are not suppressed by this version check. Catch-up is limited by audit-log retention. See [Superseded record updates](../resources/resource-api.md#superseded-record-updates). +Durable subscriptions with QoS 1 or 2 explicitly include superseded record versions during both live delivery and reconnect catch-up. This preserves delivery of intermediate retained publications (`retain: true`), even though ordinary table subscriptions now default to current-state delivery. Non-retained publications remain independent messages. Catch-up is limited by audit-log retention. See [Superseded record updates](../resources/resource-api.md#superseded-record-updates). **Session expiry** — In MQTT v5, the `sessionExpiryInterval` property on `CONNECT` controls how long the session is retained after the client disconnects. With `sessionExpiryInterval: 0` (or a clean session connect), Harper deletes the session record at disconnect. Connecting with the same client ID and `clean: true` also explicitly deletes any existing durable session. diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index e00f733ef..26185dcf6 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -254,7 +254,7 @@ All properties are optional: -By default, table subscriptions skip record mutations whose version differs from the current stored record's version. This applies both to catch-up with `startTime` or `previousCount` and to live delivery. If a record was updated at versions 2, 3, and 4 before catch-up begins, and version 4 is still current, the subscription delivers only version 4. Deletion events require a retained tombstone with a matching version; older updates or deletions superseded by a later write are skipped. If the record or its tombstone has been removed, record mutations are skipped because no current version remains to compare. Published messages are independent events and are not suppressed by this version check. +By default, table subscriptions skip record mutations whose version differs from the current stored record's version. This applies both to catch-up with `startTime` or `previousCount` and to live delivery. If a record was updated at versions 2, 3, and 4 before catch-up begins, and version 4 is still current, the subscription delivers only version 4. Deletion events require a retained tombstone with a matching version; older updates or deletions superseded by a later write are skipped. If the record or its tombstone has been removed, record mutations are skipped because no current version remains to compare. Published messages are independent events and are not suppressed by this version check. The comparison uses the stored primary entry's metadata version, which publishing also advances. A publication can therefore cause a preceding record mutation to be skipped; this is the existing live-delivery rule, not a separate timestamp tracking only record mutations. `startTime` is a local audit-log cursor, not a record version. Audit events expose this cursor as `localTime`, which multiple events in a transaction can share. Resuming with `startTime` excludes all events at that cursor, so callers must account for shared cursors before checkpointing. diff --git a/release-notes/v5-lincoln/5.3.md b/release-notes/v5-lincoln/5.3.md index d5f743af2..48e84ce3d 100644 --- a/release-notes/v5-lincoln/5.3.md +++ b/release-notes/v5-lincoln/5.3.md @@ -81,7 +81,7 @@ Every replication retry now follows one jittered backoff schedule. Subscription Table subscriptions now skip superseded record updates during catch-up with `startTime` or `previousCount`, matching live delivery. A record updated at versions 2, 3, and 4 is delivered only at version 4 if that version is still current. **Consumers that relied on replaying intermediate record versions must set `includeSuperseded: true`.** Raw-event subscriptions retain their existing default; `rawEvents: true` includes superseded events unless `includeSuperseded: false` is explicit. -MQTT durable-session reconnects use the same default: retained publications update records, so catch-up delivers current record state. Non-retained published messages remain independent and are not suppressed by this version check. See [Subscription options and migration](/reference/v5/resources/resource-api#superseded-record-updates) and [MQTT durable sessions](/reference/v5/mqtt/overview#durable-sessions). +MQTT durable subscriptions with QoS 1 or 2 explicitly include superseded versions in live delivery and reconnect catch-up, preserving intermediate retained publications. Non-retained published messages remain independent and are not suppressed by the table version check. See [Subscription options and migration](/reference/v5/resources/resource-api#superseded-record-updates) and [MQTT durable sessions](/reference/v5/mqtt/overview#durable-sessions). ## Node Identity From 6d7c931a3112f01c88f4b235ac5d6d5e8e827f3f Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Wed, 23 Sep 2026 08:04:12 -0600 Subject: [PATCH 07/12] docs: clarify subscription migration and durable ordering --- reference/mqtt/overview.md | 2 +- release-notes/v5-lincoln/5.3.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/reference/mqtt/overview.md b/reference/mqtt/overview.md index 4845156a5..7eec70b80 100644 --- a/reference/mqtt/overview.md +++ b/reference/mqtt/overview.md @@ -111,7 +111,7 @@ Harper is designed for distributed, low-latency message delivery. Messages are d In a distributed cluster, messages may arrive out of order due to network topology. The behavior depends on whether the message is retained or non-retained: -- **Retained messages** (published with `retain: true`, or written via PUT/upsert) maintain eventual consistency across the cluster. Harper keeps the message with the latest timestamp as the winning record state. An out-of-order earlier message will not be re-delivered to clients; the cluster converges to the most recent state. +- **Retained messages** (published with `retain: true`, or written via PUT/upsert) maintain eventual consistency across the cluster. Harper keeps the message with the latest timestamp as the winning record state. Ordinary subscriptions skip superseded updates. Durable QoS 1/2 subscriptions include superseded versions, so their delivered history can include older updates even though the stored record converges to the most recent state. - **Non-retained messages** are always delivered to local subscribers when received, even if they arrive out of order. Every message is delivered, prioritizing completeness over strict ordering. **Non-retained messages** are suited for applications like chat where every message must be delivered. **Retained messages** are suited for sensor readings or state updates where only the latest value matters. diff --git a/release-notes/v5-lincoln/5.3.md b/release-notes/v5-lincoln/5.3.md index 48e84ce3d..84cb250ff 100644 --- a/release-notes/v5-lincoln/5.3.md +++ b/release-notes/v5-lincoln/5.3.md @@ -79,7 +79,7 @@ Every replication retry now follows one jittered backoff schedule. Subscription `update_node` now works; the documented operation was never registered. A request that changes only `revoked_certificates` or `shard` updates the node record locally without resetting replication ([harper-pro#916](https://github.com/HarperFast/harper-pro/pull/916)). See [Update Node](/reference/v5/replication/clustering#update-node). A clone's wait for the initial sync is now bounded by the data size the leader reports and resumes across restarts, so a large clone no longer times out on a fixed budget or starts over ([harper-pro#661](https://github.com/HarperFast/harper-pro/pull/661)). ## Subscription Catch-up -Table subscriptions now skip superseded record updates during catch-up with `startTime` or `previousCount`, matching live delivery. A record updated at versions 2, 3, and 4 is delivered only at version 4 if that version is still current. **Consumers that relied on replaying intermediate record versions must set `includeSuperseded: true`.** Raw-event subscriptions retain their existing default; `rawEvents: true` includes superseded events unless `includeSuperseded: false` is explicit. +Table subscriptions now skip superseded record updates during catch-up with `startTime` or `previousCount`, matching live delivery. A record updated at versions 2, 3, and 4 is delivered only at version 4 if that version is still current. The comparison uses the stored primary metadata version: a later publication can suppress a preceding mutation, and removing a record or tombstone makes its mutations ineligible. **JavaScript consumers that need retained mutation history, including these cases, must set `includeSuperseded: true`.** Raw-event subscriptions retain their existing default; `rawEvents: true` includes superseded events unless `includeSuperseded: false` is explicit. MQTT durable subscriptions with QoS 1 or 2 explicitly include superseded versions in live delivery and reconnect catch-up, preserving intermediate retained publications. Non-retained published messages remain independent and are not suppressed by the table version check. See [Subscription options and migration](/reference/v5/resources/resource-api#superseded-record-updates) and [MQTT durable sessions](/reference/v5/mqtt/overview#durable-sessions). From 6a313d081a04b9277631bb9a0cc0777b8347d771 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Wed, 30 Sep 2026 07:46:12 -0600 Subject: [PATCH 08/12] docs(release-notes): add blank line before Subscription Catch-up heading Rebase conflict resolution merged the section without the blank line prettier requires before a heading. Co-Authored-By: Claude Sonnet 5 Dispatch-Task: pr-maint-92e4b54bf5b8178a01b68ecdd48dc2b0 --- release-notes/v5-lincoln/5.3.md | 1 + 1 file changed, 1 insertion(+) diff --git a/release-notes/v5-lincoln/5.3.md b/release-notes/v5-lincoln/5.3.md index 84cb250ff..fd648cd98 100644 --- a/release-notes/v5-lincoln/5.3.md +++ b/release-notes/v5-lincoln/5.3.md @@ -77,6 +77,7 @@ Every replication retry now follows one jittered backoff schedule. Subscription ### Nodes and Clones `update_node` now works; the documented operation was never registered. A request that changes only `revoked_certificates` or `shard` updates the node record locally without resetting replication ([harper-pro#916](https://github.com/HarperFast/harper-pro/pull/916)). See [Update Node](/reference/v5/replication/clustering#update-node). A clone's wait for the initial sync is now bounded by the data size the leader reports and resumes across restarts, so a large clone no longer times out on a fixed budget or starts over ([harper-pro#661](https://github.com/HarperFast/harper-pro/pull/661)). + ## Subscription Catch-up Table subscriptions now skip superseded record updates during catch-up with `startTime` or `previousCount`, matching live delivery. A record updated at versions 2, 3, and 4 is delivered only at version 4 if that version is still current. The comparison uses the stored primary metadata version: a later publication can suppress a preceding mutation, and removing a record or tombstone makes its mutations ineligible. **JavaScript consumers that need retained mutation history, including these cases, must set `includeSuperseded: true`.** Raw-event subscriptions retain their existing default; `rawEvents: true` includes superseded events unless `includeSuperseded: false` is explicit. From 6324711fa59c7a992085b5728ef9e936eba898ed Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Wed, 30 Sep 2026 08:01:50 -0600 Subject: [PATCH 09/12] docs(release-notes): scope durable MQTT superseded-delivery claim to QoS 1/2 The Upgrade Notes bullet said durable MQTT subscriptions unconditionally receive every publication, but core only sets includeSuperseded for durable QoS 1/2 sessions -- QoS 0 durable subscriptions get current-state delivery like everything else. That contradicted the new Subscription Catch-up section landed just below it. Cross-model pre-push review caught the self-contradiction (harper-domain adjudication, finding 5.3.md:27). Co-Authored-By: Claude Sonnet 5 Dispatch-Task: pr-maint-92e4b54bf5b8178a01b68ecdd48dc2b0 --- release-notes/v5-lincoln/5.3.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/release-notes/v5-lincoln/5.3.md b/release-notes/v5-lincoln/5.3.md index fd648cd98..6bb612790 100644 --- a/release-notes/v5-lincoln/5.3.md +++ b/release-notes/v5-lincoln/5.3.md @@ -24,7 +24,7 @@ Read these before upgrading a production node or cluster. - **Each node rewrites `/ssh/config` once at startup** to add `# BEGIN/END harper ssh key ` lines around each key's block. An ssh config damaged by the pre-5.3 block-matching bug is not repaired automatically: run `get_ssh_key`, then `delete_ssh_key`, then `add_ssh_key` with the returned sealed key and the correct `host`/`hostname`. See [SSH Key Blocks Are Marked in the ssh Config](#ssh-key-blocks-are-marked-in-the-ssh-config). - **npm installs that use S3 export or import must add `@aws-sdk/client-s3` and `@aws-sdk/lib-storage`.** The AWS SDK is now an optional peer dependency. Without it, `export_to_s3` and `import_from_s3` return `501` with the install command. The official Docker image still includes it ([harper#2609](https://github.com/HarperFast/harper/pull/2609)). See [`export_to_s3`](/reference/v5/operations-api/operations#export_to_s3). - **Restoring a backup taken with `exclude_blobs` requires `allow_engine_only`.** Such a restore previously succeeded with only a log warning and could leave records pointing at the wrong blobs. A scripted engine-only restore, including one into a new `target_database`, fails until the flag is added ([harper#2646](https://github.com/HarperFast/harper/pull/2646)). -- **Subscription catch-up delivers only the current version of each record by default.** Pass `includeSuperseded: true` to `Table.subscribe` to also receive the retained history. `rawEvents: true` still includes every event, and durable MQTT subscriptions still receive every publication ([harper#2767](https://github.com/HarperFast/harper/pull/2767)). +- **Subscription catch-up delivers only the current version of each record by default.** Pass `includeSuperseded: true` to `Table.subscribe` to also receive the retained history. `rawEvents: true` still includes every event, and durable MQTT subscriptions with QoS 1 or 2 still receive every retained publication ([harper#2767](https://github.com/HarperFast/harper/pull/2767)). See [Subscription Catch-up](#subscription-catch-up). - **`searchByIndex` and a custom index's `search()` take an options object.** A positional fifth argument to `searchByIndex` now throws a `TypeError` instead of silently permitting the scan it was meant to forbid ([harper#2187](https://github.com/HarperFast/harper/pull/2187)). - **Startup waits at most `deployment.startupInstallTimeout` for component installs** before opening listeners. The default is 10 minutes, and `0` restores the previous unbounded wait. An install that runs past the limit finishes in the background, and when it succeeds Harper requests a restart to load it ([harper#2784](https://github.com/HarperFast/harper/pull/2784)). - **Some requests Harper used to accept are now refused:** `set_configuration` with unrecognized parameters ([harper#2272](https://github.com/HarperFast/harper/pull/2272)); a deploy or `drop_component` whose root config change `HARPER_CONFIG` or `HARPER_SET_CONFIG` would undo at the next start, with a `409` ([harper#2801](https://github.com/HarperFast/harper/pull/2801)); and `Table.clear()` on a table with an active full-text declaration, with a `501` ([harper#2616](https://github.com/HarperFast/harper/pull/2616)). From 2a6c18b00e904c66e4b7a06ef597980da409c61c Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 5 Oct 2026 18:34:16 -0600 Subject: [PATCH 10/12] docs(resources): format resuming-from-a-position table (prettier) Rebase conflict resolution introduced ragged column widths when merging the databaseGeneration row into the SubscriptionRequest options table; prettier re-pads it. Dispatch-Task: pr-maint-92e4b54bf5b8178a01b68ecdd48dc2b0 Co-Authored-By: Claude Sonnet 5 --- reference/resources/resource-api.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index 26185dcf6..da064a259 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -289,10 +289,10 @@ As of v5.4.0, a subscription on a RocksDB database can resume from a position th Harper refuses a position it cannot replay completely: -| Error | Status | Code | When | -| -------------------------------- | ------ | ----------------------------- | ---------------------------------------------------------------------------------------------------------- | -| `DatabaseGenerationChangedError` | 409 | `DATABASE_GENERATION_CHANGED` | The database was replaced by a restored or copied state after the position was recorded, or it is LMDB. | -| `ResumeHistoryUnavailableError` | 410 | `RESUME_HISTORY_UNAVAILABLE` | Audit retention pruned history after the position, or one record has more than 10,000 versions after it. | +| Error | Status | Code | When | +| -------------------------------- | ------ | ----------------------------- | -------------------------------------------------------------------------------------------------------- | +| `DatabaseGenerationChangedError` | 409 | `DATABASE_GENERATION_CHANGED` | The database was replaced by a restored or copied state after the position was recorded, or it is LMDB. | +| `ResumeHistoryUnavailableError` | 410 | `RESUME_HISTORY_UNAVAILABLE` | Audit retention pruned history after the position, or one record has more than 10,000 versions after it. | A subscription to one record is checked against that record's own history: it resumes as long as the record's retained version history still reaches back to the position, even when retention has pruned other history past it. A record first written after the position is checked against retention like a table subscription. From da421cf71d6e25c62b69ef4e204f1601a2e7d398 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 5 Oct 2026 18:49:08 -0600 Subject: [PATCH 11/12] docs(release-notes): note the 5.3 same-transaction ack gap, fix link label MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Post-push review: 5.3's durable QoS 1/2 claim ("preserving intermediate retained publications") overclaimed once 5.4's release notes documented that a saved position could still skip an unacknowledged message sharing a transaction with one already acknowledged — fixed only in 5.4. Add that caveat with a link to the 5.4 fix. Also fix the link label mismatch ("Subscription options and migration" -> "Superseded record updates") to match its target heading. Dispatch-Task: pr-maint-92e4b54bf5b8178a01b68ecdd48dc2b0 Co-Authored-By: Claude Sonnet 5 --- release-notes/v5-lincoln/5.3.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/release-notes/v5-lincoln/5.3.md b/release-notes/v5-lincoln/5.3.md index 6bb612790..b1d97de1b 100644 --- a/release-notes/v5-lincoln/5.3.md +++ b/release-notes/v5-lincoln/5.3.md @@ -82,7 +82,7 @@ Every replication retry now follows one jittered backoff schedule. Subscription Table subscriptions now skip superseded record updates during catch-up with `startTime` or `previousCount`, matching live delivery. A record updated at versions 2, 3, and 4 is delivered only at version 4 if that version is still current. The comparison uses the stored primary metadata version: a later publication can suppress a preceding mutation, and removing a record or tombstone makes its mutations ineligible. **JavaScript consumers that need retained mutation history, including these cases, must set `includeSuperseded: true`.** Raw-event subscriptions retain their existing default; `rawEvents: true` includes superseded events unless `includeSuperseded: false` is explicit. -MQTT durable subscriptions with QoS 1 or 2 explicitly include superseded versions in live delivery and reconnect catch-up, preserving intermediate retained publications. Non-retained published messages remain independent and are not suppressed by the table version check. See [Subscription options and migration](/reference/v5/resources/resource-api#superseded-record-updates) and [MQTT durable sessions](/reference/v5/mqtt/overview#durable-sessions). +MQTT durable subscriptions with QoS 1 or 2 explicitly include superseded versions in live delivery and reconnect catch-up, preserving intermediate retained publications — except that a saved position could still skip an unacknowledged message sharing a transaction with one already acknowledged; 5.4 closes that gap (see [Durable MQTT Sessions No Longer Replay Short](/release-notes/v5-lincoln/5.4#durable-mqtt-sessions-no-longer-replay-short)). Non-retained published messages remain independent and are not suppressed by the table version check. See [Superseded record updates](/reference/v5/resources/resource-api#superseded-record-updates) and [MQTT durable sessions](/reference/v5/mqtt/overview#durable-sessions). ## Node Identity From d0464bde64a00988f60cb55ca05f0a4b5b03e81a Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 5 Oct 2026 18:54:47 -0600 Subject: [PATCH 12/12] docs(release-notes): scope the upgrade-notes QoS 1/2 claim to match 5.3's actual behavior MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Upgrade Notes bullet repeated the same "still receive every retained publication" overclaim already scoped in the Subscription Catch-up section (release-notes/v5-lincoln/5.3.md:85) — caught by delta pre-push review after that fix. Dispatch-Task: pr-maint-92e4b54bf5b8178a01b68ecdd48dc2b0 Co-Authored-By: Claude Sonnet 5 --- release-notes/v5-lincoln/5.3.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/release-notes/v5-lincoln/5.3.md b/release-notes/v5-lincoln/5.3.md index b1d97de1b..4dda67657 100644 --- a/release-notes/v5-lincoln/5.3.md +++ b/release-notes/v5-lincoln/5.3.md @@ -24,7 +24,7 @@ Read these before upgrading a production node or cluster. - **Each node rewrites `/ssh/config` once at startup** to add `# BEGIN/END harper ssh key ` lines around each key's block. An ssh config damaged by the pre-5.3 block-matching bug is not repaired automatically: run `get_ssh_key`, then `delete_ssh_key`, then `add_ssh_key` with the returned sealed key and the correct `host`/`hostname`. See [SSH Key Blocks Are Marked in the ssh Config](#ssh-key-blocks-are-marked-in-the-ssh-config). - **npm installs that use S3 export or import must add `@aws-sdk/client-s3` and `@aws-sdk/lib-storage`.** The AWS SDK is now an optional peer dependency. Without it, `export_to_s3` and `import_from_s3` return `501` with the install command. The official Docker image still includes it ([harper#2609](https://github.com/HarperFast/harper/pull/2609)). See [`export_to_s3`](/reference/v5/operations-api/operations#export_to_s3). - **Restoring a backup taken with `exclude_blobs` requires `allow_engine_only`.** Such a restore previously succeeded with only a log warning and could leave records pointing at the wrong blobs. A scripted engine-only restore, including one into a new `target_database`, fails until the flag is added ([harper#2646](https://github.com/HarperFast/harper/pull/2646)). -- **Subscription catch-up delivers only the current version of each record by default.** Pass `includeSuperseded: true` to `Table.subscribe` to also receive the retained history. `rawEvents: true` still includes every event, and durable MQTT subscriptions with QoS 1 or 2 still receive every retained publication ([harper#2767](https://github.com/HarperFast/harper/pull/2767)). See [Subscription Catch-up](#subscription-catch-up). +- **Subscription catch-up delivers only the current version of each record by default.** Pass `includeSuperseded: true` to `Table.subscribe` to also receive the retained history. `rawEvents: true` still includes every event, and durable MQTT subscriptions with QoS 1 or 2 still receive every retained publication except one gap closed in 5.4 ([harper#2767](https://github.com/HarperFast/harper/pull/2767)). See [Subscription Catch-up](#subscription-catch-up). - **`searchByIndex` and a custom index's `search()` take an options object.** A positional fifth argument to `searchByIndex` now throws a `TypeError` instead of silently permitting the scan it was meant to forbid ([harper#2187](https://github.com/HarperFast/harper/pull/2187)). - **Startup waits at most `deployment.startupInstallTimeout` for component installs** before opening listeners. The default is 10 minutes, and `0` restores the previous unbounded wait. An install that runs past the limit finishes in the background, and when it succeeds Harper requests a restart to load it ([harper#2784](https://github.com/HarperFast/harper/pull/2784)). - **Some requests Harper used to accept are now refused:** `set_configuration` with unrecognized parameters ([harper#2272](https://github.com/HarperFast/harper/pull/2272)); a deploy or `drop_component` whose root config change `HARPER_CONFIG` or `HARPER_SET_CONFIG` would undo at the next start, with a `409` ([harper#2801](https://github.com/HarperFast/harper/pull/2801)); and `Table.clear()` on a table with an active full-text declaration, with a `501` ([harper#2616](https://github.com/HarperFast/harper/pull/2616)).