Skip to content

Document schema-defined sharding and regional placement - #753

Draft
kriszyp wants to merge 23 commits into
mainfrom
docs/slot-residency
Draft

kriszyp wants to merge 23 commits into
mainfrom
docs/slot-residency

Conversation

@kriszyp

@kriszyp kriszyp commented Oct 24, 2026 •

Copy link
Copy Markdown
Member
An engineer explains how keys select slots and shards

⊙ Problem

Operators need a precise activation and recovery procedure before using schema-defined sharding, particularly when converting existing owners or replacing a lost data root.

🤔 Scope and recovery contract

Document the bounded recovery contract explicitly. Snapshot-only recovery uses a fresh root and new node name. Same-name recovery requires a complete stopped-node backup preserving audit through the last accepted write. Do not imply arbitrary relay recovery or online slot migration.

💡 Solution

Add the @sharded guide, configuration and operations references, conversion assertions, recovery diagnostics, compatibility limits, and release notes. Keep legacy residency APIs separately named.

Refs HarperFast/harper-pro#980.

Companions: core #3527, Pro #1116.

⚖️ Alternatives

Keep the existing residency API reference for applications that use it. The new guide names the declarative activation and recovery restrictions explicitly; it does not promise online slot migration or incarnation-based recovery.

Framing-Verdict: chosen-approach-sound

Documentation scope decisions

🤔 Supported rollback route

The in-place recipe is explicitly for equivalent legacy by-id conversions. A fresh declarative table has no old placement policy to restore and no promised in-place downgrade. A general migration workflow is outside these companions. The guide URL remains stable; the title and public directive use sharding terminology.

🔧 Changes

Product behavior, admission, and recovery walkthrough

Product and architecture tour

Declare and inspect sharding

The sharding guide shows key, tenant, and regional placement. Schema reference links to it. Deployment topology and configuration options separate the local shard from the complete shard list. Map storage explains the immutable database map; confirmation usage covers explicit acknowledgment and post-commit uncertainty. Subscription row payloads limits sharded non-owner filters to indexed projections.

Convert and recover safely

Operator operations distinguish map creation, comparison, and trusted existing-owner assertions. The conversion procedure pauses legacy writers through activation and gives rollback boundaries. Lost-node recovery explains complete-backup restoration, fresh-name co-owner copies, retired-origin retention, and whole-shard loss. Status fields names visible recovery holds and their remedies. Incarnation-based identity is future work tracked in Pro #1123.

Find the new workflow

The replication overview and sidebar point readers to the new guide. Release notes summarize activation, recovery holds, and the new-name recovery restriction.

✅ Verification

Full formatting and the production documentation build pass, with no broken-anchor errors. Schema, operation names, configuration fields, and recovery remedies were checked against core #3527 and Pro #1116.

Native placement and recovery checks passed with the replication pool off and on at local Pro b44aa492 / core a9e3dd71, including previous-binary conversion, same-name refusal, new-name recovery, and complete-backup restoration. This documentation PR introduces no runtime behavior.

🤖 Generated by OpenAI Codex; posted via @kriszyp.

Release target: v5.4 — companion documentation for the next minor release

Complexity: medium

Related PRs: #733 overlaps, #740 overlaps, #746 overlaps, 7 others independent

Deferred findings

Review-Coverage: authored=codex; ran=gemini,cursor-composer,claude; adjudicated=domain; declined=cursor-grok,muse; rounds=14; full=7 @ 7483f62

Review-Attention: read ~3m (decisions: conversion-completeness-trust, rollback-maintenance-interface) @ 7483f62

@kriszyp kriszyp added this to the v5.4 milestone Oct 24, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces comprehensive documentation for the new Declarative Residency feature in version 5.4.0, which allows defining table placement policies directly in the schema using the @residency directive. The changes span configuration options, schema directives, system tables, operations APIs, replication/sharding guides, and release notes. The review feedback suggests a minor improvement in reference/replication/residency.md to restructure a paragraph for better readability and to avoid grammatically disjointed phrasing regarding the 503 response.

Comment thread reference/replication/residency.md Outdated
@github-actions
github-actions Bot temporarily deployed to pr-753 October 24, 2026 03:49 Inactive
@github-actions

Copy link
Copy Markdown

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-753

This preview will update automatically when you push new commits.

@github-actions
github-actions Bot temporarily deployed to pr-753 October 24, 2026 03:57 Inactive
@github-actions

Copy link
Copy Markdown

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-753

This preview will update automatically when you push new commits.

@github-actions

Copy link
Copy Markdown

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-753

This preview will update automatically when you push new commits.

kriszyp and others added 16 commits October 26, 2026 08:57
Dispatch-Task: hp-980-slot-residency
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Dispatch-Task: hp-980-slot-residency
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Dispatch-Task: hp-980-slot-residency
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Dispatch-Task: hp-980-slot-residency
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Dispatch-Task: hp-980-slot-residency
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Dispatch-Task: hp-980-slot-residency
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Dispatch-Task: hp-980-slot-residency
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Document conditional shard configuration, initialization and verification operations, status diagnostics and schema metadata. Give operators an executable replica-drain procedure and require custom-map installation before declarative application deployment on joining nodes.

Dispatch-Task: hp-980-slot-residency
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Dispatch-Task: hp-980-slot-residency
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Dispatch-Task: hp-980-slot-residency
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Dispatch-Task: hp-980-slot-residency
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Dispatch-Task: hp-980-slot-residency
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Dispatch-Task: hp-980-slot-residency
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Dispatch-Task: hp-980-slot-residency
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Dispatch-Task: hp-980-slot-residency
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Dispatch-Task: hp-980-slot-residency
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
kriszyp and others added 6 commits October 26, 2026 08:57
Dispatch-Task: hp-980-slot-residency
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Dispatch-Task: hp-980-slot-residency
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Dispatch-Task: hp-980-slot-residency
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Explain hashed tenant affinity, logical regional slots, immutable policy, older residency APIs, and the spread follow-up in #1122.

Dispatch-Task: hp-980-slot-residency
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Dispatch-Task: hp-980-slot-residency

Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Dispatch-Task: hp-980-slot-residency

Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Dispatch-Task: hp-980-slot-residency
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
@github-actions

Copy link
Copy Markdown

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-753

This preview will update automatically when you push new commits.

This branch was successfully deployed

1 active deployment
pr-753 — 7483f62b Deployed Oct 26, 2026 by github-actions[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant