Skip to content

Add configurable table-backed view definitions - #72

Merged
nicosuave merged 3 commits into
mainfrom
codex/view-definition-store
Nov 12, 2026
Merged

nicosuave merged 3 commits into
mainfrom
codex/view-definition-store

Conversation

@nicosuave

@nicosuave nicosuave commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

Yardstick measure definitions currently live in process memory, so reopening a database does not recover their semantics. Add the session setting yardstick_definition_table to store definitions in a fully qualified regular table in a writable DuckDB attachment:

SET yardstick_definition_table = 'lake.semantic.yardstick_views';
CREATE VIEW sales_v AS
SELECT region, SUM(amount) AS MEASURE revenue FROM lake.sales.orders;

The selected table is the sole source of truth. CREATE, IF NOT EXISTS, semantic or plain REPLACE, and DROP use ordinary table operations in the caller's transaction. Later sessions resolve definitions automatically after selecting the same store, without duplicate native views or export/import calls. An unset setting preserves existing behavior; temp.main.yardstick_views supports session-local storage.

Definition mutations acquire and verify a reserved guard row in the same table before writing, rejecting conflicting or stale writers without requiring backend UNIQUE constraints. New tables initialize the guard atomically. Existing guard-free tables remain readable and provide one-time initialization instructions requiring exclusive writer access. Direct external table edits bypass this protocol.

Query-local namespaces isolate stores and sessions. Lookup and DROP respect temporary native views; saved sources retain lexical CTE visibility and qualified catalog/schema names. Prepared queries rebind after replacements. Simple native-table queries skip definition scans; semantic lookups use DuckDB and attachment caching without a separate TTL cache. Includes native and legacy binder integration and usage documentation.

Validation:

  • Builds passed against DuckDB 1.5.5 and pinned 2.0.
  • Native SQL suite: 5,000 assertions passed; one optional grammar-fixture test skipped.
  • Rust baseline: 164 tests passed, 17 ignored; no Rust changes in the subsequent fixes.
  • Cross-process CLI suite: 11 passed on 2.0, including real DuckLake create/replace/rollback/drop/reopen; 10 passed on 1.5.5 with the optional DuckLake case skipped.
  • Covered competing case-variant creates, old transaction snapshots, repeated mutations and rollback, single-thread execution, temporary shadowing, nested/self-named CTEs across reopened sessions, guard initialization/corruption, read-only recovery, prepared parameters, and EXPLAIN without duplicate definitions.

PostgreSQL and Iceberg connectors were not directly exercised. The selected attachment must provide transactional table reads/writes and reject conflicting updates; all participating definition writers must use the guard protocol.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-12T04:09:28.937413Z 1672c3d PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1672c3db9c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/definition_store.cpp Outdated
Comment on lines +879 to +882
auto definition = definitions.Find(GetName(drop));
if (definition) {
return BindDefinitionWrite(context, parent,
"DELETE FROM " + table + " WHERE " + KeyPredicate(definition->name));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Respect temporary-view shadowing before deleting stored definitions

When a temporary native view shadows a stored permanent view with the same name, an unqualified DROP VIEW finds the permanent definition here and deletes its row instead of dropping the temporary view. For example, creating stored v, then CREATE TEMP VIEW v AS SELECT 1, then DROP VIEW v destroys the persistent definition while leaving the temporary view intact. Resolve the native target and its precedence before choosing the definition-table DELETE, consistent with the native-entry check already used by YardstickStoredRelationName.

Useful? React with 👍 / 👎.

Comment thread src/definition_store.cpp
visit_ref = [&](TableRef &ref) {
if (ref.type != TableReferenceType::BASE_TABLE || !ref.query_location.IsValid()) return;
auto input = GetName(ref.Cast<BaseTableRef>());
if (input.catalog.empty() && input.schema.empty() && cte_names.count(StringUtil::Lower(input.name))) return;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Track CTE names within their lexical scope when qualifying sources

If a nested subquery declares a CTE named sales, the query-wide cte_names set also exempts an unrelated outer FROM sales from qualification. A declaration such as SELECT SUM(amount) AS MEASURE total FROM sales WHERE EXISTS (WITH sales AS (SELECT 1) SELECT * FROM sales) therefore saves the outer source unqualified. After a later session changes its schema/search path, that stored measure can silently read a different sales table, contrary to the source-binding guarantee. Track visible CTEs per query scope rather than excluding every occurrence of any collected CTE name.

Useful? React with 👍 / 👎.

Comment thread src/definition_store.cpp
Comment on lines +870 to +873
auto sql = existing
? "UPDATE " + table + " SET definition_sql = " + QuoteString(definition_sql) + " WHERE " + KeyPredicate(name)
: "INSERT INTO " + table + " (view_catalog, view_schema, view_name, definition_sql) VALUES (" +
QuoteString(name.catalog) + ", " + QuoteString(name.schema) + ", " + QuoteString(name.name) + ", " + QuoteString(definition_sql) + ")";

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Enforce definition-key uniqueness across concurrent creates

With an initialized definitions table, two sessions can concurrently create the same previously absent view: both binding-time lookups see no existing definition, and both unconstrained INSERTs can commit. The automatically created table has no unique constraint on the definition key, so the next semantic lookup encounters duplicate rows and DefinitionScope throws, preventing queries against otherwise unrelated stored views as well. Enforce case-insensitive key uniqueness atomically, or otherwise serialize conflicting creates; the snapshot existence check alone does not protect the store.

Useful? React with 👍 / 👎.

@nicosuave
nicosuave merged commit 2b13184 into main Nov 12, 2026
28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant