Repository navigation
1.2.0 rc: parser as WebAssembly (no cgo), PostgreSQL 18 parser embedded - #1
Merged
Merged
Conversation
…ssembly and run by wazero — Parse / Deparse / SplitWithScanner / ParsePlPgSqlToJSON with pg_query_go's signatures, one pooled instance per call, protobuf bytes and error messages identical to cgo (tested against it); every call site now goes through it, node types still pg_query_go's wasm/build.sh rebuilds pg_query_<major>.wasm from a libpg_query tag (emscripten longjmp emulation for the syntax-error path, stderr redirect compiled out), wasm/shim.c wraps the struct-returning C API in pointer-returning entry points.
…o into internal/pgparse (pgparse.Node, pgparse.ParseResult, ...), pg_query_go is gone from go.mod, and the binary builds with CGO_ENABLED=0 — every test, the regress baseline, the golden files and the Facts snapshot unchanged wazero compiles the module through an on-disk cache (user cache dir / sqlshape / wazero): the first process pays about a second, every later one about 30 ms, which matters because go vet starts one analyzer process per package. README and release.yml no longer ask for a C compiler; CHANGELOG [Unreleased] records the change.
…side 17 — pgparse.PG17 / PG18, Default = PG17 until the rest of sqlshape supports 18 — and every version's tree is read from the parser's JSON output into node types generated from the newest version's proto protobuf field numbers are not stable across libpg_query versions (17→18 renumbers 267 fields and 434 enum values), so the protobuf path cannot serve two versions; the JSON output names fields and node types, protojson reads it into the newest types, and it is faster than the protobuf path anyway (215 µs vs 273 µs a statement). An older version's renamed fields are rewritten on the JSON before decoding (17→18: returningList → returningClause.exprs, plus three fields a raw tree never carries); a rewrite the table misses fails the decode on an unknown field instead of vanishing. Over the 43,226 regress statements 17 accepts: all decode through the rewrite, 18 rejects none, and 18's JSON path equals its protobuf path. ReturningList call sites read GetReturningClause().GetExprs(); Deparse runs on the newest module. All tests, the regress baseline, the golden files and the Facts snapshot unchanged.
…o) are dropped from the coverage profile — 40k lines of accessors pulled the total from 93% to 69% and tripped the 90% gate
…8`, anywhere in the file — and every statement, the schema's own DDL and the application's SQL, is parsed with that version's grammar (Schema.Version; pgparse.Default when undeclared) The declaration belongs to the file, not to the statement below it: directives() skips it so it is never a stray directive, and it is read before the first parse. Two declarations that disagree, a version without an embedded parser, and a declaration inside DDL applied later (Schema.Apply) are errors — nothing is judged until they are fixed. The analyzer does not yet know 18's constructs (RETURNING OLD/NEW resolves as a missing table); this commit only routes the grammar.
…replays a profile with block positions from an earlier build of the packages it covers but does not depend on, and the merged total fell from 93% to 82% after an edit that moved lines in analyze and schema
…on — internal/catalog/data/<major>/ (catalog.Load(major); 18 generated from 18.3 with all 16 extension dumps), oracle.StartVersion (18.3.0 beside 17.5.0), and the probe follows $SQLSHAPE_PG with a baseline per version schema.Load / analyze.Load read the declaration first and load that version's catalog; a catalog of another version handed to LoadWithHooks is an error. pgtest starts the PostgreSQL the schema declares. The probe writes the declaration above the corpus DDL so the analyzer judges with the probed version's grammar and catalog. gen takes -pg and -all-ext. CI gains a pg18 job (catalog, oracle, the 18 probe over REL_18_3; nothing asserted until 18 has a baseline). First 18 probe: 23380 statements, 420 hits — 321 of them 42P01 after the loader dropped 18 DDL (WITHOUT OVERLAPS, virtual generated columns, NOT ENFORCED). That is the work that follows.
…with the C++ actions stripped (one grammar per version, as with libpg_query), the lexer is rewritten, the catalog of built-in functions is measured from a real mysqld rather than transcribed, mysql-test is the corpus; goyacc vs bison-to-wasm is decided by a spike on the corpus's acceptance rate
…ken reads five pieces of server state (sql_mode, default_collation_for_utf8mb4, the connection charset, comment collection, a deprecation warning), so a THD shim of that size lets sql_lex.cc, gen_lex_hash's keyword table and strings/ compile with the stripped sql_yacc.yy — bison + emcc to wasm, as libpg_query; sql_mode becomes a schema declaration; hints (sql_hints.yy) come later
…create.cc's func_array (347 names with arities), the 81 keyword functions in sql_yacc.yy's actions, the Item class hierarchy for the result category, and the declarative resolve_type calls for parameter types and nullability; a real mysqld probes what is left and checks the rest
…l — the MySQL server is GPLv2 and cannot be combined with the Apache 2.0 sqlshape; the runtime users link stays Apache, the parser is only needed by the tool, shared parts are imported Apache-to-GPL; a parser subprocess (gray), an Apache-licensed compatible parser (drops the principle) and hand-written grammar (drops it where it matters most) were set aside
…T NULL col, dropped by DROP CONSTRAINT), ENFORCED / NOT ENFORCED on CHECK and FOREIGN KEY (Constraint.NotEnforced — a NOT ENFORCED constraint is no failure mode), ALTER CONSTRAINT ... ENFORCED / DEFERRABLE, WITHOUT OVERLAPS keys and PERIOD foreign keys (rendered back as such), VIRTUAL generated columns (Column.GeneratedVirtual; 18's default when neither word is written) A 17 tree is read into 18's shape with 18's new fields set to what 17 meant: constraints enforced, generated columns stored. The regress probe parsed every statement with the default (17) grammar to classify it, so 18-only DDL was "unparsed" and never reached the analyzer's schema — 321 of 18's 420 hits were downstream of that; the probe now parses with the probed version. 18 probe: 102 hits, 45 of them RETURNING OLD / NEW.
…te, and the aliases of RETURNING WITH (OLD AS o, NEW AS n), resolve as row variables — reached only by a qualified reference (old.col, old.*, old, old.ctid), never by a bare column name or a bare *; old does not exist for an inserted row and new not for a deleted one, so those columns and whole rows are nullable (the oracle reports the column's own NOT NULL there, which is all its Describe knows) A FROM item named old or new hides the default alias; an alias the clause chooses may not clash with a FROM item or with the other alias (42712); OLD or NEW written twice is 42601, as is a RETURNING that expands to no column. A 17 schema has no row variables. 18 probe: 67 → 62 hits, 10 of them the nullability the oracle cannot see.
… before the zone's first transition) is a valid zone in a datetime literal, as 18 accepts it; and the 18 regress probe has its baseline — 58 hits over 23,384 statements, 10 of them the RETURNING old / new nullability the oracle cannot see, the rest 17's known set plus a handful of 18 checks not yet ported (JSON_VALUE collation 42P21, jsonpath argument type 42804, DISTINCT ON with grouping sets, aclitem input, an outer aggregate over a nested CTE, one horology input form) The pg18 CI job now asserts that baseline.
…file without `-- sqlshape: postgres N` (schema text built in code keeps pgparse.Default), and the migration side follows the declaration — the canonical server, pg_dump's text (which carries no declaration) and the consumer index are all judged with the declared version The regress probe's crash skip was a PG 17 workaround applied to 18 as well; 18 takes those MERGE statements, so it is 17-only and the "4 skipped (oracle crash)" are gone. Every schema.sql in the repository opens with the declaration; README, checks.md / checks.ja.md (Part 3) and the CHANGELOG say so.
…ALUE DEFAULT whose collation is not the RETURNING type's (42P21; a domain now carries its COLLATE, and a cast to it takes it), a JSON path that is not a jsonpath or a string (42804), datetime number fields of anything but digits and decimal points and fractions with anything but digits (22007), aclitem's getid where "" opens and closes a name so `/""` is a missing grantor (22P02; the aclitem input syntax is checked at all now), and an outer-level aggregate over a CTE of the subquery it is written in (0A000) DISTINCT ON is matched against ORDER BY by expression rather than by the expression's position in the text — an analyzer bug 18's corpus found, in effect for 17 too. The 18 baseline is down to 47: 17's known set, the RETURNING old / new nullability the oracle cannot see, and one rule-rewritten DELETE RETURNING where the oracle's Describe returns nothing while the analyzer raises 0A000 as PostgreSQL does.
…ows, amd64 and arm64 each (Windows and Intel macOS are new; a plain Go build needs no per-platform runner) — packs tar.gz / zip, smoke-tests the runner's own binary and publishes with checksums
…when its scalar columns are fixed by equality and its range column equals a known value or contains a known point of the element type (valid_at @> $1::date, also through a join on another single row's column): no two rows with the same scalar values have overlapping ranges; a range on the known side (possibly empty) or an overlap does not do Tests for 18's DML: a PERIOD foreign key raises 23503 like any other, a VIRTUAL generated column reads like a column and is never assigned; an UPDATE of a generated column says "can only be updated to DEFAULT" as PostgreSQL does.
…other PostgreSQL major version than schema.sql declares — the DDL is still derived and judged by the declared version's rules, and the message says which line to change if the database is the one meant; a database that cannot be asked leaves the report to the step that follows
…schema.sql declares the version (required), one parser / catalog / oracle / corpus per major version, one set of node types read from JSON with per-version rewrites, version branches in the analyzer, the fidelity numbers and the rejected alternatives; internal/pgparse/README.md holds the runbook for adding a version
… per-release cache), the regress probe follows SQLSHAPE_PG with a baseline per version, the PG 17 MERGE crash skip is 17-only
…ser as WebAssembly, prebuilt binaries for six platforms, the temporal One proof; fixed: DISTINCT ON matched by expression, the generated-column UPDATE message
…irst run downloads and caches (the wasm module for the checker; a PostgreSQL only for pgtest and the migration commands)
…first run caches; the quickstart schema declares its version
…QL's own so the syntax is, and the regress numbers per version (19 of 22,103 on 17, 31 of 23,384 on 18, every one listed)
… a DELETE may be NULL whatever the column declares; a NOT ENFORCED constraint is no failure mode
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release candidate branch for 1.2.0. Opened as a draft so
test.ymlruns on every push (it runs on pull requests and main only).internal/pgparse: libpg_query compiled to WebAssembly, run by wazero; pg_query_go and cgo are gone,CGO_ENABLED=0builds.Merged by fast-forward when 1.2.0 is cut, as 1.1.0 was.
🤖 Generated with Claude Code