[OmarAlJarrah]

Summary

Two readings of the same facts lived side by side. internal/bind already knew where schemas are, which resource holds a position, and when a merge past the bound may hide a key. internal/anchors and schema.declaresResourceIDAbove each walked the raw tree again with their own, looser idea of all three, and another document's identifiers were read by neither. Most of what is closed here follows from that one split:

• An $anchor, $dynamicAnchor or $id in another document named nothing (openapi: an $anchor or $id another document declares names nothing #1049), and a $dynamicRef written there could not expand (openapi: a $dynamicRef written in another document is never expanded #1031).
• A merge past the bound hid an $id from the second walk, so a $dynamicRef expanded across a resource boundary (openapi: a merge chain past MergeDepthLimit hides an $id, expanding a $dynamicRef across a resource boundary #401).
• The second walk's warning named its own bounds when the merge bound had stopped it (openapi: the $dynamicAnchor index names its own walk bounds when the view's merge bound stopped it #454).
• It also said two false things nobody had filed: a $dynamicAnchor in an example or an x- extension counted as a declaration, so a reference read "declared 3 times", and a property named $id read as a resource boundary.

bind now reads each document's identifiers, and internal/anchors and declaresResourceIDAbove are gone. dynamicExpansion asks the table for the sole anchor (Table.DynamicAnchors) and for the resource a position is in (Table.InOwnResource), so a $dynamicRef and a $ref read one set of facts.

Reading another document

A document whose root names an openapi version is read by the grammar, as the source is, so an example or an extension in it declares nothing. Any other document has no grammar. Its root and any key of no schema vocabulary are read as holding schemas, and within a schema a keyword reads as it does in the source, so a literal, a value or an extension declares nothing. Reading only a pure schema document would not cover the shape #1049 is filed on, a map of schemas under keys of its own. $id gives its base below itself by the same reading, in every document.

Reading in passes

Whether a URI is an identifier depends on which documents were read, and which are read depends on what the URIs name. OAS 3.1.1 §4.3.1 requires complete documents before a reference is deemed unresolvable. So Bind runs passes: each answers every reached reference against the URIs the documents the pass before it reached declare, and, as it goes on past what it held, those the documents it reaches since declare, while no URI it asked for changes. A pass runs again if what the documents it reached declare changes a URI it asked about: it was told nobody declares that URI, a second schema now does, or a document no pass reaches any longer withdrew it. Only the last pass's answers stand. Each pass is a set operation, so no declaration order reaches an answer, and a document that neither declares nor names such a URI runs one pass.

A document a pass read as a location before another declared that URI is reached by no later pass, so it declares nothing, lowers nothing and is no Sources entry. Where what is reached turns on what is declared, passes need not settle: a document reached only through a URI it declares a second time is unreached when counted and reached when not. Once a pass would return to declarations an earlier one read against, every document read counts, which only grows, so the passes end, and that URI names neither schema.

The first read of an absolute URI waits until the pass's walks are done, so a URI a document read during the pass declares is never fetched. The documents held are then read together, a round for all of them rather than one each, so one of them may declare another's URI, which was then read for nothing, and the pass is read again. With a local server holding both, A: {$ref: 'http://127.0.0.1:18765/u1.yaml#/S'} and B: {$ref: 'http://127.0.0.1:18765/u2.yaml'}, where u1.yaml's S writes $id: 'http://127.0.0.1:18765/u2.yaml', the server logs a request for each, B resolves to that S, and u2.yaml is no Sources entry. Reading them one at a time would spend a round on each.

A pass that held documents reads them once its walks are done and goes on, for as long as what the documents it reached declare changes no URI it asked for. Only the hops and mapping values that waited for them are read again. A chain that meets a held hop waits and remembers nothing until nothing is held, so it is followed once, and each drain counts only what the pass reached since the last (identifiers.grow). A pass is read again only when a URI's declarations change, so a chain of documents each named by an absolute URI, none declaring a URI a reference asked for, is read in one pass, its cost growing with its length (#1349). It is the name that counts: a relative $ref in a document fetched by URL, or under an absolute $id, is held the same way. Every answer a continued pass keeps is one a new pass would give, since no URI it asked for changed, an anchor or a definition answers from its own document, and what a pass reaches only grows.

Counted in the binding's steps, a chain of 512 such documents costs 11,278 here and 5,645 on main, where beginning a pass for each held document cost 1,323,533; a chain of 16,000 costs 352,014, all resolved. Compiling a chain of file:// documents with external references allowed, GOMAXPROCS=1 GOGC=off, median instructions retired of three alternating runs a side, at 5fa0d5f against main at 02c972b, and the pass per link at 1db320a:

links	main	a pass per link	this PR
1,024	1.825 G	47.52 G	1.910 G
4,096	14.68 G	591.5 G, out of budget, 542 sources	15.07 G, 4,097 sources

The few percent over main is the identifiers each document is read for.

The golden spec, which names no document by an absolute URI, takes 862,702 binding steps here and 862,700 on main, and 106.9 G instructions here against 104.7 G on main (medians of three, collector on, ranges 105.3–107.5 G and 103.9–107.8 G, overlapping).

Every pass spends the one budget. The pass the budget runs out in is the last, wherever in it the budget runs out, and its answers stand. A reference a later pass would have answered otherwise keeps the cut pass's answer: Y: {$ref: 'a.yaml'}, beside a z.yaml whose schema writes $id: 'a.yaml', resolves to that schema, but to a.yaml itself under a budget the first pass alone uses up. That is main's answer, and the compile reports openapi/budget-exceeded.

This departs from the rounds sketched on #1049, which open every document each document's references spell. That reads documents nothing reaches, and an unreached document's $id could then turn a resolving URI into a duplicate. Passes read only what a reached reference names.

Other decisions

• A plain name stays in its resource. openapi: an $anchor or $id another document declares names nothing #1049's example expects other.yaml#tag to name an anchor declared under an $id, "as the same schemas written in spec.yaml resolve". They do not: the same document on main refuses #tag from outside that $id, as JSON Schema says. So Tag in that example stays unresolved, with a true reason, while a, b, ById and https://x.test/pet#tag resolve.
• $dynamicRef in another document may name an anchor at any position there. The source's restriction to a component rests on its names, and what another document declares has IDs from its pointer. Nothing but a reference builds a schema of another document, so the binder lands on the anchor as a reference would (finder.landOnAnchors).
• An expansion reaches what it names. The schema it lands on is walked and validated as one a $ref reaches, and draws the same findings.
• A read that was cut keeps the reference. Past the merge bound the index may undercount, so a $dynamicRef in that document is kept verbatim, where main warned and expanded anyway. The one warning that remains names the bound that fired.
• openapi: the component classifier collapses "no such entry" with "an entry keyed \"\"", so callers can restate a falsehood #405's third state lives in the classifier. ids.ComponentSchema and resolve.Scope.ComponentRef return an ids.ComponentState, and ComponentSchemaName and ComponentSchemaNamedEmpty are gone, so no caller can report a component keyed "" in another's words.

Behaviour to check on review

• A $dynamicAnchor in an example or an extension no longer makes a name a duplicate, and a property named $id or an $id in raw YAML no longer stops an expansion. Each of those references now expands.
• An anchor name 2020-12's grammar refuses is declared by none, as for a $ref, so $dynamicAnchor: 'pct%name' no longer expands. The validator already reports that document as an error.
• A URI two documents declare now names neither, as within one document, however far apart they are read. The same $id in the source and in a document it reaches resolved to the source's on main, since another document's was never read, and is openapi/unresolved-ref here.
• An $id in another document now gives the base a relative $ref below it is read against, as one in the source already did (openapi: an absolute-path $ref in a remote document is read as a local path #936). schemas/pet.json writing $id: https://example.com/schemas/pet.json and $ref: tag.json read the sibling file on main; here that $ref names https://example.com/schemas/tag.json, refused by default as a network read. openapi: let the caller map an identifier's URL to a local location #1051 proposes letting a caller map such a URL to a local file.
• The merge-bound warning adds that "where that mapping is a schema, no $dynamicRef in its document is expanded". It fires once, at the source, for a mapping in any document, and only a schema merging past the bound keeps its document's $dynamicRef, so a response past the bound leaves the source's references expanding.

Limitations

• Whether an $id spelled as an absolute path names the file a relative $ref reaches turns on whether the spec was named by an absolute path, since the two keys meet only then. An $id in the source does this on main; reading another document's identifiers reaches it there too. openapi: whether an absolute $ref names the spec depends on how the working directory is spelled #1040 weighs naming the spec once, which settles every such site, so this PR states it in the references rule's status rather than choosing for it.
• The held documents are read together, as above.
• A chain in which each document declares, by $id, the URI the one before it asked for is read again for each link, 21 passes for 20 links, so its cost grows with the square of its length: 256 such file:// links retire 11.8 G instructions at fba9c3f, where 1,024 links declaring nothing retire 1.8 G. main reads none of their identifiers, so it fails the chain's first and pays nothing. The references rule's Status and reread's comment state it, and openapi: a chain of documents each declaring the URI the one before asked for costs a binding pass per link #1372 proposes the reading that makes it linear.

Test plan

• bind.TestBind_ReadsWhatAnotherDocumentDeclares holds the identifier readings in both declaration orders, bind.TestBind_DeclarationOrderReachesNoAnswer a tangle of three documents in three orders, and bind.TestBind_AURIASecondDocumentDeclaresNamesNeither a duplicate a pass apart.
• e2e.TestCompile_AnIdentifierIsNotRequested records the reader with external references allowed: https://x.test/pet is not requested.
• e2e.TestCompile_ADocumentNothingReachedIsNoSource and bind.TestBind_ADocumentOnlyAnIdentifierReachesIsReached hold both sides of what Sources lists.
• e2e.TestCompile_ASchemaAnExpansionReachesIsValidated compares what a schema draws reached by an expansion and by a $ref.
• e2e.TestCompile_ADynamicRefExpandsInTheDocumentItIsWrittenIn is openapi: a $dynamicRef written in another document is never expanded #1031's example with a two-order diff, and schema.TestDynamicRef_RestsOnNoReadTheMergeBoundCut is openapi: a merge chain past MergeDepthLimit hides an $id, expanding a $dynamicRef across a resource boundary #401's reproducer.
• bind.TestBind_ADocumentNoPassReachesDeclaresNothing holds, in both orders, that a document read as a location before another declared it adds no duplicate and no merge-bound verdict, and bind.TestBind_AReadingNoPassSettlesCountsEveryDocument that the passes end, within a budget, on the input no pass settles. bind.TestDynamicAnchors_SaysWhenTheCountProvesNothing holds that a response past the bound hides no anchor, and bind.TestDynamicAnchors_ANullAnchorNamesNothing the null spellings main pinned in the removed anchors package.
• testdata/openapi/identifiers_in_another_document puts the shapes above where the harness oracles and the sweep reach them. Its source draws two openapi/unresolved-ref errors on main, which TestHarness_InRepoCorpus reports.
• Six more planted defects, each reddening one of the tests above: counting every document read, never falling back to that, falling back on any repeat, a merge past the bound in a document nothing reached, a null $dynamicAnchor read as a name, and a merge past the bound anywhere counted as the document's.
• Sixteen of this PR's own guards planted again, each reddening a named test, among them the read of an extension or a schema keyword in a document with no grammar, the hold on an absolute URI, an anchor in another document where no schema could be built, and content no document places. One survived: the source's walk declaring on every pass, so its own $id and plain names read as duplicates from the second pass on. bind.TestBind_TheSourceDeclaresOnceWhateverThePasses now pins it.
• 35 planted defects across the guards this adds, each confirmed compiling and each reddened by a named test. Survivors along the way showed two lines that changed nothing, since removed, and gaps in the tests, since filled.
• sweep.py against main at 737ffb5 over testdata and morphic-references (2,183 specs), and over testdata with external references allowed (229): only the new fixture differs, its two openapi/unresolved-ref errors gone, Pet and Tag resolving into other.yaml and Leaf's $dynamicRef expanding. That shows no regression, not that the fix works: no corpus spec uses these shapes across documents.
• bind.TestBind_ThePassTheBudgetRunsOutInStands sweeps the budget across a binding whose second pass a declaration forces: one step short of the least budget that begins it, the first pass's table stands with every answer it gave. The code at 9fb547f also ran a pass with nothing to spend after reading held documents for it, its empty table replacing the cut one; a pass now reads what it held itself, so that case cannot arise.
• bind.TestBind_CountsWorkLinearAcrossHeldDocuments counts the binding's steps at 500, 1,000 and 2,000 documents for a chain, a chain spelled relatively, a loop, a chain of mapping values and links naming the one before by its $id, and checks each document is read once and no identifier is. Every shape reddens on the code that began a pass per held document. bind.TestBind_AHeldDocumentMayDeclareAnothersURI pins two documents held together, one declaring the other's URI, in both orders.
• Planted against the continued pass, each reddening a named test: a held position still waiting after its documents are read, a drain blind to a changed URI, a first drain that only grows, an unrecorded held hop, a release keeping what waits, a requeue skipping mapping values, a drain leaving new declarations unanswered, a waiting chain remembered, every drain committing whole, a drain list never cleared, identifyAll rescanning from the first document, and goOn running on a spent budget. Five survive, each with a reason: an unrecorded held mapping value, which a release recomputes too; the bound on releases, which correct code never reaches; reread ignoring a spent budget, as a pass begun with nothing to spend and one begun with a step left both answer nothing; an uncharged identifyAll scan, which its cursor keeps linear; and the budget check in run's rounds, a line as main writes it.
• sweep.py against 1db320a, the head before this change: the 2,181 specs of testdata and morphic-references with default flags, and the 227 of testdata with external references allowed, compile alike, IR, exit codes and diagnostics. Seven generated shapes of held file:// documents compile to the same bytes as on that head, and the three declaring more than one component to the same IR in both declaration orders.
• bind.TestBind_AnIDNamingItsDocumentReturnsToIt holds that an $id naming its own document, below another $id, returns what is below it to that document's resource. It fails on returning to the source's resource, as main does where it reads the source's identifiers alone, and on keeping the outer $id's, neither of which any test caught. A row of bind.TestInOwnResource_ReadsTheResourceAPositionIsIn holds that a position's own $id counts where the grammar reads raw YAML: reading it by the grammar alone expanded a $dynamicRef across that boundary, and nothing caught that either.
• With resolveAt planted to fail for another document's anchors, a $dynamicRef that dynamicExpansion expands lowered to any with no record and no diagnostic, since lowerSchemaBody fell back to the body and recordUnexpandedDynamicRef keeps only one that does not expand. With the fallback gone the same plant fails --verify with ir/type-ref-no-target.
• Generated tangles of two to six documents, named by paths and by URLs, whose schemas write colliding $ids, anchors, $ref chains, discriminator mappings and nested relative $ids, 30,000 of them: the table Bind returns equals that of one fresh pass read against what its last pass reached, and three shuffled declaration orders give each the same answers. 8,409 took two passes or more and 62 counted every document read. A drain blind to a changed URI breaks the first at the seventh tangle.
• bind.TestIdentifiers_AReadingIsTheDocumentsCounted pins what makes the passes return to a reading: the documents counted, which settle every declaration a pass reads against, each read whole before a drain counts it. The key also held the number of declarations read, which answers nothing and only put the fallback to counting every document read off past a reading the passes had already returned to. Without it the input no pass settles takes three passes rather than five, and over 300,000 generated tangles of documents declaring, asking for and passing on a shared pool of URIs every answer is the same, 2,011 reach the fallback either way, and 1,754 reach it in fewer passes. The test fails with the count put back, with a reading never recorded, and with the fallback taken at any change.
• CI's gate on the final head, both jobs, at 538ea34, which holds main at 6996ce4.

Closes #1049
Closes #1031
Closes #401
Closes #454
Closes #405
Closes #1349