[rejojer]

The FULL-tier bookmark merge in pageindex/flash/embedded_toc.py drops thousands of duplicated headings, files unbookmarked subsections under their own numbered section, puts a heading printed above a bookmark's own heading in the section before it, and moves unbookmarked References out of the last chapter, without cutting any other bookmark section short. On 325 local PDFs (295 with FULL-tier bookmarks), 101 bookmark sections in 100 PDFs end earlier than on main, every one of them a tail section (References, Limitations, Acknowledgments, Index, ...) leaving the last chapter or an unbookmarked peer taking its place; none ends later.

Main grafts every detected heading under the deepest bookmark open at its page and checks only that one bookmark for a duplicate. Duplicates of shallower same-page bookmarks survive (a numbered 5 Experiments inside bookmark Experiments, often with its children), a subsection printed above the next bookmark lands inside that bookmark (7.2.1 under 7.3), an unbookmarked References lands inside Conclusion, so Conclusion runs to the end of the paper, and a heading printed above a bookmark's own heading on the same page lands inside that bookmark (Abstract under Introduction, Acknowledgments under References).

Changes

• Duplicates: a detected heading dissolves into any bookmark starting on its page that it matches (equal, letter spacing aside; apart by a numbering prefix such as 5, IV, Item 7, 第三章, letter-spaced or not (M e tho do logy under 3 Methodology); or garbled at fuzzy ≥ 0.9 under the same section number, letter-spaced too unless the two end in different numbers, as Part I and Part II do), or into one starting on the page before with an equal title. A bookmark that has no heading on its own page also takes its title behind numbering from the next page (9 Conclusion on the page after bookmark Conclusion), while one that has (4.1.1 User factual memory) leaves the next page's I. User factual Memory alone; it still takes a next-page copy under the same section number (a running header, 1.1. WHAT IS A GRAPH? after 1.1 What is a graph?) and its title carried on with a parenthetical (J15ZJ (15 Ton) (Continued)). fix(flash): a running "Table of Contents" link no longer hides headings #577 pinned A. Appendix there as a child because main's looser match (ends-with, fuzzy 0.7) swallowed real subsections; numbering alone keeps them. Its children are then placed one by one. Extra words that are not numbering do not count: 5.2 Ablation Results and Executive Results stay under bookmark Results, and 13.6.1 does not fold into 13.6. The SKELETON tier shares this matcher.
• Printed headings that differ from the bookmark: a bookmark that nothing has matched yet takes the first heading on its page that prints its title after a header (JPMorgan Chase & Co. Consolidated balance sheets; for a numbered bookmark any text before its number, as in Chapter 2 My second chapter or a merged 4 SYSTEM OVERVIEW 4.1 DOT), before a parenthetical (Revenue (dollars in millions)), cut short (Derivatives netting for Derivatives netting and collateral, or without its number: DO THE RIGHT THING FOR CONSUMERS for 2. “Do the Right Thing” for consumers: ...), with words inserted inside it (Line of business and Corporate equity), with a number for its label word (A. Proofs for Appendix Proofs), with a short reference resolved where the bookmark kept a LaTeX label (E. WordNet: Table 1 Visualizations for WordNet: tab:wordnet-results Visualizations), continued in lowercase (Financial Performance Summary for Fiscal 2015, while an enumerator such as a. Nomenclature stays a merged subheading), followed by a short remark (Item 4. Mine Safety Disclosures. Not applicable.), or in other words under the same label and number (Note 26 – Restricted cash ... for Note 26 - Restrictions on cash ..., 2. Defined Terms for 2. Detailed Terms, 2.1 Reasonably long section title for LaTeX's short title 2.1 Short title, a doubled 第十二条 不可抗力 第 十 二 条 ... or 二、 邻里公 约 二 、 邻 里 公 约 ...). Any other line that starts with the whole bookmark title is a merged subheading and stays (Note 18 - Leases Firm as lessee), list items and table rows without a label word never match, and a bookmark takes at most one, so Assets measured at fair value stays under Fair value.
• Reading order: a heading read before the heading of a bookmark that starts on its page (or on the page before, for a bookmark that targets the page above its title) belongs to the bookmark before it, when that one starts there too, and goes ahead of the later bookmark among its siblings. The order comes from the detected tree, not from vertical position, because in two columns the top of the right column is read after the bottom of the left one. 522 headings in 166 PDFs move this way, and none of them changes the length of a bookmark section.
• Numbered subsections: a heading whose number belongs to a section ending on its page (the page shared with the next bookmark) joins that section, so 7.2.1 printed above bookmark 7.3 goes under 7.2.
• Other placement: an unbookmarked heading climbs from its anchor along the open sections on positive evidence that it is a peer. The evidence is a section number outside the candidate's and after it in the same scheme, or the same style fingerprint, with at least 0.15 page heights below a same-page candidate. A climb must reach a numbering parent or the top level (just below the root when one root bookmark titles the whole document, so Authors and Affiliations leaves References). Stopped halfway (missing evidence, an earlier number, a same-style detected heading numbered like it already under the candidate, or a candidate with content after the heading's page), the heading stays at its anchor, because at the halfway level it would cut that level's last section short. Numbered like it means both open with a section number, or both with a 一、-style one, so an unnumbered Acknowledgements still leaves a Conclusion that holds another tail section.
• Closed sections: a bookmark section that a climb passes is closed, and a later heading that anchors in it starts from its nearest open ancestor, so the prompt templates printed after an unbookmarked References no longer go back into Conclusion ahead of it. Detected headings out of reading order drop to 0 (2 in the corpus and 1 in the holdout before).
• Section numbers are at most three digits and end at a break, so 3D, 2.5D, 10-K and 2019 Survey are words.
• Spanning sections: a detected section without a bookmark that runs past the next bookmark still becomes a leaf, but its children numbered directly under it (9.3.2.1 under 9.3.2) stay with it while no bookmark starts in between, instead of landing beside it.
• Recurring sections: the noise filter that drops a title recurring three or more times keeps one printed in the style of a bookmark's own heading and never on adjacent pages, a section each chapter repeats (a textbook's Notes and Study Problems, a 10-K's Loan modifications in each portfolio). On main these were deleted with the running labels; 251 come back in 12 PDFs, 15 of them stray API-documentation labels in one manual.
• Detection exports each heading's style and position as _style/_y, and extract_toc strips them from the result.

Known differences from main

• About 6 headings that a climb would place well follow main's nesting instead: chapter-end Study Questions and Additional resources (three in one book), and mid-level peers such as JPM's Economic Value Sensitivity inside Earnings-at-risk. Unnumbered headings between consecutive numbers (Autoregressive Model between 1.1.1 and 1.1.2) also stay inside the previous subsection, which the missing number supports, and an AIR paper's References stays under 10 Conclusions because its bookmarks put the appendix there too. Every extra signal tried to place them also moves real subsections: keeping halfway climbs for headings in the second half of the section cuts 13 bookmark sections short ((Continued) pages, table captions), and following the detector's own nesting agrees with misplaced JPM regulation subsections and CMC sub-items.
• 4 copies that main folds at its 0.7 fuzzy bar stay as leaves: two form code names (headershealth for Health), where the readable print is worth keeping, other words (timer state for Timer status), and a hyphen plus a qualifier (Non-Operating Income (Expense), Net (dollars in millions) for Nonoperating Income (Expense)). At that bar main also deletes real headings (Daily inspections into Weekly inspections, 16.1.2 into 16.2.2), which this PR keeps.
• A heading without a bookmark printed in the top-level style mid-document climbs to the top level, as a References after Conclusion should: in one university catalog Residence Accommodations leaves Student Life and Housing, and the six housing sections printed after it follow it to the top level. Stopping such a climb when headings in a smaller style follow (the section has subsections) fixes it but puts References back into the last chapter in 3 corpus papers whose appendix follows, so it is not done.

Verification

• 25 tests in tests/test_flash_embedded_toc.py. Each rule above has one that fails when the rule is removed, and main's test_bookmark_title_printed_a_page_after_its_target_is_no_child now expects A. Appendix to fold and pins the bookmark that already has its heading.
• Corpus: detection ran once per PDF, and main and this branch merged the same detected tree. Every bookmark section that ends earlier, every title gone with no copy left, every node this branch keeps that main drops, and every moved subsection was checked by hand.
• Holdout: 10 bookmarked PDFs never used for tuning (textbooks of 640 to 1,100 pages, four university catalogs, a 1,300-page manual). No bookmark section ends later than on main; 3 end earlier (PRML's exercise pages leave 7.2.3, numbered like sections, and the catalog case above); the 5 titles gone are copies of their bookmark; numbered headings outside their numbering parent drop from 132 to 2.
• Full suite: 1221 passed, 7 skipped.

[BukeLy]

Thanks for the contribution. Closing this PR because the relevant indexing behavior has been superseded by the current Flash pipeline, PageIndex File System direction, or the later documentation/assets implementation. The branch is no longer suitable to merge as-is.