Skip to content

feat(flash): a bookmarked PDF places each detected heading by reading order, its section number and the printed table of contents - #661

Open
rejojer wants to merge 15 commits into
mainfrom
feat/flash-bookmark-address-merge
Open

rejojer wants to merge 15 commits into
mainfrom
feat/flash-bookmark-address-merge

Conversation

@rejojer

@rejojer rejojer commented Nov 6, 2026 •

Copy link
Copy Markdown
Member

When a PDF's bookmarks are deep enough to be the frame, the detected headings they lack now land under the right parent far more often. On 312 bookmarked PDFs, hiding bookmarks and checking where their printed headings land: with the deepest bookmark level hidden, 97.7% go under the right parent (main: 77.0%), and 99.7% on a 13-book holdout (main: 91.0%); with 30% of leaf bookmarks hidden, 92.6% (main: 21.1%) and 98.4% (main: 22.8%).

The merge walks the detected headings in reading order with one stack of open sections, and takes each heading's parent from evidence on the page: its section number read as an address against the open bookmarks, the printed table of contents, the detector's own nesting, or a numbering or style climb. With no evidence a heading stays a child of the deepest open bookmark. An independent rewrite of the merge proposed in #391.

Changes

  • merge_bookmark_tree: one reading-order walk. Each bookmark takes its place at its printed heading, else at its /XYZ target on the page (in flash's reading order: the column region the target's x places it in, then height; by height alone without an x), else at its page's start. A heading's parent comes from the first evidence that holds:
    • References/Acknowledgments the detector typed go to the top level when only back matter follows.
    • The open bookmark numbered as the heading minus its last part ("5.3" for "5.3.1") wins over a box title detected above it.
    • The detected parent, while it is open and spans no later bookmark, unless their numbers set them apart ("NOTE 19" detected under "NOTE 18").
    • The section number as an address ("4.3" is the third child of "4"), in its own family (digits, Roman, letters, CJK, "Chapter"/"Part"/"Appendix" labels in any language flash's keyword dictionaries know): a descendant of the nearest open bookmark numbered alike, a fork at one of its ancestors (checked against that numbering's bookmark depths), or a sibling before the next bookmark numbered as its younger sibling, also when those depths disagree. One parser reads every leading number, for addresses, labels, printed-TOC cells and page lines; an enclosed number ("②") is a list style of its own. A bookmark's reading that none of its numbered siblings share is dropped ("V." read as a letter beside "IV." and "VI."). An address a bookmark already holds, or one the next numbered bookmark does not follow, is no evidence.
    • The printed TOC, when it nests the bookmarks it lists the way they nest themselves (all of them, or those under that parent).
    • Else a climb on a numbering prefix, the same style fingerprint, or a print larger in the same face, weight and caps ("EXHIBIT B" after a smaller "8. MISCELLANEOUS"), never past a bookmark numbered in another list style ("2." below "(三)") or unnumbered among siblings numbered so ("3.2." below an "APPENDIX" that follows "VII."), and never past one the heading would leave empty, with no content flash reads between its printed heading and this one ("Keywords" atop the right column beside "Abstract" is a peer). A climb stopped by evidence leaves the heading at the deepest open bookmark, one level below it, unless the bookmarks print that level and never like the heading, numbered like the level it climbed to ("Overall Summary" in bold beside a bookmark whose children print in bold italic), or it is a peer the bookmarks skipped: printed in a style they print only that level in and mostly bookmark, under a title that does not recur, numbered and punctuated like the peer ("Engage Phase" after a bookmarked "Prepare Phase"; not "Model" beside "5.1 Main Results", nor "Subsidiary banks:"); an empty section is read at the bookmark's print, not its target page; one stopped only because the next bookmark's section must stay open keeps the height it reached, beside that bookmark, unless the two are numbered in another list style or depth ("3." is no peer of "1.1.2").
  • A node printed at a bookmark's /XYZ target that starts with the bookmark's title and runs on into a capitalized heading ("Item 1C. Cybersecurity Risk management and strategy") keeps only that heading, in the style of its own line, and the children flash nested under it in that style become its peers; same-titled bookmarks on one page take their numbered prints in order ("7.1.2", "7.2.2"), and a print takes the bookmark it prints exactly over one it only resembles ("Sociology, B.A." over "Sociology, B.S."); a print that adds words to a bookmark's title away from its target is another heading, not the bookmark's copy, and initials are no remark ("Earth Sciences, B.A." below "Earth Sciences").
  • A bookmark's address comes from its printed heading, its title, or the numbered line printing its title on its target page (flash's page text).
  • read_printed_toc (new): the printed TOC from flash's TOC blocks, with label and title cells joined, wrapped entries joined, levels from indent (each indent at the level of the bookmarks listed at it), and PDF pages by the offset the bookmarked entries agree on.
  • read_bookmarks returns each target's x/y, the same on pdfium 4 and 5, its height through _viewport_y_fraction, the function that places flash's headings (rotated pages included), in flash's page frame (its per-page view box, so no page is loaded); called alone, it reads each target page's inherited bounding box.
  • Detection exports _style (flash's block_style_signature), _lines (each line's text and style), _y, _content_follows (real content after the heading on its page before the next one, in reading order), _col (its column region), _type and _prominent on detected nodes, and extract_toc hands the merge each page's column regions; extract_toc strips them after the merge.
  • _normalize_title and _line_key are cached, and the TOC page-offset vote compares a TOC title only with bookmarks whose title ends in the same character. End to end, flash takes 167.9 s on 57 sampled PDFs against main's 164.9 s.

Verification

  • Full suite: 1309 passed, 7 skipped. New tests cover address reading, forks, page-line addresses, younger siblings, the numbered parent over a box title, TOC levels (calibrated by the bookmarks listed) and placement, TOC label cells and wrapped titles, /XYZ targets on one page, across two columns by flash's regions, label words in another language and on a page whose box is inherited, headings numbered in another list style, same-titled bookmarks on one page, a heading glued after a bookmark's title and its same-style peers, a larger print in the same face, a print adding words to a bookmark's title, initials as no remark (alone or glued after a bookmark's title), a print matching one bookmark exactly and another closely, a section emptied or not by the content flash reads between two headings (columns, a label above its heading), a numbered heading beside an unnumbered bookmark, a climb the next bookmark stops (and one numbered unlike it), a climb stopped halfway against how the bookmarks print the level below (and beside numbered bookmarks), a climb stopped halfway beside a peer the bookmarks skipped (and each print that does not vouch for one), a bookmark printed a page after its target, detected nesting the numbers set apart, a bookmark reading unlike its siblings', and back matter.
  • Benchmark above: leave-out runs on 299 corpus and 13 holdout PDFs with FULL-tier bookmarks, each hidden bookmark paired with the print the merge dissolves into it. Prints shared by two same-name bookmarks a page apart are not scored, and one book whose bookmarks nest sections under author names is skipped.
  • read_bookmarks x/y checked on pypdfium2 4.30 and 5.13.

… order, its section number and the printed table of contents

When a PDF's bookmarks are deep enough to be the frame, the detected
headings they lack now land where the page puts them. On 305
bookmarked PDFs, hiding bookmarks and checking where their printed
headings land: with the deepest bookmark level hidden, 96.6% go under
the right parent (main: 77.2%), 99.4% on a 9-book holdout (main:
92.0%); with 30% of leaf bookmarks hidden, 86.5% (main: 21.3%) and
88.7% (main: 22.6%).

Placement:
- One walk in reading order keeps one stack of open sections. Each
  bookmark takes its place at its printed heading, else at its /XYZ
  target on the page (column-aware), else at its page's start.
- A heading's parent comes from the first evidence that holds:
  References/Acknowledgments before back matter go to the top level;
  the open bookmark numbered as the heading minus its last part; its
  detected parent; its section number read as an address against the
  open bookmarks (descendant, fork at an ancestor, or the next bookmark
  numbered as its younger sibling); the printed table of contents, when
  it nests the bookmarks it lists the way they nest themselves; else a
  climb on numbering or same-style evidence.
- A bookmark's address comes from its printed heading, its title, or
  the numbered line printing its title on its target page.
- Detected copies of a bookmark's heading (same title, behind a number,
  letter-spaced, garbled, cut off, or carried on with "(Continued)")
  dissolve into it; recurring labels and overlong titles are pruned
  first.

Plumbing:
- read_bookmarks returns each target's x/y, the same on pdfium 4 and 5.
- read_printed_toc reads the printed TOC from flash's TOC blocks: label
  and title cells joined, wrapped entries joined, levels from indent
  then numbering scheme, PDF pages by the offset the bookmarked entries
  agree on.
- Detection exports _style, _y, _type and _prominent; extract_toc
  strips them after the merge.
- _normalize_title is cached: the TOC page-offset vote compares every
  entry with every bookmark (worst case 2.6 s on a 1,308-page manual
  whose flash run takes 58 s).
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-07T13:11:32.789490Z 55028d6 New commits
🔒 Security Review ✅ Completed 2026-11-06T14:57:11.419076Z 8b46f99 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Comment thread pageindex/flash/main.py
from .classification import is_body_paragraph, detect_header_footer, HeaderFooterContext, mark_watermarks, mark_toc_and_boilerplate, mark_build_steps
from .labels import detect_captions, build_caption_regions, CaptionContext
from .model import Rect, numbering_kind, block_text, deaccented_text, Block
from .model import Rect, numbering_kind, block_text, deaccented_text, Block, dominant_style_of, raw_text_of_line

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8b46f99e07

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +444 to +446
if frame_tokens in (tokens[-tail:],
_title_tokens(_PAREN_TAIL.sub("", title))[-tail:]):
return numeral is not None or _numeral_prefix(title) is None

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Require more than a title suffix before dissolving a heading

When the detector misses a bookmark's own printed heading but finds a distinct unnumbered subsection on the same page, this suffix check can silently delete that subsection. For example, with a bookmark titled Results, a detected Qualitative Results on its target page makes _printed_as return true; dissolve_target then treats it as the bookmark's copy and removes it from the merged outline. The previous merge retained this heading. Require additional evidence that the preceding words are a header or numbering rather than treating every unnumbered title ending in the bookmark title as a duplicate.

Useful? React with 👍 / 👎.

if not any(_BACK_WORDS.search(t) or _APPENDIX_LETTER.match(t)
for t in (flat[ref]["title"], printed)):
return None
break

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Check all remaining sections before promoting back matter

For a PDF containing chapter-local appendices followed by more main content, this break makes the back-matter test succeed after checking only the first qualifying bookmark. For example, bookmarks ordered as Chapter 1, Methods, Appendix A, Proofs, Chapter 2, Experiments, with a detected type-7 References before Appendix A, promote References to the document root even though Chapter 2 follows. This overrides the normal placement and removes the references from their containing section. Continue checking subsequent bookmarks at the relevant level so a later main-content section prevents the promotion.

Useful? React with 👍 / 👎.

Comment on lines +1489 to +1490
i = next((i for i in listed if near(i, node["start_index"])
and _toc_same(toc[i]["title"], node["title"])), None)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Prefer exact-page TOC matches over neighboring-page matches

When the printed TOC repeats a title on adjacent pages, taking the first match within the one-page tolerance can select the wrong occurrence even when an exact-page match exists. For example, a TOC can list Summary on page 3 under Chapter One, then a bookmarked Scope on page 4 with another Summary on page 4 beneath it. A detected Summary on page 4 matches the earlier entry first and is attached directly to Chapter One, rather than to Scope; this reproduces with an otherwise fully trusted TOC and changes the correct placement obtained without the TOC. Rank matches by page distance and reject unresolved ambiguity instead of selecting by TOC order.

Useful? React with 👍 / 👎.

…e compares only titles ending alike, and merge rules that changed nothing are gone

With 30% of leaf bookmarks hidden, 86.9% of the hidden headings now go
under the right parent (was 86.5%), and 89.0% on the holdout (was
88.7%). With the deepest level hidden, nothing changes (96.6% / 99.4%).
On the full bookmark trees, one heading moves. Over the 305 PDFs,
reading bookmarks drops from 22.0 s to 13.8 s and reading the printed
TOC drops from 4.9 s to 1.1 s. The bookmarks and TOCs read stay
identical.

- A section number with a doubled dot ("A..1") read as no number
  instead of crashing.
- read_bookmarks loads each target page's box once, not once per
  bookmark.
- The TOC page-offset vote compares a TOC title only with bookmarks
  whose normalized title ends in the same character.
- Removed, because switching each one off changed no score or raised
  one: label/appendix prefix matching in addresses (families must
  match), the repeated-title back-matter check, the "TOC top level is
  the top level" check, TOC levels by numbering scheme (indent only),
  and a bare 1-2 digit line as a section label (a page number above a
  heading is not its number).

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8282669566

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread pageindex/flash/embedded_toc.py Outdated
Comment on lines +802 to +807
elif len(number) > 1 or word in ("part", "book", "volume", "lecture", "unit"):
if (roman := _roman_number(number)) is None:
return []
parts = [roman]
else:
parts = [ord(number.upper()) - 64]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve Roman readings for single-character chapter numbers

For documents using Roman-numbered chapters, this branch interprets Chapter I as chapter 9 and Chapter V as chapter 22, while interpreting Chapter II and Chapter VI as 2 and 6. This breaks the ordering evidence used to place detected chapters missing from the bookmarks. For example, between bookmarked Chapter I Introduction and Chapter III Methods, a detected Chapter II Background is incorrectly nested inside Chapter I instead of becoming its sibling; the equivalent Arabic-numbered frame places it correctly. Retain a Roman reading for single-character chapter numbers, resolving ambiguity against neighboring bookmarks if necessary.

Useful? React with 👍 / 👎.

Comment on lines +1503 to +1506
if _numeral_prefix(cand_title) is not None and _numeral_prefix(title) is not None:
if _numeral_child(cand_title, title):
return "parent"
return "peer" if _numeral_after(title, cand_title) else "stop"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Honor address ownership in fallback numbering climbs

When a detected local heading reuses an address already owned by a bookmark, free_address() correctly prevents numbered placement, but this fallback still treats the larger numeral as sufficient evidence to promote it. For example, in a FULL frame containing 1. Introduction → 1.1 Background, followed by 2. Methods on page 8, a detected 2. Contributions child of Background on page 3 becomes a document-root section. This also truncates Introduction and Background to page 3; the previous merge retained Contributions under Background. Apply the address-ownership and next-bookmark ordering checks to this fallback too, rather than overriding their rejection with _numeral_after().

Useful? React with 👍 / 👎.

Comment on lines +1628 to +1632
for x in sorted({item["x"] for item in items}):
if prev is None or x - prev > gap:
level += 1
ranks[x], prev = level, x
return [ranks[item["x"]] for item in items]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Account for page-margin shifts when inferring TOC levels

For a multi-page printed TOC with different left margins, such as mirrored book pages, ranking absolute x coordinates across all pages mistakes a margin shift for deeper nesting. This reproduces with six correctly matched bookmarks on the first TOC page at x=0.10/0.15 and unbookmarked root entries Results and Conclusion on the next TOC page at x=0.12: both acquire the last first-page root, Discussion, as their parent. The bookmark checks still mark the TOC trusted, so this incorrect nesting overrides the style-based placement that otherwise keeps both headings at the document root. Account for page-local margin shifts, or withhold TOC placement when the indentation frame cannot be established reliably.

Useful? React with 👍 / 👎.

… peer, and bookmark targets sit in flash's page frame

A style climb now stops at a bookmark numbered in another list style
than the heading: "2." printed in the same face as "(三)" and its
chapter used to climb to the top level, and the four headings after it
followed (人工智能发展报告2024). Label words (Appendix, Exhibit,
Chapter) stay exempt, since "Chapter 3" and "Appendix A" are siblings.
With the deepest bookmark level hidden, 96.9% of the hidden headings
go under the right parent (was 96.6%); nothing else changes, and no
heading moves on the full bookmark trees.

- extract_toc hands read_bookmarks flash's per-page view box and
  /Rotate, the frame its headings' _y uses, so no target page is
  loaded (27 sampled PDFs: 1.09 s -> 0.05 s). Called alone,
  read_bookmarks takes each target page's bounding box, which inherits
  the page tree's boxes; get_cropbox does not (pdfium bug 1786) and
  read an A4 page whose MediaBox sits on /Pages as US Letter.
- _line_key is cached: a 12-page paper with 417 bookmarks re-normalized
  every line of a page for each bookmark on it.
…s in order, and a print at a bookmark's target keeps the heading glued after its title

- Two "Future Perspective" bookmarks on one page (under 7.1 and 7.2)
  both took "7.1.2 Future Perspective" and "7.2.2 Future Perspective",
  so the second opened too early and swallowed headings printed before
  its own. A numbered print now takes the first same-titled bookmark
  no differently numbered print has taken.
- Flash sometimes reads a bookmark's heading and the heading below it
  as one ("Item 1C. Cybersecurity Risk management and strategy"); 153
  such nodes in 16 of 305 PDFs showed under the bookmark with its
  title repeated. A node at the bookmark's /XYZ target that starts with
  the bookmark's title and runs on into a capitalized heading now keeps
  only that heading, which never dissolves back into the bookmark. A
  remark the bookmark's print already takes ("Not applicable.") stays
  absorbed, and a heading below the target keeps its full title
  ("Evaluation Metrics"). 153 -> 25; the rest are not at a target.

On a bench that pairs each hidden bookmark with the print the merge
dissolves into it, 97.2% / 99.4% of the hidden headings go under the
right parent with the deepest level hidden (main: 77.1% / 92.0%), and
87.0% / 89.3% with 30% of leaves hidden (main: 21.3% / 22.7%).
…'s style, and the children flash nested under it in that style are its peers

When flash reads a bookmark's heading and the heading below it as one
block, the merge keeps the second heading on its own (7df96ca), but
flash had nested the block's children under the whole block: "J18ZF
(17.5 Ton)" and "J20ZF (20 Ton)" stayed under "J15ZF (15 Ton)", "Cello"
and "Clarinet" under "Bassoon". Detection now exports each heading
line's text and style as `_lines` (stripped by extract_toc like the
other signals). The split-off heading takes the style of the line after
the bookmark's title, and a child printed in that style is its peer:
flash's nesting no longer holds it. Children in another style ("BSSN
3100 - Private Study: Bassoon") stay where flash put them.

With the deepest bookmark level hidden, 97.4% of the hidden headings go
under the right parent (was 97.2%); nothing else changes, and every
heading that moves on the full bookmark trees is such a peer.
…er print in the same face is a peer, and "Mathematics, B.A." is no copy of "Mathematics"

- read_printed_toc took a contents' levels from its indents alone, so
  a contents whose indents don't line up with the bookmark levels
  nested its entries wrong. Each indent now takes the level most
  bookmarks listed at it hold; an indent no bookmark sits at keeps its
  place between the known ones. With 30% of leaves hidden, 73 more
  holdout headings land right (one PDF), 4 more in the corpus, 1 lost.
- The style climb passes a bookmark whose print differs from the
  heading's in size alone and is smaller by over half a point:
  "EXHIBIT B" after a smaller "8. MISCELLANEOUS" is its peer, not its
  child. Sizes compare only when face, weight and caps agree, so an
  IEEE "Lemma 5" in 10 pt stays under its 8 pt small-caps section.
  Corpus: 7 more right, none lost; 4 headings move, all right.
- The "Earth Sciences" print at the top of a page dissolved into the
  "Earth Sciences, B.A." bookmark that points further down, so the
  degree opened at the department's heading and took its "General
  Information"; and "Mathematics, B.A." dissolved into "Mathematics"
  as if "B.A." were a remark, so the department's opening fell into
  the one before. A print that adds words to a bookmark's title, on
  its page but away from its /XYZ target, is now another heading, and
  initials are no remark. 26 headings move in the SMU catalogs, all
  into their department; nothing moves in the corpus.

On the bench pairing each hidden bookmark with the print the merge
dissolves into it, 97.4% / 99.5% of the hidden headings go under the
right parent with the deepest level hidden (corpus / holdout), and
87.5% / 97.4% with 30% of leaves hidden (was 87.0% / 89.3%; main:
21.3% / 22.7%).
…s the bookmark it prints exactly over a similar one

- "Dedman College Interdisciplinary Programs and Courses Biochemistry,
  B.S." at the college bookmark's target kept "Biochemistry, B.S." as a
  remark, like "Not applicable.", so the degree heading was absorbed
  whenever its own bookmark was missing. A remark's period now ends a
  word, not initials; this replaces the "B.A." rule of ef77c89 and
  covers it.
- A print that resembled several bookmarks on its page took the last
  one, even when an earlier one carried its exact title: "J15ZF (15
  Ton) Side Duct" went to the "J18ZF (17.5 Ton) Side Duct" bookmark,
  and "Sociology, B.A." to "Sociology, B.S.", so "Requirements for the
  Major" printed under B.A. landed under B.S. A print now takes the
  bookmark it prints exactly. On the full bookmark trees 2 headings
  move, both right; none appears or vanishes.

The bench pairs each hidden bookmark with the print the merge dissolves
into it, so the corrected pairing also changes which cases it scores:
99.7% / 97.4% of the hidden headings land under the right parent on the
holdout (was 99.5% / 97.4%); the corpus stays at 97.4% / 87.5%.
…en its heading and the next, and a numbered heading is no peer of an unnumbered bookmark among siblings numbered otherwise

- The merge took a bookmark's section for empty when the next heading
  sat less than 15% of a page below its print, so on a two-column page
  "Keywords" at the top of the right column stayed under "Abstract",
  whose text fills the left one. Detection now exports, per heading,
  whether real content follows it on its page before the next heading
  in flash's reading order (any text block but header, footer or
  watermark, as _heading_appears_at_page_top counts it), stripped by
  extract_toc with the other signals. A section is empty when the
  heading comes before the bookmark's print ("EIGHTEEN" above "LOG
  ANALYSIS") or right after it with nothing between; the vertical gap
  stays for nodes without the signal.
- The gap had also hidden a weak style climb: in an FDA guidance the
  "3.2. S." and "3.2. P." headings print in the face of "APPENDIX" and
  climbed out of it. An unnumbered bookmark now stands for its level's
  numbering, so "3.2." is no peer of an "APPENDIX" whose siblings run
  "I." to "VII.".

On the full bookmark trees 7 headings move, all right: "CCS Concepts"
and "Keywords" leave "Abstract" (5), "Limitations" leaves "Conclusion"
(2); none appears or vanishes. With 30% of leaves hidden, 87.8% of the
corpus's hidden headings land under the right parent (was 87.5%); the
deepest-level runs and the holdout are unchanged.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8e1a320992

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +329 to +331
next_heading = min((later.reading_order_index for later in after if later.type in (7, 8)), default=float("inf"))
return any(later.reading_order_index < next_heading and later.char_count() > 0
and later.type not in (1, 2, 12) # header / footer / watermark

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Identify accepted headings when checking for intervening content

When an outline passes is_outline_valid(), extract_toc() skips mark_outline_block_types(), so ordinary accepted heading blocks can still have type 0 despite having used_as_heading=True. This scan consequently counts an immediately following heading as real content and sets _content_follows=True even when nothing separates the headings. In the bookmark merge, that defeats _claims_empty(): for example, a same-style Setup immediately below the bookmarked Methods heading becomes a root sibling instead of remaining its child. Identify the accepted heading blocks rather than relying exclusively on block types 7 and 8.

Useful? React with 👍 / 👎.

…and label words from flash's keyword dictionaries

- The merge guessed columns three ways: a heading's from reading order
  jumping back up its page by 5%, a bookmark target's from x >= 0.45,
  and an x-less target's from its page's bookmarks jumping back up.
  flash already splits each page into column regions and reads them in
  order. Detection now exports each heading's region (`_col`, stripped
  by extract_toc) and extract_toc hands the merge each page's regions;
  a target sits in the first region not left of its x nor wholly above
  it, and the merge compares region, then height. A target without an
  x has no column to give, so it is placed by height alone (JPM's,
  whose x-less targets a region lookup put in the right column). The
  guesses and _COLUMN_JUMP are gone.
- The merge kept its own label words in five lists (chapter, part,
  appendix ...; references, appendix ...; table, figure ...), each a
  little different, while flash ships multilingual dictionaries for
  them. One label set now comes from flash's chapter, appendix and
  volume words, back matter adds its references, captions its table and
  chart words, each plus the few words flash lacks; the page-line
  reader's own label patterns use it too. "Kapitel 3" reads as chapter
  3, "Anhang A" and "Literaturverzeichnis" as back matter.

With the deepest bookmark level hidden, 97.6% of the corpus's hidden
headings land under the right parent (was 97.4%: four Chinese section
items return to their own 第N节); with 30% of leaves hidden one heading
is lost where flash itself reads the top of the right column before the
bottom of the left. On the full bookmark trees 2 junk headings move and
none appears or vanishes.
… number words come from flash's own definitions

- A bookmark's /XYZ height used its own page-box math, which gave up on
  rotated pages; it now goes through `_viewport_y_fraction`, the
  function that places flash's headings, so targets and headings share
  one frame and a rotated page's target has a height (x stays for
  upright pages, where it is plain).
- Assembly's `_style_of` duplicated flash's `block_style_signature`
  (dominant style plus the caps-heavy flag); the merge's fingerprints
  are now flash's, and the larger-print check reads its format.
- Caption words take flash's figure dictionary and its equation words,
  which keyword_tables now names (`EQUATION_KEYWORDS`) instead of
  burying them in the trie; number words take flash's one to twenty.
  Assembly names the block types that are no page content and the two
  outline heading types once, for `_heading_appears_at_page_top`,
  `_content_follows` and `mark_outline_block_types`.

On the corpus and holdout every bookmark target (22,758), style
comparison and title reading comes out as before; rotated pages and
non-English figure and equation labels ("Abbildung 3:") are what
change.
… number whose depth the bookmarks disagree on falls back to its younger sibling

- Six patterns each read "the number at the start of a line" their own
  way: the dotted numeral of the numbering rules, the section key, the
  CJK enumerator, the label alone on a line, the printed-TOC label cell
  and the page-line reader. The address parser now returns the text
  after the number too (`_lead`), and all six read through it. On the
  corpus a number glued to its CJK title ("4.7多维度…"), a spaced one
  ("A.1. 1" = A.1.1) and a BOM-wrapped one now read; a bare number
  ("319") no longer counts as a heading's; a title may start anyhow
  after an explicit "、" or "."  ("5.1、AI超算…"); an enclosed number
  ("②", "⑵") is a list style of its own instead of the digit NFKC makes
  of it; a lone number still names no label above a heading, where it
  may be a folio.
- With "1、AIGC…" now read, the address rule met a numbering whose
  depth the bookmarks disagree on ("1.1" and "1、" both at level 2) and
  gave up, so "2、" climbed, stopped at "七、产业全景图" and fell back
  under "1、". The depth check failing is no evidence against the
  younger sibling, so the rule now tries it ("4、" names "七").

With 30% of leaves hidden, 88.0% of the corpus's hidden headings land
under the right parent (was 87.7%; the deepest-level runs and the
holdout are unchanged). On the full bookmark trees 2 headings move, both
right ("2.2 Glossary" out of "2.1", a duplicated print dissolved), and
none appears or vanishes.
…height it reached, beside that bookmark, unless the two are numbered in another list style or depth

The halfway fallback to the deepest open bookmark answers a climb that evidence stopped. A climb only the next bookmark's floor stopped had evidence all the way up, so the heading stays as high as the bookmarks allow. Siblings are numbered alike, now by depth as well, so a list item "3." is no peer of the next bookmark "1.1.2". Leave-out bench: corpus S2 88.0% to 88.9% (18 fixed, none broken), S1 and holdout unchanged.
…ks print the level below unlike the heading, and numbers overrule the detector's nesting

The halfway fallback puts a heading one level below the deepest open bookmark. The bookmarks' own copies show how that level is printed; a heading printed unlike every one of them, and numbered like the level it climbed to, is not of that level and keeps the height it reached ("Overall Summary" in bold beside a bookmark whose children print in bold italic). An unnumbered heading beside numbered bookmarks still falls back, so a LaTeX paragraph head printed like its subsection stays inside it. An empty section is now read at the bookmark's print rather than its target page, and a detected parent whose number sets the heading apart ("NOTE 19" under "NOTE 18") is no parent. Leave-out bench: corpus S1 97.6% to 97.7%, S2 88.9% to 90.7%; holdout S2 97.4% to 98.7%; 55 fixed, none broken.
…is dropped, not only an address all of whose readings they don't

A lone "V." reads as Roman 5 and as the letter V. Beside "IV." and "VI." it is Roman, but the letter reading stayed, so a heading "D." after "C." under it failed the address check that every open bookmark numbered in the letter family be its prefix, and fell to the climb, which left it under the last digit-numbered bookmark. Siblings are numbered alike, so each reading is now checked against them, as whole addresses already were. Leave-out bench: corpus S2 90.7% to 90.8% (+3), none broken, no heading moves on the full bookmark trees; holdout unchanged.
…hen they print that level alone in its style and mostly bookmark it, the title does not recur, and it is numbered and punctuated like its peer

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant