Skip to content

Skip fuzzy search for exact normalized matches - #1391

Open
harshitgupta31415 wants to merge 1 commit into
python-babel:masterfrom
harshitgupta31415:harshitgupta31415/skip-fuzzy-search-for-exact-normalized-matches
Open

harshitgupta31415 wants to merge 1 commit into
python-babel:masterfrom
harshitgupta31415:harshitgupta31415/skip-fuzzy-search-for-exact-normalized-matches

Conversation

@harshitgupta31415

Copy link
Copy Markdown

Fixes #1390

When a message only changes capitalization or surrounding whitespace, its fuzzy key can already be in the candidate dictionary. This checks that dictionary first instead of comparing the text with every candidate again.

The existing search gives identical normalized strings a score of 1, so the shortcut picks the same key. The merge code is unchanged: translations, fuzzy flags, comments, previous IDs and obsolete messages still go through the same path. Actual wording changes still use get_close_matches(). No extra cache is added.

For Q exact-normalized lookups among C candidates, this avoids Q × C candidate comparisons and uses Q dictionary lookups instead, after the existing candidate preparation. String normalization still takes time, and nonexact fuzzy matching has the same complexity as before. This does not make every catalog update linear.

This leaves the current normalized-key collision behavior alone; #1330 handles that separately.

Checks:

  • 419 before/after comparisons matched, covering update options, plurals, Unicode, empty normalized text, collisions, comments, flags, previous IDs, obsolete messages and template contents.
  • The full suite ran on both implementations. PR: 7,835 passed; base: 7,834 passed. Both had the same two failures in test_get_timezone_name_misc on this Windows machine (GMT+00:00 versus GMT+05:30), plus 9 skips and 2 expected failures. The base run excludes the new test that specifically checks fuzzy-search call elimination.
  • Ruff and git diff --check pass. The full messages tests pass on both versions.

Benchmark scripts, raw samples and reproduction steps.

All 40 benchmark rows, including controls and slower results

Measurements use Python 3.14.5 on Windows. Each row uses one fixed pair of physical cores; pairs rotate between rows. These Python functions are single-threaded, not parallel workers. Results from different pairs are shown separately because this laptop has different core types.

Each row has five paired timing samples per version (short calls batched), plus five separate tracemalloc measurements. Times and peaks are medians. GC stays enabled. Peaks cover newly traced Python allocations during the call, not total process RAM or a copy of the installed library.

The full workflow parses existing PO bytes, updates from a prepared template, and writes the entire PO. Reading files and preparing the template are outside the measurement. The catalogs are real Django/Wagtail translations; the edits are controlled capitalization, surrounding-whitespace, or punctuation changes, not production traffic.

All output bytes matched. The unchanged and punctuation-change controls do not use the shortcut, so their timing differences are not claimed as improvements. Every measured row is included, including slower results.

Catalog / edits Messages CPU mask Base time PR time Speed Base peak PR peak Memory
Django German: unchanged 348 0xf0 13.823 ms 12.249 ms 1.129× faster 591.587 KiB 591.595 KiB 1.000013× more
Django German: unchanged 348 0x300 55.629 ms 52.144 ms 1.067× faster 591.587 KiB 591.595 KiB 1.000013× more
Django German: unchanged 348 0xc00 49.053 ms 50.224 ms 1.024× slower 591.587 KiB 591.595 KiB 1.000013× more
Django German: unchanged 348 0xf 32.276 ms 28.641 ms 1.127× faster 591.587 KiB 591.595 KiB 1.000013× more
Django German: 1 capitalization edit 348 0xf0 29.950 ms 28.073 ms 1.067× faster 592.126 KiB 591.714 KiB 1.000696× less
Django German: 1 capitalization edit 348 0x300 54.517 ms 54.585 ms 1.001× slower 592.126 KiB 591.714 KiB 1.000696× less
Django German: 1 capitalization edit 348 0xc00 53.022 ms 49.043 ms 1.081× faster 592.126 KiB 591.714 KiB 1.000696× less
Django German: 1 capitalization edit 348 0xf 30.010 ms 25.383 ms 1.182× faster 592.126 KiB 591.714 KiB 1.000696× less
Django German: 20 capitalization edits 348 0xf0 56.788 ms 25.472 ms 2.229× faster 595.118 KiB 593.871 KiB 1.002100× less
Django German: 20 capitalization edits 348 0x300 120.036 ms 55.460 ms 2.164× faster 595.118 KiB 593.871 KiB 1.002100× less
Django German: 20 capitalization edits 348 0xc00 110.024 ms 51.596 ms 2.132× faster 595.118 KiB 593.871 KiB 1.002100× less
Django German: 20 capitalization edits 348 0xf 66.644 ms 30.773 ms 2.166× faster 595.118 KiB 593.871 KiB 1.002100× less
Django German: 20 whitespace edits 348 0xf0 28.975 ms 12.799 ms 2.264× faster 595.118 KiB 593.871 KiB 1.002100× less
Django German: 20 whitespace edits 348 0x300 104.334 ms 40.984 ms 2.546× faster 595.118 KiB 593.871 KiB 1.002100× less
Django German: 20 whitespace edits 348 0xc00 85.013 ms 37.325 ms 2.278× faster 595.118 KiB 593.871 KiB 1.002100× less
Django German: 20 whitespace edits 348 0xf 35.395 ms 13.387 ms 2.644× faster 595.118 KiB 593.871 KiB 1.002100× less
Django German: 5 punctuation edits 348 0xf0 15.188 ms 15.417 ms 1.015× slower 592.829 KiB 592.889 KiB 1.000100× more
Django German: 5 punctuation edits 348 0x300 47.487 ms 47.451 ms 1.001× faster 592.829 KiB 592.889 KiB 1.000100× more
Django German: 5 punctuation edits 348 0xc00 47.463 ms 47.364 ms 1.002× faster 592.829 KiB 592.889 KiB 1.000100× more
Django German: 5 punctuation edits 348 0xf 37.621 ms 42.823 ms 1.138× slower 592.829 KiB 592.889 KiB 1.000100× more
Wagtail French: unchanged 902 0xf0 75.059 ms 73.486 ms 1.021× faster 1488.002 KiB 1488.010 KiB 1.000005× more
Wagtail French: unchanged 902 0x300 146.674 ms 147.000 ms 1.002× slower 1488.002 KiB 1488.010 KiB 1.000005× more
Wagtail French: unchanged 902 0xc00 134.303 ms 141.306 ms 1.052× slower 1488.002 KiB 1488.010 KiB 1.000005× more
Wagtail French: unchanged 902 0xf 105.319 ms 81.885 ms 1.286× faster 1488.002 KiB 1488.010 KiB 1.000005× more
Wagtail French: 1 capitalization edit 902 0xf0 91.610 ms 91.494 ms 1.001× faster 1489.104 KiB 1488.127 KiB 1.000656× less
Wagtail French: 1 capitalization edit 902 0x300 153.530 ms 152.039 ms 1.010× faster 1489.104 KiB 1488.127 KiB 1.000656× less
Wagtail French: 1 capitalization edit 902 0xc00 140.547 ms 139.960 ms 1.004× faster 1489.104 KiB 1488.127 KiB 1.000656× less
Wagtail French: 1 capitalization edit 902 0xf 86.034 ms 89.647 ms 1.042× slower 1489.104 KiB 1488.127 KiB 1.000656× less
Wagtail French: 20 capitalization edits 902 0xf0 307.301 ms 72.262 ms 4.253× faster 1493.002 KiB 1490.275 KiB 1.001830× less
Wagtail French: 20 capitalization edits 902 0x300 589.803 ms 172.948 ms 3.410× faster 1493.002 KiB 1490.275 KiB 1.001830× less
Wagtail French: 20 capitalization edits 902 0xc00 520.951 ms 127.837 ms 4.075× faster 1493.002 KiB 1490.275 KiB 1.001830× less
Wagtail French: 20 capitalization edits 902 0xf 314.200 ms 76.707 ms 4.096× faster 1493.002 KiB 1490.275 KiB 1.001830× less
Wagtail French: 20 whitespace edits 902 0xf0 309.306 ms 96.431 ms 3.208× faster 1493.002 KiB 1490.275 KiB 1.001830× less
Wagtail French: 20 whitespace edits 902 0x300 539.413 ms 137.014 ms 3.937× faster 1493.002 KiB 1490.275 KiB 1.001830× less
Wagtail French: 20 whitespace edits 902 0xc00 529.044 ms 137.946 ms 3.835× faster 1493.002 KiB 1490.275 KiB 1.001830× less
Wagtail French: 20 whitespace edits 902 0xf 294.051 ms 74.587 ms 3.942× faster 1493.002 KiB 1490.275 KiB 1.001830× less
Wagtail French: 5 punctuation edits 902 0xf0 125.129 ms 115.699 ms 1.082× faster 1490.213 KiB 1490.278 KiB 1.000044× more
Wagtail French: 5 punctuation edits 902 0x300 258.087 ms 265.939 ms 1.030× slower 1490.213 KiB 1490.278 KiB 1.000044× more
Wagtail French: 5 punctuation edits 902 0xc00 206.512 ms 211.068 ms 1.022× slower 1490.213 KiB 1490.278 KiB 1.000044× more
Wagtail French: 5 punctuation edits 902 0xf 122.132 ms 118.546 ms 1.030× faster 1490.213 KiB 1490.278 KiB 1.000044× more

Base commit: e2a1791. Source, runner and fixture hashes and every raw sample are in the linked JSON.

Fixtures (downloaded from these pinned sources; not copied into the benchmark gist):

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Skip fuzzy search for exact normalized matches

1 participant