Skip to content

build(deps): bump sentencepiece from 0.2.2 to 0.2.3 - #113

Merged
fmueller merged 1 commit into
mainfrom
dependabot/uv/sentencepiece-0.2.3
Nov 20, 2026
Merged

fmueller merged 1 commit into
mainfrom
dependabot/uv/sentencepiece-0.2.3

Conversation

@dependabot

@dependabot dependabot Bot commented on behalf of github Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Bumps sentencepiece from 0.2.2 to 0.2.3.

Release notes

Sourced from sentencepiece's releases.

v0.2.3

Release Notes v0.2.3

This release introduces SentencePiece Lite, a zero-dependency C++20 runtime, new vocabulary-selection algorithms for training (Auto-character Coverage and Sparse L1 Unigram pruning), a Bazel (Bzlmod) build, a large number of security and robustness fixes found by OSS-Fuzz and community reports, and a cleanup of long-deprecated APIs and flags.

1. SentencePiece Lite (New Zero-Dependency Runtime)

  • New lite/ runtime: A compact (~45 KB stripped, < 2k LOC) C++20 tokenizer engine that runs directly on memory-mapped FlatBuffers model binaries (.spm.fb) with no runtime dependency on Protobuf or Abseil (#1330, Commit a2cb4d8). See lite/README.md.

2. Training: New Vocabulary Selection Algorithms

  • Auto-character Coverage (--auto_character_coverage): New TrainerSpec.auto_character_coverage option for both BPE and Unigram that globally optimizes the allocation between single characters and multi-character subwords within the vocabulary budget, while keeping lossless round-trip via byte fallback (Commits e23ece8, df18339, d34a6ce). See doc/auto_character_coverage.md.
  • Sparse L1 Unigram pruning (--use_sparse_pruning): Optional L1-regularized EM vocabulary selection with Post-Lasso debiased refit for the Unigram model, replacing the discrete heuristic pruning (#1295, Commit fd53b5c). Required when --auto_character_coverage is used with Unigram. See doc/sparse_l1_unigram.md.
  • PruneUnreachableSentencePieces: Remove pieces that can never appear in a Viterbi path during Unigram training, and fix the discrete pruning loop order (Commits b292ce5, 59c3b80).
  • Custom pretokenizer callback: Trainer now accepts a std::function<std::vector<std::string>(absl::string_view)> pretokenizer, exposed to Python via the pretokenizer= training argument; pretokenization_delimiter and callback are mutually exclusive (#1287, Commit d1b6696).
  • unk_id = -1 with byte_fallback: Training with --unk_id=-1 is now allowed when --byte_fallback=true; null-byte pieces are treated as UNUSED when byte fallback is enabled (fixes #1308, Commit c2fbf86).
  • spm_eval metrics: Added Bytes/Tok, Model PPL, Empirical PPL, and Coverage metrics (Commit 01305c9).

3. Python Wrapper

  • Pickling: SentencePieceProcessor now preserves encode options and the full instance dict (including subclasses) when pickled (#1349, Commits 4e66c53, 52b1a47).
  • Normalizer fixes:
    • Preserve add_dummy_prefix / escape_whitespaces / remove_extra_whitespaces flags loaded from a model or spec instead of silently forcing them to False (#1337, fixes #1334).
    • Bind SentencePieceNormalizer.LoadFromSerializedNormalizerSpec, fixing AttributeError with normalizer_spec= (#1333).
  • Decode / Encode edge cases:
    • Accept the legacy out_type alias in DecodeIds / DecodePieces (#1336).
    • Fix Decode() of empty input for non-str return types (#1289).
    • Fix encode() with an empty batch and out_type='offset_mapping' (#1320).
    • Fix decoding a batch whose first sequence is empty (#1321).
  • Buffer protocol & integer inputs: Accept 8/16/32/64-bit signed/unsigned integer buffers with saturation clamping and zero-copy lifetime safety; accept general sequences (including tuples) in decode APIs (Commit b3717da).
  • Free-threaded Python: Fix a pretokenizer callback race on free-threaded builds (cp314t) and GIL-safety issues in _TrainFromMap (Commits ca0f388, ee1d3dd). Free-threaded wheels are now tested on 3.14t / 3.15t.
  • Python version support: Requires Python >= 3.10; added Python 3.15 classifier and cibuildwheel 4.3.1 (Commits fa40c16, ab49d75).
  • AIX: Fixed the Python build on AIX (#1359).

4. Performance Optimizations

  • BPE training: Replaced the per-symbol position btree_set with a vector and parallelized initial indexing; removed the 65,535-character sentence length limit by replacing 16-bit packed positions with a Position struct (Commits a44ce92, b3717da).
  • Sentence preprocessing: Parallelized TrainerInterface::LoadSentences and SplitSentencesByWhitespace, with RunBatch-based dynamic load balancing for normalization (Commits a44ce92, d1b6696).
  • Decode: Avoid intermediate allocations and skip redundant StrReplaceAll in Decode() (Commit a44ce92).
  • Unigram: Avoid unnecessary allocations in Lattice::NBest / Lattice::Sample, heap-free score precomputation in BoundedPriorityQueue::Gc, and deduplicated corpus byte counting (Commit ee1d3dd).
  • ParallelEncode / Normalizer: Skip unused SentencePieceText allocations in ParallelEncode overloads and skip alignment bookkeeping when not requested (Commit ee1d3dd).

5. Security & Hardening

  • OSS-Fuzz findings: Bounded hypothesis allocations, agenda shrink rounds, and periodic timeout checks in unigram::Lattice::NBest to fix timeouts/OOM; added piece-ID bounds checks in IdToPiece, GetScore, IsControl, IsUnknown, IsUnused, IsByte (fixes #1342, #1344, #1347, Commit 2e432ca).
  • Lattice bounds: Reject invalid positions/lengths in Lattice::Insert, verify EOS/begin nodes before Viterbi backtrace, and guard against null prev pointers and cycles (Commit fa40c16).
  • Empty-piece guards: Guard empty pieces in PrefixMatcher::PrefixMatch and Unigram PieceToId (#1326, #1325).
  • Charsmap validation: Validate leaf node offsets and safely slice values in DecompileCharsMap; fix Builder::BuildNFC_CFMap and regenerate nfc_cf.tsv (#1329, #1327).
  • Trainer guards: RET_CHECK on zero character/word counts, clamp thread pool size to 1024, fix swap-and-pop bug when removing empty normalized sentences, and fix NaN in Gumbel sampling / thread-local RNG re-seeding (Commits b3717da, 70ed5fc, a44ce92).
  • Lite hardening: Alignment and bounds checks on FlatBuffers models, direct-mapping validation, and endian-swap overflow protection (Commit b3717da).
  • General input validation: Additional edge-case hardening across C++, Lite, and Python (Commit 5d21631).

6. Normalization

  • nmt_nfkc: Normalize U+061C (ARABIC LETTER MARK) to space (#1331).

... (truncated)

Commits
  • eafd99c python: cap build requirement to pybind11<3.1
  • e25c2b7 fix(unigram): restore SplitSentencesByWhitespace() before EM
  • 8c77bef fix: enable OpenMP_C and set CMAKE_C_FLAGS for libsais.c in CMake build
  • 4826c62 Merge pull request #1370 from google/dependabot/pip/dot-github/workflows/requ...
  • 7f0a495 Merge pull request #1371 from google/dependabot/github_actions/github-actions...
  • 48bec13 build(deps): bump the github-actions group with 3 updates
  • ccfa6de build(deps): bump the build-time-deps group
  • e4dda6e python: require pybind11>=2.13 and document load() thread-safety
  • d7510ae fix: do not re-read the input list size in Python batch encode
  • 7bde6cb fix: read batch size once in Python batch decode and reset cached ids on reload
  • Additional commits viewable in compare view

@dependabot dependabot Bot added the dependencies Pull requests that update a dependency file label Nov 20, 2026
@dependabot @github

dependabot Bot commented on behalf of github Nov 20, 2026

Copy link
Copy Markdown
Contributor Author

Labels

The following labels could not be found: python. Please create it before Dependabot can add it to a pull request.

Please fix the above issues or remove invalid values from dependabot.yml.

Copy link
Copy Markdown
Owner

@dependabot rebase

Bumps [sentencepiece](https://github.com/google/sentencepiece) from 0.2.2 to 0.2.3.
- [Release notes](https://github.com/google/sentencepiece/releases)
- [Commits](google/sentencepiece@v0.2.2...v0.2.3)

---
updated-dependencies:
- dependency-name: sentencepiece
  dependency-version: 0.2.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
@dependabot
dependabot Bot force-pushed the dependabot/uv/sentencepiece-0.2.3 branch from b28fcf3 to ece898e Compare November 20, 2026 05:28
@fmueller
fmueller merged commit 1b7f755 into main Nov 20, 2026
4 checks passed
@fmueller
fmueller deleted the dependabot/uv/sentencepiece-0.2.3 branch November 20, 2026 05:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies Pull requests that update a dependency file

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant