Issues

• [v1] meta-issue: required work for integration with transformers

  huggingface/tokenizers#2528 · Filter by author SBrandeis opened on Oct 14, 2026 · v1.0.0

• [v1] Rust: train through a tokenizer's own pipeline

  huggingface/tokenizers#2527 · Filter by author SBrandeis opened on Oct 14, 2026 · v1.0.0

• [v1] Python: expose the trainers and Tokenizer.train_from_iterator

  huggingface/tokenizers#2526 · Filter by author SBrandeis opened on Oct 14, 2026 · v1.0.0

• [v1] Pre-tokenized input

  huggingface/tokenizers#2525 · Filter by author SBrandeis opened on Oct 14, 2026 · v1.0.0

• [v1] Decode a list of token strings

  huggingface/tokenizers#2524 · Filter by author SBrandeis opened on Oct 14, 2026 · v1.0.0

• [v1] Overflowing tokens with stride

  huggingface/tokenizers#2523 · Filter by author SBrandeis opened on Oct 14, 2026 · v1.0.0

• [v1] special_tokens_mask

  huggingface/tokenizers#2522 · Filter by author SBrandeis opened on Oct 14, 2026 · v1.0.0

• [v1] Offsets and word alignment

  huggingface/tokenizers#2521 · Filter by author SBrandeis opened on Oct 14, 2026 · v1.0.0

• [v1] DecodeStream

  huggingface/tokenizers#2520 · Filter by author SBrandeis opened on Oct 14, 2026 · v1.0.0

• [v1] Encoding.tokens, with the original text for unknown pieces

  huggingface/tokenizers#2519 · Filter by author SBrandeis opened on Oct 14, 2026 · v1.0.0

• [v1] Added tokens API

  huggingface/tokenizers#2518 · Filter by author SBrandeis opened on Oct 14, 2026 · v1.0.0

• [v1] Vocabulary lookups

  huggingface/tokenizers#2517 · Filter by author SBrandeis opened on Oct 14, 2026 · v1.0.0
  

• [v1] Serialization: to_str(), save(path), from_str(json)

  huggingface/tokenizers#2516 · Filter by author SBrandeis opened on Oct 14, 2026 · v1.0.0

• [v1] Composition API: build a tokenizer from components, and get/set them

  huggingface/tokenizers#2515 · Filter by author SBrandeis opened on Oct 14, 2026 · v1.0.0

• [v1] Added-token ids differ from 0.23 when added_tokens are listed out of id order

  huggingface/tokenizers#2514 · Filter by author SBrandeis opened on Oct 14, 2026 · v1.0.0

• [v1] A vocabulary containing an empty-string token fails to load

  huggingface/tokenizers#2513 · Filter by author SBrandeis opened on Oct 14, 2026 · v1.0.0

• [v1] Normalized added token at the start of the input gets a stray ▁ with Metaspace

  huggingface/tokenizers#2512 · Filter by author SBrandeis opened on Oct 14, 2026 · v1.0.0

• [v1] Python bindings are built without a regex backend: regex Split / Replace fail

  huggingface/tokenizers#2511 · Filter by author SBrandeis opened on Oct 14, 2026 · v1.0.0

• Concurrent encode calls construct multiple shared worker pools

  huggingface/tokenizers#2499 · Filter by author ZenAlexa opened on Oct 8, 2026

• Don't emit control tokens in untrusted text segments

  huggingface/tokenizers#2498 · Filter by author SBrandeis opened on Oct 8, 2026 · v1.0.0
  3

• Build abi3t wheels for Python 3.15 support

  huggingface/tokenizers#2488 · Filter by author ngoldbaum opened on Oct 3, 2026
  2

• v1.0.0-rc.2: canonicalize_file rejects ByteLevel add_prefix_space=true supported by 0.23.2

  huggingface/tokenizers#2460 · Filter by author michaelfeil opened on Sep 26, 2026 · v1.0.0

• [1.0.0rc2] Possible regressions vs. 0.23.x: Byte atom not found in the vocabulary

  huggingface/tokenizers#2447 · Filter by author xenova opened on Sep 23, 2026 · v1.0.0
  4

  

• [Feature/Security] Support trusted and untrusted special-token spans

  huggingface/tokenizers#2225 · Filter by author kexinoh opened on Jul 21, 2026
  2

• [Security] Load-time process abort when loading a malicious tokenizer.json (BPE merge buffer overrun)

  huggingface/tokenizers#2094 · Filter by author geo-chen opened on Jun 10, 2026
  2