sqlite-tokenizer-ar

Arabic search for SQLite FTS5, shipped as a native tokenizer extension, official SQLite WASM bundle, and optional Lucene-style query helpers.

The core extension exposes an FTS5 tokenizer named sqlite_tokenizer_ar. It is designed to match the Apache Lucene Arabic analysis pipeline for tokenization, Arabic normalization, stopwords, digit folding, and light stemming, while running inside SQLite.

Reference assets are pinned to Lucene 9.9.0 for reproducibility. The ArabicAnalyzer pipeline was source-diffed against Lucene 10.4.0 on 2026-06-07 with no behavior changes found.

Caution

Disclaimer: This project was developed with heavy use of AI assistance. Treat it as an experiment rather than a stable product.

Repository Layout

• tokenizer/: loadable SQLite extension written in C.
• query_compat/: optional Python CLI/reference layer for Lucene-style query behavior; runtime analysis, parsing, planning, ranking, and highlighting primitives live in C UDFs.
• playground/: browser demo source using official SQLite WASM built with the tokenizer extension.
• tests/fixtures/: small public fixtures for smoke tests. Large/private corpus fixtures are intentionally excluded.
• ingester/sql/001_canonical_schema.sql: minimal schema fixture used by query compatibility tests.

Install

Release archives provide the loadable extension for Linux x64/arm64, macOS arm64, and Windows x64:

Platform	Release archive	Installed library
Linux x64	sqlite-tokenizer-ar-linux-x64.tar.gz	sqlite_tokenizer_ar.so
Linux arm64	sqlite-tokenizer-ar-linux-arm64.tar.gz	sqlite_tokenizer_ar.so
macOS arm64	sqlite-tokenizer-ar-macos-arm64.tar.gz	sqlite_tokenizer_ar.so
Windows x64	sqlite-tokenizer-ar-windows-x64.tar.gz	sqlite_tokenizer_ar.dll

Install the correct archive automatically with Mise:

[tools."github:yshalsager/sqlite-tokenizer-ar"]
version = "0.1.15"
asset_pattern = 'sqlite-tokenizer-ar-{{ os() }}-{{ arch() }}.tar.gz'

mise install
export SQLITE_TOKENIZER_AR_EXTENSION="$(mise where github:yshalsager/sqlite-tokenizer-ar)/sqlite_tokenizer_ar.so"

On Windows PowerShell:

mise install
$env:SQLITE_TOKENIZER_AR_EXTENSION = "$(mise where github:yshalsager/sqlite-tokenizer-ar)\sqlite_tokenizer_ar.dll"

SQLite must have FTS5 and loadable-extension support enabled.

Python

Install from PyPI with Python 3.10+:

pip install sqlite-tokenizer-ar
# Or with uv:
uv add sqlite-tokenizer-ar

Prebuilt wheels support Linux x64/arm64 (glibc 2.28+), macOS arm64 (11+) and Intel (10.13+), and Windows x64.

import sqlite3
import sqlite_tokenizer_ar

db = sqlite3.connect(':memory:')
sqlite_tokenizer_ar.load(db)
db.execute("CREATE VIRTUAL TABLE docs USING fts5(body, tokenize='sqlite_tokenizer_ar')")

load(db) enables extension loading temporarily and disables it afterward, including on failure. Call it for each connection. loadable_path() returns the bundled library's full path for other SQLite clients. Python's SQLite must support FTS5 and extension loading; the wheel does not replace SQLite.

Building and Publishing Wheels

To build a wheel locally after compiling the native extension:

# macOS arm64; on Linux use a matching manylinux build container instead.
MACOSX_DEPLOYMENT_TARGET=11.0 mise run tokenizer:build
SQLITE_TOKENIZER_AR_BINARY=tokenizer/build/sqlite_tokenizer_ar.so \
  WHEEL_PLATFORM=macosx_11_0_arm64 mise x uv -- uv build --wheel
mise x uv -- uv run --isolated --no-project --with dist/*.whl \
  python -I tests/test_python_wheel.py

The platform tag must match the compiled binary: Linux wheels are built inside manylinux_2_28 containers, and macOS builds set MACOSX_DEPLOYMENT_TARGET to 11.0 for arm64 or 10.13 for Intel. Packaging rejects macOS binaries that require a newer OS than their wheel tag claims. Only wheel distribution is supported; no source distribution is published.

Desktop and wheel builds use pinned SQLite headers; no SQLite runtime is bundled.

Packaging follows sqlite-dateutil: a native SQLite library plus a small connection loader, tagged py3-none-<platform> rather than tied to a CPython ABI. Hatchling handles wheel metadata and file hashes; uv builds, tests, and publishes.

Tag pushes automatically build, test, and publish all five wheels to PyPI; pull requests and manual runs only build and test. Before pushing a release tag, configure a PyPI trusted publisher for repository yshalsager/sqlite-tokenizer-ar, workflow python-wheels.yml, environment pypi. The v<version> tag, pyproject.toml, and package.json versions must match. Publishing checks PyPI for matching existing files so reruns can skip them.

Build From Source

mise run build

This produces:

tokenizer/build/sqlite_tokenizer_ar.so

Load it in SQLite:

.load ./tokenizer/build/sqlite_tokenizer_ar
CREATE VIRTUAL TABLE docs USING fts5(body, tokenize='sqlite_tokenizer_ar');
INSERT INTO docs(body) VALUES('قُرْآن كريم'),('هذه كتابها مفيد');
SELECT rowid, body FROM docs WHERE docs MATCH 'قران';

Tokenizer Features

• Standard-style UTF-8 token segmentation for Arabic, Latin, and digits.
• ASCII lowercase for mixed Arabic/Latin text.
• Arabic and Persian digit folding to ASCII digits.
• Lucene Arabic stopword filtering.
• Arabic normalization: diacritics and tatweel stripping, alef/hamza normalization, dotless yeh to yeh, and teh marbuta to heh.
• Lucene-style Arabic light stemming.
• Tokenizer options for custom stopwords, stopword disabling, and stem exclusions.
• Optional Unicode Arabic honorific expansion, including ﷺ, ﷻ, ﵀, and related ligatures.
• Analyzer-aware source spans and highlighting, including full stemmed words and one-glyph honorific sources.
• Helper UDFs for analysis, normalization, strict-form checks, wildcard/fuzzy matching, query planning, ranking, and snippets.

See the tokenizer reference for tokenizer arguments, UDF signatures, static registration, and integration examples.

Query Compatibility Layer

SQLite FTS5 provides indexing, MATCH, native prefix queries, BM25, highlight(), and snippet(). The compatibility layer adds behavior FTS5 does not provide by itself:

• Boolean query parsing.
• Prefix, suffix, wildcard, and fuzzy expansion.
• Strict versus relaxed Arabic-form matching.
• Field routing for page/title style schemas.
• Lucene-style scoring helpers and deterministic result ordering.
• Snippet/highlight helper paths.

The schema-independent sqlite_tokenizer_ar_plan_query_json() C UDF returns MATCH templates plus unresolved corpus-dependent expansion descriptors. sqlite_tokenizer_ar_execute_query_json() is only for the repository's canonical query-compat schema. Keep application filtering, pagination, and custom result SQL outside it.

Use the compatibility layer when you need Lucene-style query syntax. Use the tokenizer directly when Arabic analysis plus native SQLite FTS behavior is enough. See the query compatibility reference.

Test

mise run test

The public test lane uses small synthetic fixtures. Full corpus parity runs are kept outside this repository because they depend on large third-party/private datasets.

WASM Playground

Public demo

The playground uses official SQLite WASM built with this extension as an extra init module:

SQLITE_SRC_DIR=/path/to/sqlite-source-tree mise run playground:build-wasm
mise x -- python -m http.server 8080

Open http://localhost:8080/playground/.

GitHub Pages publishes the playground from main using the latest release WASM assets plus the public fixture JSONL files.

WASM npm Package

Release builds also pack the generated WASM bundle as:

@yshalsager/sqlite-tokenizer-ar-wasm

The SQLite release is pinned in mise.toml as SQLITE_TOKENIZER_AR_SQLITE_REF. On Renovate's SQLite PRs, sqlite-headers.yml refreshes both vendored headers and their hashes, commits the result to the same branch, and explicitly dispatches public tests, five wheel builds, runtime compatibility, and WASM checks for that commit. Version checks block merging stale headers. Renovate automerges minor/patch updates after CI passes; no extra App or secret is needed.

The compatibility matrix tests historical SQLite runtimes plus the current ref using the same extension binary. For manual SQLite bumps, run mise run sqlite:update, then mise run test, and commit the generated vendor/sqlite/ changes.

It exports stable asset URLs:

import {sqliteWasmUrls} from '@yshalsager/sqlite-tokenizer-ar-wasm'

For Node/Vitest:

import sqlite3InitModule from '@yshalsager/sqlite-tokenizer-ar-wasm/node'

Install the public package tarball directly from the release without npm authentication:

npm install https://github.com/yshalsager/sqlite-tokenizer-ar/releases/download/v0.1.15/yshalsager-sqlite-tokenizer-ar-wasm-0.1.15.tgz

For apps that need fixed public paths, copy package dist/* to public/sqlite-wasm/.

Android Native Artifacts

Release builds also attach sqlite-tokenizer-ar-android.zip, built with Android NDK 29.0.14206865 for:

• arm64-v8a/libsqlite_tokenizer_ar.so
• armeabi-v7a/libsqlite_tokenizer_ar.so
• x86_64/libsqlite_tokenizer_ar.so

The Android API level and ABI list are pinned in mise.toml as SQLITE_TOKENIZER_AR_ANDROID_API and SQLITE_TOKENIZER_AR_ANDROID_ABIS.

These are loadable SQLite extensions. The Android app still needs a SQLite runtime with FTS5 and extension loading enabled, or a custom SQLite build that registers the tokenizer directly.

iOS Native Artifact

Release builds attach a static XCFramework:

sqlite-tokenizer-ar-ios.xcframework.zip
└── SQLiteTokenizerAr.xcframework

It includes ios-arm64 and ios-arm64_x86_64-simulator slices. Link it with Apple system libsqlite3, then register the tokenizer once per SQLite connection before creating/querying FTS tables:

sqlite_tokenizer_ar_register(db);

Minimal local podspec shape:

s.vendored_frameworks = 'SQLiteTokenizerAr.xcframework'
s.libraries = 'sqlite3'
s.ios.deployment_target = '15.0'

Current Scope

The tokenizer is the stable core product. The query compatibility layer is useful but broader: it includes planner, scorer, parser, and snippet helpers that are intentionally outside tokenizer responsibilities.

See THIRD_PARTY_NOTICES.md for attribution and licensing notes.