tokenizer-checker

OpenVINO Tokenizer Checker

Validates that a HuggingFace tokenizer converts to OpenVINO correctly and produces matching outputs for encoding, decoding, normalization, and GenAI compatibility.

When to Use

• Verify a HuggingFace tokenizer converts to OpenVINO and matches HF outputs
• Check if a newly supported tokenizer works end-to-end with OpenVINO GenAI
• Diagnose which test categories (English, multilingual, emoji, whitespace) fail
• Test normalization steps individually to isolate mismatches

Inputs

The user must provide:

• model_id: HuggingFace model identifier or local path (e.g. zai-org/GLM-4.7)

Optional flags the user may request (pass through to the CLI):

• --trust-remote-code — required for some models with custom tokenizer code
• --subfolder — tokenizer subfolder inside a HuggingFace repo or local model directory (used when tokenizer is in a subfolder)
• --no-detokenizer — skip detokenizer conversion and testing
• --use-sentencepiece-backend — use SentencePiece backend during conversion
• --no-special-tokens — encode without special tokens
• --no-skip-special-tokens — decode keeping special tokens
• --skip-missing-outputs — ignore HF outputs absent in OV result (e.g. token_type_ids)
• --use-fast-false — load the legacy (slow) tokenizer
• --max-length — max length for conversion and HF truncation checks (default: None)

Prerequisites

Activate the Python virtual environment before running any commands.

1. Locate the virtual environment — check for common directories at the repository root: .venv/, venv/, env/. Use list_dir to find it. If none is found, ask the user for its location.
2. Activate based on the current platform:
   • Linux/macOS: source <venv_path>/bin/activate
   • Windows (cmd): <venv_path>\Scripts\activate.bat
   • Windows (PowerShell): <venv_path>\Scripts\Activate.ps1

Procedure

Step 1: Run the tokenizer check

Run from the repository root:

openvino_tokenizers check <model_id> [flags]

This executes:

• [1/5] Load HF tokenizer — downloads and loads the tokenizer via AutoTokenizer.from_pretrained
• [2/5] Convert to OpenVINO — converts to OV tokenizer + detokenizer models
• [3/5] Test against 31 strings — compares HF vs OV encode/decode on English, multilingual, emoji, and edge-case strings
• [4/5] GenAI Tokenizer encode + decode — tests openvino_genai.Tokenizer encode/decode with and without special tokens (skipped if openvino_genai is not installed)
• [5/5] GenAI padding + pair inputs — checks batch padding and pair-input behaviour. For tokenizers-backend tokenizers (PreTrainedTokenizerFast / TokenizersBackend), mismatches are reported as errors and affect the exit code. For other tokenizers, mismatches are reported as warnings only (skipped if openvino_genai is not installed)

[Optional] Step 2: Run the normalization check

Run this step if there are issues in the [3/5] Test against 31 strings step of the previous command:

openvino_tokenizers check_normalization <model_id> [flags]

This executes:

• [1/3] Load HF tokenizer — same as above
• [2/3] Parse normalizer pipeline — extracts individual normalizer steps from tokenizer.json and prints the HF → OV mapping
• [3/3] Test normalizer steps — tests each normalizer step independently, then tests the full stacked pipeline

Step 3: Interpret Results

Both commands print ✓ / ✗ per step and exit with code 0 (all passed) or 1 (any failure).

Pass criteria:

• Exit code 0 for each command
• All test strings matched in step 3 of check
• All normalizer steps matched in step 3 of check_normalization

Failure output includes:

• The input string that failed
• Expected (HF) vs actual (OV) values — token IDs, decoded text, or normalized text
• Shape mismatches, value mismatches, or missing output keys

Step 5 results (padding + pair inputs):

• Batch padding mismatches across different configurations (longest, max_length, left/right padding)
• Pair-input encode mismatches
• For tokenizers-backend tokenizers (PreTrainedTokenizerFast / TokenizersBackend): these are errors that affect the exit code
• For other tokenizers (e.g. SentencePiece-only): these are warnings that do NOT affect the exit code but should be reported

Step 4: Report Results

Provide a structured report to the user:

If all steps pass:

• State that the tokenizer is fully compatible
• Note whether GenAI steps were tested or skipped (if openvino_genai is not installed)
• Note any step-5 warnings if present

If any step fails, build a failure report covering:

1. Which step failed — conversion, tokenizer comparison, detokenizer, GenAI encode/decode, or normalization
2. Which string categories failed — identify patterns:
   • English strings only → basic tokenization issue
   • Multilingual strings → Unicode/encoding issue
   • Emoji strings → multi-byte / surrogate handling issue
   • Empty/whitespace strings → edge-case handling issue
   • All strings → fundamental conversion issue
3. Nature of the mismatch — token ID mismatch, shape mismatch, missing output key, decode mismatch, or normalization mismatch
4. Normalization isolation — if check_normalization identifies a specific normalizer step as the root cause, report which step type (e.g. NFC, Lowercase, Precompiled) and its parameters

Security

• NEVER install any packages. Assume the environment is pre-configured.
• NEVER modify model_id — pass it exactly as provided by the user.
• NEVER call internal Python functions directly — always use the openvino_tokenizers CLI commands.