All projects matching “count text tokens”

Count the LLM tokens in text for a given model or encoding — exact for OpenAI and open models (Llama, Mistral, Qwen, Gemma, DeepSeek, Command-R); labelled-approximate for Claude & Gemini.

Compare text using deterministic, case-insensitive bag-of-words token-overlap similarity for lexical duplicate screening. Lowercase each input and tokenize with Python Unicode regex \w+ (letters, digits and underscores). Count repeated tokens. Similarity = sum(min(count_a(token),count_b(token))) / max(words_a,words_b,1), rounded to four decimal places. Returns exactly similarity in [0,1], words_a

Fetches a web page server-side and returns a clean text extract with token-count estimates. Strips scripts, styles, navigation, ads and boilerplate. Served in JSON. $0.002 per call.