Tagged articles
tokenizer
27 articles · Page 1 of 1
This article details a three-tier semantic chunking approach for RAG systems using LangChain4j: recursive punctuation-based splitting, tokenizer-aware overlapping windows, and parent-child document mapping with Redis and Qdrant to balance retrieval precision and context completeness.
LangChain4jQdrantRAG
0 likes · 9 min read

This article explains how tokenization works in large language models, why it causes failures like miscounting letters in 'strawberry', details the BPE algorithm, discusses costs like language inequality and security risks, and explores alternatives like byte-level and dynamic tokenization.
BPELLM limitationsNLP
0 likes · 20 min read

The article examines Sarvam AI’s lofty claim to build a sovereign Indian LLM, its $41 million funding, the use of Mistral and DeepSeek foundations, the government‑provided 4096 H100 GPUs, technical breakthroughs like a custom tokenizer, and the ensuing industry debate over copying versus genuine innovation.
DeepSeekIndia AIMistral
0 likes · 12 min read

The article explains how different tokenization strategies (word, character, subword) affect token counts, model cost, context length, and multilingual fairness, and details the engineering trade‑offs of BPE, WordPiece, Unigram, SentencePiece, special tokens, and embedding matrices in large language models.
AIEmbeddingLLM
0 likes · 23 min read

Anthropic’s Claude Sonnet 5 arrives with markedly higher reasoning, tool‑use and programming abilities than Sonnet 4.6, closing the gap to Opus 4.8 while offering a lower price tier, improved safety scores, a new tokenizer that raises token counts, higher rate limits, and mixed developer cost feedback.
AI agentsAnthropicClaude Sonnet 5
0 likes · 10 min read

A fan's test of the idol Ma Jiaqi exposed a large‑language‑model's inability to generate his name, leading to an analysis that links the failure to low‑frequency token degradation, academic papers on frequency‑aware prompting and training, and a confirming tokenizer change by Anthropic.
ACLAnthropicEMNLP
0 likes · 14 min read


Geek LabsMay 6, 2026 · Artificial Intelligence
The article introduces two practical GitHub repositories—how-to-train-your-gpt, a step‑by‑step guide that builds a LLaMA‑style GPT model across 12 chapters, and dictionary-of-ai-coding, a plain‑language glossary of AI‑coding terms—showing how they together provide a complete understanding of modern LLM fundamentals and terminology.
AIAttentionGPT
0 likes · 9 min read

Kronos, an open‑source large model trained on OHLCV data from over 45 exchanges, treats financial time‑series as a specialized language, using a custom tokenizer and a two‑stage Transformer to enable price prediction, market state detection, signal generation, and risk simulation, with easy Hugging Face integration and a live demo for BTC/USDT.
KronosTransformerfinancial time series
0 likes · 6 min read

The article breaks down MiMo-Audio, a next‑token‑prediction‑style large‑scale audio model built on Qwen2, detailing its acoustic front‑end, RVQ tokenizer, patch‑based transformer architecture, streaming capabilities, performance advantages, engineering constraints, and recommended application scenarios.
Audio ModelingQwen2few-shot
0 likes · 9 min read

This article walks through the complete workflow of loading and running the open‑source Qwen3‑8B model, explaining each core file (weights, config, generation config, tokenizer), how the model tokenizes input, applies chat templates, generates responses, and decodes output, all illustrated with code and diagrams.
ModelScopePythonQwen3
0 likes · 16 min read

This article walks through constructing a small large‑language model from the ground up, covering model architecture, tokenization methods, BPE vocabulary building, embedding, positional encoding, attention mechanisms, multi‑head attention, transformer blocks, training pipelines, inference, and sampling strategies, all with runnable Python code.
Deep LearningLLMPython
0 likes · 34 min read

This article reviews a recent study that demonstrates how a highly compressed one‑dimensional tokenizer, using only 32 discrete tokens and gradient‑based test‑time optimization, can generate high‑quality images without training a separate generative model, and explores its methodology, findings, applications, and limitations.
1D tokenizerAI researchImage Generation
0 likes · 10 min read


ArchitectMay 14, 2025 · Artificial Intelligence
This article explains how Qwen3 implements hybrid (fast/slow) reasoning by using the enable_thinking flag in the tokenizer's apply_chat_template method, detailing the underlying Jinja2 chat template, example prompts, the effect of toggling the flag, and design considerations for future autonomous thinking control.
AI ModelChatMLHybrid Reasoning
0 likes · 13 min read

This guide explains how to install, configure, and use the PHP‑based TNTSearch engine, covering its key features, required dependencies, index creation, various search modes, dynamic updates, custom tokenizers, geo‑search, and text classification with practical code examples.
full-text searchgeo-searchsearch engine
0 likes · 9 min read

This article explains how language models evolved from character‑level embeddings to word‑level and finally to sub‑word tokenizers, highlighting the efficiency, vocabulary coverage, and practical engineering challenges of sub‑word segmentation in modern AI systems.
AI FundamentalsLLMSubword Tokenization
0 likes · 8 min read

This article introduces the core concepts of tokenizers and embeddings in large language models, explains how they convert text into numeric IDs and dense vectors, compares different tokenization strategies, and provides practical JavaScript and TensorFlow.js code examples for beginners.
AI FundamentalsJavaScriptLLM
0 likes · 10 min read

This comprehensive guide walks you through every stage of LLM pretraining—from data sourcing, cleaning, and deduplication to tokenizer design, model architecture choices, training framework selection, optimization tricks, and evaluation methods—highlighting common pitfalls and practical solutions for building robust models.
LLM Pretrainingcurriculum learningdata cleaning
0 likes · 34 min read

The Cloud Music team integrated a lightweight on‑device full‑text engine using SQLite FTS5 with a simple tokenizer, replaced JavaScript matching with SQLite’s bm25(), parallelized queries, and cut search latency by 75%, boosting CTR 13% and average playback by 17 seconds while preserving user privacy.
FTS5Performance OptimizationSQLite
0 likes · 15 min read


政采云技术Dec 19, 2023 · Backend Development
This article explains the fundamental concepts of search engine technology—including forward and inverted indexes, tokenizers, stop words, synonym handling, ranking algorithms, and NLP integration—and provides a concise Go implementation with code examples and performance testing.
GoInverted IndexNLP
0 likes · 21 min read

This step‑by‑step tutorial shows how to set up a GPU environment, prepare custom text data, train a tokenizer, configure and train a GPT‑2‑based large language model, test its generation, and run the entire pipeline using only 61 lines of Python code.
AIGPT-2LLM
0 likes · 10 min read

iOS WeChat’s full‑text search was upgraded by selecting SQLite FTS5, creating a VerbatimTokenizer with multi‑level delimiter support, optimizing table formats to cut index size by 30 %, improving batch index updates and parallel search logic, resulting in 40‑60 % faster query latency.
SQLite FTS5WeChatdatabase optimization
0 likes · 26 min read

This article details the 2021 overhaul of iOS WeChat's full‑text search, covering engine selection, segment‑merge optimization, a new VerbatimTokenizer, multi‑level separator support, table schema choices, asynchronous index updates, and extensive performance gains across chat, contacts, and favorites.
FTS5Performance OptimizationSQLite
0 likes · 27 min read


Code DAODec 21, 2021 · Artificial Intelligence
This article explains four Keras utilities—text_to_word_sequence, hashing_trick, one_hot, and Tokenizer—showing how each converts raw text into token lists, hash indices, integer encodings, or document matrices, with code examples and sample outputs.
Kerashashing_trickone-hot
0 likes · 6 min read

This article explains why HTML entities like "<" and ">" can disappear in rich‑text fields, analyzes the underlying tokenizer state machine, and provides a lightweight hack that inserts empty comment nodes to preserve the original text without breaking legacy rendering logic.
HTMLJavaScriptbug fix
0 likes · 12 min read

This article explains why large‑scale Elasticsearch clusters may miss log entries during keyword searches, dives into the fundamentals of inverted indexes and tokenization, and demonstrates practical index‑time and query‑time tokenizer optimizations—including custom analyzers for English and Chinese—to dramatically improve search recall and precision.
ElasticsearchInverted IndexSearch Optimization
0 likes · 13 min read

This article explains how Elasticsearch uses Analyzer components—character filters, tokenizers, and token filters—to perform text analysis, reviews the built‑in analyzers such as standard, simple, stop, whitespace, keyword, pattern, language, ICU and IK, and provides practical _analyze API examples with code snippets and result screenshots.
ElasticsearchICU PluginIK Analyzer
0 likes · 11 min read

This article explains the core components of Elasticsearch's full‑text search analysis—Analyzers, Tokenizers, and Token Filters—detailing their roles, building blocks, built‑in types, and how they combine to customize text processing for effective indexing and querying.
ElasticsearchToken Filteranalyzer
0 likes · 5 min read
