Tagged articles
tokenization
89 articles · Page 1 of 1
Former Tencent AI Lab director Shi Shuming introduces CNF-MCE, a tokenizer replacing BPE's greedy merging with counting-based vocabulary construction and min-cost encoding, achieving 26-60% compression improvements over BPE while maintaining model performance on most benchmarks.
BPECNF-MCEcompression efficiency
0 likes · 9 min read

This article explains how tokenization works in large language models, why it causes failures like miscounting letters in 'strawberry', details the BPE algorithm, discusses costs like language inequality and security risks, and explores alternatives like byte-level and dynamic tokenization.
BPELLM limitationsNLP
0 likes · 20 min read

Google's 32-author survey on tokenization reveals it as the most underestimated component in LLMs, showing 94% of models reuse identical tokenizers, a 54.3x cost disparity between English and low-resource languages, and that BPE dominates generative models while fundamental fairness and efficiency challenges remain largely unsolved.
BPEHugging FaceSurvey
0 likes · 10 min read

The article presents a practical LLM security gateway that replaces phone numbers and emails with per-request random tokens, blocks credentials like keys and tokens, and restores original values only for authorized callers, with a runnable FastAPI implementation and test suite.
FastAPILLM security gatewayPII masking
0 likes · 21 min read

This article demystifies twelve essential AI terms—LLM, tool, MCP, script, prompt, skill, token, context window, RAG, loop, harness, and agent—using a workplace analogy to show how each component transforms a language model into a functional AI assistant.
AgentLLMMCP
0 likes · 10 min read

Tokens are the basic units AI models process, and the context window limits how many tokens can be handled in a single request; the article explains tokenization, differences for Chinese, the impact on cost, and engineering tricks like sliding windows, map‑reduce, and recursive summarization to manage long texts.
Chinese NLPCost Optimizationcontext window
0 likes · 10 min read

The article explains how different tokenization strategies (word, character, subword) affect token counts, model cost, context length, and multilingual fairness, and details the engineering trade‑offs of BPE, WordPiece, Unigram, SentencePiece, special tokens, and embedding matrices in large language models.
AIEmbeddingLLM
0 likes · 23 min read

Large language models generate text autoregressively from flat token streams, but real‑world conversations require structured roles, system prompts, and multi‑turn history, so Hugging Face’s Jinja2‑driven chat templates serialize these elements, handle BOS/EOT tokens, enforce role alternation, and provide debugging tricks across Meta‑Llama‑3.1, Qwen2.5, and Mistral models.
ChatTemplateDebuggingJinja2
0 likes · 16 min read

The article breaks down the exact five‑stage pipeline—data collection, pre‑training, supervised fine‑tuning, reward modeling, and reinforcement learning—that transforms raw internet text into powerful LLMs like GPT and Claude, and explains how understanding each step lets you build a miniature version yourself.
ClaudeGPTLLM
0 likes · 15 min read

This article walks through the essential steps of turning raw natural language into machine‑readable numbers, covering categorical vs. numerical features, one‑hot encoding of categorical data, tokenization, building vocabularies, and using word embeddings, illustrated with an IMDB sentiment‑analysis example in Keras.
IMDB sentiment analysisKerasNLP
0 likes · 7 min read

This article walks through the four core steps of Chinese NLP corpus preparation—collecting data, cleaning it with regex and encoding detection, tokenizing using dictionary‑based or statistical methods such as jieba, HMM and CRF, and finally standardizing with stop‑word removal, vocabulary building and one‑hot encoding—while illustrating each step with concrete code snippets and practical examples.
CRFChineseNLP
0 likes · 12 min read

The article introduces Natural Language Processing as an AI subfield, outlines its four main task categories—classification to sequence, sequence to classification, synchronous and asynchronous seq‑to‑seq—and details the typical preprocessing pipeline including corpus collection, cleaning, tokenization, stemming, lemmatization, POS tagging, NER, and chunking.
NLPPreprocessingTask Types
0 likes · 3 min read

This guide introduces Stanford CoreNLP, a Python interface for fundamental NLP tasks such as tokenization, part‑of‑speech tagging, named‑entity recognition, constituency and dependency parsing, showing installation steps, model download links, and example outputs.
NLPNamed Entity RecognitionPOS tagging
0 likes · 4 min read

The article explains how Byte Pair Encoding (BPE) works as a greedy, linear‑time subword segmentation technique, walks through its step‑by‑step token merging process with a concrete sentence example, discusses its strengths in handling OOV words, and outlines its limitations and alternatives such as WordPiece and SentencePiece.
BPENLPSentencePiece
0 likes · 8 min read

This article explains ten core AI concepts—including tokens, embeddings, attention, the Transformer architecture, large language models, hallucination, temperature, context windows, Retrieval‑Augmented Generation, and AI agents—so developers can understand model behavior, avoid common pitfalls, and build reliable AI applications.
AI AgentsAI FundamentalsAttention
0 likes · 15 min read

LLaVA‑OneVision‑2.0 replaces uniform frame sampling with a codec‑stream visual unit, integrates a OneVision‑Encoder that tokenizes video as state‑plus‑incremental evidence, and demonstrates consistent gains on 18 video, 11 spatial‑reasoning and 4 tracking benchmarks while open‑sourcing its model, data and code.
JumpScoreLLaVA-OneVision-2.0codec stream
0 likes · 17 min read

This article walks through the full LLM pipeline—from tokenization and embedding to transformer attention, pre‑training, instruction tuning, RLHF, inference, context windows, Retrieval‑Augmented Generation and agents—explaining each component, its role, and the engineering challenges such as hallucination and safety.
AttentionEmbeddingLLM
0 likes · 17 min read

The article examines why the newly AI‑powered Google Search fails at simple letter‑count questions like “how many P’s are in Google,” tracing the issue to token‑based language models, illustrating it with examples, and discussing both short‑term prompts and long‑term architectural solutions such as byte‑level models.
Google SearchJagged IntelligenceLLM
0 likes · 13 min read

The article analyzes how large language models process only tokenized text, compares the traditional LLM‑plus‑toolchain pipeline with emerging multimodal models, evaluates their cost, speed, controllability, and hallucination risks, and proposes a hybrid architecture that matches each approach to specific document scenarios.
LLMRAGToolchain
0 likes · 16 min read

The article demystifies large language models by explaining their core task of next‑token prediction, tokenization, vector semantics, Transformer attention, massive training, hallucination risks, prompt design, and tool integration, showing how these mechanisms work together and why verification is essential.
AI HallucinationAgentAttention
0 likes · 14 min read

Tokens are the smallest language building blocks that AI models process, representing characters, words, subwords, punctuation or emojis; they determine context window size and generation speed, so tokenization directly impacts model understanding accuracy and efficiency, as explained in the 2026 Token Report.
AI FundamentalsModel Efficiencycontext window
0 likes · 4 min read

The article explains that a token is the smallest semantic unit processed by large language models, describing how text is split into tokens, mapped to IDs, converted to vectors, and how token counts affect context windows, costs, and the distinction between characters, words, and tokens.
AI Fundamentalscontext windowcost
0 likes · 4 min read

This article explains why large‑model prompts can yield inconsistent answers, examines the roles of temperature, top‑p/top‑k, tokenization, context windows, position bias, and model randomness, and provides a step‑by‑step debugging workflow and production‑grade best‑practice checklist to achieve stable outputs.
DebuggingLLM stabilityprompt engineering
0 likes · 13 min read


Geek LabsApr 20, 2026 · Artificial Intelligence
This open‑source tutorial breaks down large language model internals into 11 detailed topics—covering BPE tokenization, attention mathematics, backpropagation, transformer architecture, KV‑Cache, Paged and Flash Attention, and frontier techniques—each with numeric derivations and Python code, making it ideal for developers and interview preparation.
AttentionFlash AttentionInference Optimization
0 likes · 5 min read

Anthropic’s Claude Opus 4.7 launches with enhanced handling of complex, long‑running tasks, higher‑resolution visual analysis, stricter instruction compliance, improved benchmark scores, expanded file‑system memory, new effort levels (xhigh), API task‑budget beta, reinforced security measures, and migration guidance on tokenization and prompt adjustments.
AI ModelAnthropicClaude Opus
0 likes · 4 min read

The article defines tokens (now officially called “词元”), explains why large language models require numeric input, and details three main tokenization strategies—word‑based, character‑based, and subword—along with the sub‑methods BPE, WordPiece, and Unigram, highlighting their advantages and drawbacks.
BPELLMSubword
0 likes · 6 min read

This article provides a comprehensive, engineering‑focused breakdown of large language models, covering their Transformer roots, tokenization, context windows, prompt engineering, tool integration via MCP, and autonomous agents, while offering practical examples and actionable insights for developers.
AI FundamentalsAgentLLM
0 likes · 10 min read

The article explains what tokens are, how different models tokenize text, the role of token embeddings, positional encoding, self‑attention, KV cache, and why output tokens cost far more than input tokens, while also covering pricing differences and prompt‑caching savings across major LLM providers.
KV CacheLLM pricingPrompt Caching
0 likes · 13 min read

The 2026 CB Insights FinTech report forecasts nine major trends—including new‑bank entrants, BNPL giants entering banking, Robinhood’s super‑app shift, crypto firms targeting banks, stablecoin‑driven AI payments, and the rise of trusted‑data prediction markets—offering a data‑driven roadmap for industry players and users alike.
AI AgentsBNPLbanking
0 likes · 10 min read

The article analyzes why text models naturally combine understanding and generation, explains the fundamental conflicts that prevent images from sharing the same tokenization, and details LongCat-Next’s discrete autoregressive approach—using SAE visual encoders, residual vector quantization, and a unified LLM backbone—to achieve a single model that can both comprehend and create multimodal content.
LongCat-NextMultimodal ModelsRVQ
0 likes · 21 min read

The LongCat-Next model from Meituan introduces a native multimodal architecture that uses discrete tokenization for vision and audio, achieving unified understanding and generation across modalities while delivering state‑of‑the‑art benchmark performance and simplifying training pipelines.
AIBenchmarkMeituan
0 likes · 11 min read

This article examines why encrypted data hampers fuzzy queries, categorizes three implementation approaches—from naïve in‑memory decryption to conventional token‑based indexing and advanced algorithmic schemes—evaluates their performance, storage overhead, and security trade‑offs, and provides practical references.
Securityfuzzy searchtokenization
0 likes · 10 min read

The article explains that a token is the smallest discrete text unit a large language model processes, detailing why tokenization is essential, how tokenizers work, how tokens flow through the transformer, and how token counts affect context windows, cost, latency, and overall model behavior.
EmbeddingLLMTransformer
0 likes · 20 min read

GLM-4-Voice is an end-to-end Chinese-English speech dialogue model that aligns discrete speech tokens with GLM-4-9B, uses VQ-based tokenization at 12.5 token/s, supports emotion, tone, speed and dialect control, and offers streaming inference with low latency, while detailing its architecture, advantages, limitations and suitable use cases.
Flow MatchingGLM-4-Voicelow-latency streaming
0 likes · 10 min read

This guide walks you through building, training, and fine‑tuning a Transformer‑based large language model entirely from scratch using PyTorch, covering tokenization, self‑attention, multi‑head attention, positional encoding, model architecture, data preparation, training loops, and fine‑tuning on custom lyrics.
GPTLLMPyTorch
0 likes · 43 min read

This article explains the three evolutionary stages of AI—from large language models that generate text, through workflow‑enhanced systems using retrieval‑augmented generation, to fully autonomous agents capable of self‑directed decision‑making—while detailing the four core technologies that power each stage.
AI evolutionAgentEmbedding
0 likes · 9 min read


ArchitectDec 15, 2025 · Artificial Intelligence
This comprehensive guide explains the fundamentals of large language model (LLM) architectures, covering the original Transformer, tokenization, embeddings, positional encoding, attention mechanisms, feed‑forward networks, layer stacking, a step‑by‑step translation example, and the latest open‑source and hybrid LLM designs shaping the field.
AttentionEmbeddingLLM
0 likes · 41 min read

This article walks through the complete workflow of AI large language models, from turning user queries into token matrices via tokenization and embedding, through the Transformer’s self‑attention and multi‑head mechanisms, to decoding logits into human‑readable text, while also covering position encoding, long‑context strategies, generation parameters, and practical engineering tips.
Inference OptimizationSelf-AttentionTransformer
0 likes · 29 min read

This episode explains that a token is the smallest text chunk an LLM processes—ranging from characters to subwords—covers why subword tokenization avoids vocabulary explosion, compares token counts across languages, describes the computational cost of sequential generation, and introduces visual tokens for multimodal models.
AI FundamentalsSubwordlarge language models
0 likes · 7 min read

AION-1, developed by a consortium including UC Berkeley, Cambridge and Oxford, is the first large‑scale multimodal foundation model for astronomy that unifies images, spectra and catalog data via an early‑fusion backbone, achieving zero‑shot and linear‑probe performance that rivals or surpasses task‑specific models across diverse scientific tasks.
astronomycross‑modal generationfoundation model
0 likes · 18 min read

This article explains how Spring AI handles prompts, covering the Prompt and Message APIs, role-based message types, PromptTemplate creation, custom template renderers, tokenization basics, and practical code examples to help developers craft effective AI prompts.
AI IntegrationJavaMessage
0 likes · 16 min read

This article breaks down five fundamental concepts of large AI models—parameter count, tokenization, context window, context length, and temperature—explaining their impact on model capability, computational cost, generation quality, and how to balance them for optimal performance.
AIParameterscontext window
0 likes · 7 min read

This guide breaks down the complex terminology of large language models—explaining tokens, transformers, self‑attention, RAG, scaling laws, dense vs. sparse architectures, and training stages—using clear analogies and step‑by‑step explanations so readers can confidently understand and work with modern AI systems.
AI FundamentalsModel TrainingRAG
0 likes · 35 min read

This article explains how natural language is converted into neural network inputs, covering tokenization with BPE using DeepSeek's 128K vocabulary, special tokens, and word embeddings via Word2Vec CBOW/Skip-gram, demonstrated with cosine similarity examples from a 7B model's 4096-dimensional embedding layer.
BPEDeepSeekLLM
0 likes · 12 min read

This article reviews Kronos, a unified and scalable pre‑training framework designed for financial K‑line data, detailing its tokenization approach, autoregressive architecture, large‑scale pre‑training on 12 billion records, and experimental results that show substantial gains in price prediction, volatility forecasting, synthetic data generation, and investment simulation.
Kronosautoregressive pretrainingfinancial time series
0 likes · 9 min read


Qborfy AIAug 16, 2025 · Artificial Intelligence
This article explains what tokens are in large language models, how they are counted and priced, compares tokenization methods across major models, and provides practical guidelines and code examples for optimizing token usage and selecting the appropriate model for different scenarios.
AICost OptimizationLLM
0 likes · 8 min read


Qborfy AIAug 12, 2025 · Artificial Intelligence
This article explains how massive Transformer‑based large language models compress text data into mathematical representations, why scale, self‑attention, and training paradigms enable emergent general intelligence, and walks through tokenization, embedding, multi‑layer attention, architecture choices, energy costs, and hallucination mitigation.
AIEmbeddingLLM
0 likes · 6 min read

The article provides a comprehensive technical analysis of state space models (SSM) versus Transformers, covering their core mechanisms, three essential design factors, training efficiency, scaling behavior, tokenization debates, and experimental evidence that highlights the trade‑offs and potential advantages of SSMs in modern AI systems.
MambaState Space ModelTransformer
0 likes · 21 min read

This article explains the LLM inference process, detailing the prefill and decode stages, tokenization, KV cache mechanics, and key metrics like TTFT and ITL, while introducing inference frameworks MindIE and LMDeploy with their optimization features.
DecodeKV CacheLLM inference
0 likes · 8 min read

The article explains how ERC-1400 extends ERC-20 with built‑in compliance features—such as KYC checks, transfer restrictions, tranche handling, on‑chain document storage, and forced transfer mechanisms—to enable legally compliant tokenization of real‑world assets like equity, bonds, and real‑estate.
BlockchainComplianceERC-1400
0 likes · 7 min read

This article analyzes the fundamental differences between state space models (SSM) and Transformer architectures, highlighting their three core components, training efficiency, memory handling, tokenization impact, and empirical performance trade‑offs, and argues why SSMs can outperform Transformers on many sequence tasks.
AI ArchitectureSequence ModelingTransformers
0 likes · 19 min read

The article explains that tokens are the fundamental computational units of large language models, describes how tokenization works, why charging per token reflects the high inference cost, and outlines token characteristics, pricing, and their impact on applications such as dialogue and document Q&A.
AI pricingBPEinference cost
0 likes · 6 min read

This article provides a detailed Chinese‑to‑English summary of Andrej Karpathy’s 7‑hour LLM tutorial, covering chat process analysis, tokenization, pre‑training data pipelines, model architecture, training strategies, post‑training fine‑tuning, reinforcement learning, chain‑of‑thought reasoning, and current industry applications.
AILLMPretraining
0 likes · 25 min read

The article provides a comprehensive technical walkthrough of Qwen’s large‑model series, covering data preparation, tokenization, model tweaks, training settings, RLHF pipeline, Code‑Qwen specifics, Qwen2 and Qwen3 architectural changes, scaling‑law experiments, and detailed source‑code analysis with illustrative diagrams.
MoEQwenRLHF
0 likes · 7 min read


DevOpsApr 13, 2025 · Artificial Intelligence
This article reviews the breakthrough image‑generation capabilities of GPT‑4o, showcases diverse examples, and offers a detailed speculation on its underlying autoregressive architecture, tokenization methods, VQ‑VAE/GAN advances, and training strategies that could explain its performance.
AI researchGPT-4oImage Generation
0 likes · 16 min read


58 TechApr 11, 2025 · Artificial Intelligence
This report details a comprehensive set of optimizations for multimodal visual large‑model (VLM) inference—including image pre‑processing acceleration, TensorRT integration for the ViT module, CUDA‑Graph replay, token‑count reduction, prefix‑cache handling, and weight quantization—demonstrating up to three‑fold throughput gains while maintaining accuracy.
CUDA GraphTensorRTinference-optimization
0 likes · 19 min read

This article walks readers through the fundamentals of large language models, covering what tokens are, how tokenization works, the conversion of tokens to numeric IDs, the transformer architecture—including positional encoding, self‑attention, feed‑forward networks and softmax—and explains how these components enable next‑token prediction.
EmbeddingLLMSelf-Attention
0 likes · 9 min read

This article explains the fundamentals, workflows, examples, and trade‑offs of three major subword tokenization algorithms—Byte Pair Encoding, WordPiece, and SentencePiece—helping practitioners choose the right method for their large language model pipelines.
BPENLPSentencePiece
0 likes · 12 min read


NewBeeNLPJan 17, 2025 · Artificial Intelligence
This comprehensive survey examines the foundations, tokenization techniques, model architectures, training paradigms, evaluation benchmarks, and open challenges of multimodal next‑token prediction (MMNTP), offering researchers a clear roadmap for future advances in multimodal AI.
Next Token PredictionTraining Paradigmsevaluation
0 likes · 9 min read

This article compares five LangChain text splitters—CharacterTextSplitter, RecursiveCharacterTextSplitter, TokenTextSplitter, MarkdownTextSplitter, and LatexTextSplitter—by examining their principles, pros and cons, and ideal use cases, helping developers select the most suitable splitter for their frontend large‑model applications.
Frontend DevelopmentJavaScriptLLM
0 likes · 10 min read

The article explains how tokenization mismatches and biased next‑token prediction cause LLMs to miscount letters in “Strawberry” and incorrectly compare 9.9 versus 9.11, and shows that step‑by‑step Chain‑of‑Thought prompting with reason‑first output dramatically improves accuracy.
AIChain-of-ThoughtLLM
0 likes · 13 min read

The article explains that a token is the unit processed by large language models, describes three common tokenizer methods—word‑level, character‑level, and sub‑word level—with English and Chinese examples, discusses their advantages and limitations, and shows how OpenAI’s tokenizer varies across model versions.
NLPSubwordcharacter-level
0 likes · 5 min read

A compiler transforms source code into executable machine instructions by first tokenizing the text into keywords, identifiers and literals, then parsing these tokens into an abstract syntax tree, generating and optimizing intermediate code, and finally assembling and linking the output for the target architecture or LLVM IR.
ASTCompilerLLVM
0 likes · 4 min read

This article walks through how a compiler turns human‑readable source code into binary machine instructions, covering tokenization, parsing, abstract syntax tree construction, code generation, optimization, and linking, while highlighting the role of LLVM as a portable backend.
ASTCompilerLLVM
0 likes · 5 min read


21CTOAug 11, 2024 · Artificial Intelligence
This article explains the fundamentals of large language models, covering tokenization, probability prediction, Markov chain basics, training data limitations, context windows, and the transition to neural network architectures like Transformers, while providing Python examples and insights into model scaling and the illusion of intelligence.
AILLMTransformer
0 likes · 18 min read


ArchitectAug 11, 2024 · Artificial Intelligence
This article explains how generative AI models work by demystifying tokens, tokenization with tools like tiktoken, simple Markov‑chain training, the limitations of small context windows, and how modern LLMs use neural networks, transformers and attention mechanisms to predict the next token.
LLMMarkov chainTransformer
0 likes · 20 min read

This article describes a business approval scenario where a company name entered by a business user must be checked for duplicates, and explains how to implement fuzzy matching using MySQL RegExp, tokenization with IKAnalyzer, and Java service code to extract, preprocess, match, and rank results by relevance.
BackendJavaMySQL
0 likes · 11 min read

The article examines why leading large language models such as GPT‑4o, Gemini Advanced, and Claude 3.5 incorrectly claim that 9.11 is larger than 9.9, analyzes tokenization and prompting strategies that cause the error, and discusses recent research and OpenAI model updates.
AI ReasoningNumerical Comparisonlarge language models
0 likes · 7 min read

Recent tests reveal that popular large language models—including GPT‑4o, Gemini Advanced, and Claude 3.5—often claim 9.11 is larger than 9.9 because their tokenizers split the numbers, but rephrasing, zero‑shot chain‑of‑thought prompts, or treating the values as floating‑point numbers can correct the mistake, a pattern also seen variably in Chinese models.
AI-evaluationLLMnumeric comparison
0 likes · 7 min read

This article explains the fundamental architecture of large language models, from the dual file nature of parameters and code, through neural network basics, perceptrons, and weight training, to the Transformer’s tokenization, positional encoding, self‑attention, and inference processes, illustrated with diagrams and examples.
Neural NetworkSelf-AttentionTransformer
0 likes · 22 min read

This tutorial introduces AIGC concepts and walks through practical implementations of tokenization, part‑of‑speech tagging, and named entity recognition using the Transformers library, NLTK, and spaCy on Google Colab, complete with code snippets and visual results.
AIGCNLPNLTK
0 likes · 10 min read

This article demonstrates how to implement a basic big‑data search engine in Python by creating a Bloom filter for fast existence checks, designing tokenization functions for major and minor segmentation, building an inverted index, and supporting AND/OR queries with example code and execution results.
Bloom Filterbig datainverted index
0 likes · 12 min read

The article explains how tokenization transforms data security into a built‑in attribute that automatically scales with data growth, detailing its design principles, generation methods, architectural layers, security safeguards, and practical engineering experiences to address exposure risks in modern digital businesses.
PIIdata governancedata security
0 likes · 24 min read

This guide walks through setting up a Java GraalVM 17 environment, installing Nginx to serve static dictionary files, configuring a HanLP‑based Elasticsearch analyzer plugin, packaging and deploying it, and testing the analyzer with JUnit5 and curl commands.
ElasticsearchHanLPJava
0 likes · 14 min read

The article details WebKit’s rendering pipeline in WKWebView, describing how the network process streams HTML bytes to the rendering process, which decodes them via TextResourceDecoder, tokenizes the characters with HTMLTokenizer’s state machine, and constructs an efficient DOM tree using HTMLTreeBuilder and queued insertion tasks.
DOMWebKitbrowser engine
0 likes · 33 min read

WKWebView parses HTML by streaming bytes from the network process to the rendering process, decoding them into characters, tokenizing into HTML tokens, building a DOM tree through node creation and insertion, and finally laying out and painting the document using a doubly‑linked in‑memory structure.
DOMWKWebViewWebKit
0 likes · 37 min read

This article explains why MySQL's default full‑text index struggles with Chinese, demonstrates how to configure token size parameters, activate the ngram parser plugin, and adjust queries (including Boolean mode) to achieve reliable Chinese full‑text search results.
Boolean ModeChineseMySQL
0 likes · 12 min read

This tutorial walks you through installing the jieba Python library, explains its three segmentation modes—precise, full, and search—demonstrates how to add or delete words, manage custom dictionaries, handle stop words, perform weight analysis, adjust word frequencies, and retrieve token positions, all with clear code examples and visual output.
NLPPythonchinese segmentation
0 likes · 10 min read

This article explains how PHP_CodeSniffer parses PHP source code into tokens using lexical analysis, demonstrates token extraction with token_get_all, and guides readers through creating a custom rule to prohibit hash‑style comments, covering rule library setup, Sniff implementation, and execution.
PHPPHP_CodeSniffercustom rules
0 likes · 12 min read

This article explains what an inverted index is, why it’s essential for full‑text search, how it is built and queried, and common token transformations such as stop‑word removal, lemmatization, and stemming.
full-text searchinformation retrievalinverted index
0 likes · 4 min read

This article explains how to implement a lexical analyzer in JavaScript that tokenizes simple arithmetic expressions by using a finite state machine, covering the conversion from infix notation to an abstract syntax tree, token definitions, state transitions, and complete source code examples.
ASTFinite State MachineJavaScript
0 likes · 9 min read

This article explains the fundamentals of JSON, its object and array structures, maps JSON types to Java equivalents, and provides a complete Java implementation of a JSON parser including token definitions, lexical analysis, and object/array construction with detailed code examples.
JSONJavaParser
0 likes · 14 min read

This tutorial walks through building a simple Python search engine for big data, covering Bloom filter basics, tokenization with major and minor segmentation, inverted index creation, and implementing both simple and complex (AND/OR) queries, complete with code examples and visual illustrations.
AND/OR queriesBloom FilterPython
0 likes · 15 min read

This article demonstrates how to implement a basic Bloom filter, tokenization, and inverted index in Python to illustrate the core principles of big‑data search, including fast negative lookups, term segmentation, and support for AND/OR queries.
AND/OR queriesBloom Filterbig data search
0 likes · 13 min read

The article summarizes the author’s takeaways from the GIAC Shenzhen conference, analyzing various public‑chain projects, their architectural choices, competitive focuses such as scalability, asset tokenization, DApp support, and the role of alliance chains in finance, traceability, and anti‑counterfeiting.
UTXOalliance chainpublic chain
0 likes · 8 min read

This article explains how blockchain, as a decentralized and trust‑less ledger, enables digital identity, tokenized assets, smart contracts, automated governance, and streamlined settlement across sectors such as finance, government, and healthcare, while also discussing its challenges and future potential.
Smart Contractsdecentralized ledgerdigital identity
0 likes · 7 min read

This article walks through implementing a basic big‑data search system in Python, covering Bloom filter basics, tokenization of text, inverted index construction, and how to combine these techniques to support fast AND/OR queries.
Bloom FilterPythonbig data
0 likes · 13 min read

This article explains how Search SQL enables easy full‑text search on Transwarp Search by using standard SQL syntax, covering tokenization, analyzer configuration, CONTAINS queries, and advanced NEAR and FUZZY operators to improve performance and query semantics.
FUZZYNEARSearch SQL
0 likes · 9 min read


ITPUBDec 23, 2015 · Artificial Intelligence
This article explains how natural language processing stores word meanings as numeric vectors, builds token dictionaries, represents sentences as binary vectors, and uses dot‑product calculations to measure similarity, illustrating concepts with simple examples and highlighting current limitations and future directions.
NLPartificial intelligencetokenization
0 likes · 7 min read
