This article explains how computers store Chinese characters using Unicode and UTF-8, covering the history of encoding chaos, Unicode code points, UTF-8 variable-length encoding rules, a step-by-step example of encoding '中', common mojibake causes, BOM, and programming best practices like using utf8mb4 in MySQL.
This article explains how Python 3 treats all strings as Unicode by default, removing the need for encoding declarations required in Python 2, and demonstrates the behavior on Windows and Linux where the default environment encoding shifts from ASCII to UTF-8.
Unicode 18.0 adds 13,007 characters, raising the total to 172,808, highlighted by 11,328 Chinese seal script characters, Jurchen script, nine new emojis, three currency symbols, and a critical security fix for variation selectors to prevent hidden AI injection attacks.
After 14 years of dormancy, Go’s proposal #3939 to remove the confusing string(int) conversion—originally added for early formatting—has been moved to the regular proposal committee, prompting analysis of its semantics, historical reasons, compatibility concerns, and possible future outcomes.
This article explains Java's 16‑bit char type, how Unicode literals are written, the role of the Character wrapper class, boxing and unboxing mechanics, immutability, common static methods, and proper use of escape sequences such as Unicode and quoted strings, with concrete code examples.
This guide walks programmers through Unicode’s massive code space, its diverse scripts, encoding schemes like UTF‑8 and UTF‑16, combining marks, canonical equivalence, normalization forms, and grapheme clusters, explaining why the system is complex yet essential for global text handling.
Zero‑width characters are invisible Unicode symbols that can silently break form validation, URL parsing, and data storage in web applications, but with proper detection, visualization, and input‑filtering techniques developers can mitigate these hidden risks and even use them for legitimate purposes.
Version 1.3.5 of the Bun runtime introduces native PTY support, compile‑time feature flags that enable dead‑code elimination, and accurate Unicode string width handling, along with several other improvements such as V8 type‑check APIs, S3 upload options, and numerous bug fixes.
When debugging a web app, the author discovered that certain emojis occupy more than one character slot in JavaScript strings, revealing that emoji length varies because they are composed of multiple Unicode code points such as variation selectors and zero‑width joiners.
A developer turned the browser’s URL bar into a 40×4 pixel playground, using Unicode Braille characters and less than 400 lines of JavaScript to create a fully playable Snake game that updates the address bar in real time.
Although MySQL historically labeled its three‑byte character set as “utf8”, it actually implements a truncated version (utf8mb3) that cannot store the full Unicode range, leading to bugs with emojis and rare characters; the newer utf8mb4 restores true UTF‑8 support and is now the default in MySQL 8.0.
This article walks through a common Chinese character garbling problem in Python web crawlers, explains why automatic encoding detection can fail, and provides clear code examples—including manual GBK setting and re‑encoding tricks—to reliably extract readable text.
This article explains why Chinese characters appear as garbled text in Jupyter notebooks and VS Code, demonstrates the issue with screenshots, evaluates several Python code fixes, and ultimately provides a reliable solution by setting the console code page to UTF‑8.
Apple's exclusive emojis often appear as unrelated symbols on Android devices, leading to misunderstandings; this article explains the technical reasons behind the display differences, shares real‑world examples, and traces the history and global adoption of emojis from their Japanese origins to today’s Unicode standards.
The article explains what zero‑width Unicode characters are, lists common types, describes their applications such as steganography and text formatting, and provides Python code examples for inserting, hiding, extracting, bypassing filters, and removing these invisible characters.
Linus Torvalds sharply criticized the case‑insensitive feature in file systems as a huge mistake, warning that it introduces serious security vulnerabilities by allowing mismatched filenames and Unicode characters to be treated as equivalent, undermining user‑space security checks and exposing systems to attacks.
This article explains how character encoding works—from ASCII and its extensions to Chinese GB2312 and GBK, through Unicode's UCS‑2, UCS‑4, and the versatile UTF‑8—showing why mismatched encodings produce garbled text and why UTF‑8 is the default in Spring Boot.
This article explains the Unicode foundation of Emoji, shows how to insert and style them in HTML, CSS and JavaScript, discusses common pitfalls with surrogate pairs and string slicing, and presents modern solutions such as Intl.Segmenter and libraries like grapheme‑splitter and emoji‑regex for reliable handling.
This article dives deep into MySQL's charset and collation system, explaining concepts, configuration levels, system variables, string literals, conversion rules, Unicode sorting algorithms, binary collations, and practical tips to avoid common encoding pitfalls and ensure correct string handling.
This article reviews the six key ECMAScript 2024 proposals—including well‑formed Unicode strings, asynchronous atomic wait, the new RegExp v flag, ArrayBuffer transfer, array grouping, and Promise.withResolvers—explaining their purpose, API changes, and providing runnable code examples.
This article explains why emojis become garbled when transmitted via sockets, explores Unicode encoding fundamentals—including UTF‑8, BMP and high‑code‑point characters—and provides practical solutions using codePointAt, TextEncoder, and TextDecoder to ensure correct emoji handling.
The article reviews ECMAScript 2024, highlighting new small‑scale features such as improved WebAssembly interop, enhanced Promise utilities, group‑by methods, better Unicode handling, async locking with Atomics.waitAsync, and resizable ArrayBuffers, while also discussing upcoming proposals for 2025.
Rob Pike and Ken Thompson devised UTF‑8 in 1992 at Bell Labs, turning a three‑day prototype into the web’s dominant Unicode encoding by using a variable‑length, ASCII‑compatible, length‑prefixed and prefix‑free scheme that maximizes efficiency, robustness, and universal adoption across more than 96 % of sites.
This article reveals five advanced Java tricks—using Unicode escapes in comments, over‑complicating simple logic with bitwise shifts, tampering with Boolean.TRUE via reflection, forcing both branches of an if‑else to run, and leveraging the Unsafe class for low‑level memory manipulation—to deliberately make code hard to understand.
This article explains how computer text is represented by assigning unique numeric codes to characters and converting those codes into binary, then compares the most common Chinese encodings—ASCII, GB2312, GBK, GB18030, and UTF‑8—detailing their compatibility, byte lengths, and practical impact on software development.
This article explains why Chinese filenames become corrupted when extracted with Python's zipfile module and provides a complete, step‑by‑step solution that re‑encodes the names correctly using standard libraries.
This article explains why JavaScript string length behaves unexpectedly with Unicode characters, describes UTF‑16 encoding and surrogate pairs, and demonstrates ES6 techniques such as for‑of loops, spread syntax, the u regex flag, codePointAt, and normalize to handle Unicode correctly.
This article explains the purpose and mechanics of common Chinese character encodings—including ASCII, GB2312, GBK, GB18030, UTF‑8, ANSI and Latin‑1—detailing how they map characters to numbers, handle byte boundaries, and maintain compatibility to avoid garbled text.
The article explains how a binary message on a multilingual public‑restroom sign was decoded by identifying UTF‑8 byte patterns, extracting the first 24 bits to reveal the Chinese character “向”, and providing a Java program that parses the entire bit string into readable Chinese text.
This article explains how an invisible zero‑width Unicode character embedded in a MySQL table name can make the table appear to disappear, demonstrates the problem with reproducible examples, analyzes the root cause, and provides practical steps to detect and fix such issues.
A programmer discovers a binary message on a multilingual restroom sign, deduces it uses UTF‑8 encoding, explains the encoding rules, and provides a Java program that converts the binary sequence into the Chinese character "向", demonstrating practical UTF‑8 decoding.
This article explains the history, purpose, and differences of major character encodings—including ASCII, GB2312, Unicode, and UTF-8—while showing how they are used and converted in modern computing environments.
During a Spring Festival park visit, I discovered a binary‑encoded poster, used OCR and custom Go code to translate the bits, learned it was UTF‑8 Unicode for Chinese characters, and explained the byte‑pattern rules that reveal the hidden message.
This article explains the differences between utf8, utf8mb3 and utf8mb4 character sets in MySQL, demonstrates how utf8mb4 enables full Unicode support including emojis, and provides step‑by‑step SQL examples for creating tables, inserting data, and querying results with the proper character set.
The article explains that JavaScript’s native btoa() and atob() functions only handle ASCII, so to correctly base64‑encode Unicode strings you must convert them with TextEncoder to UTF‑8 bytes, use Uint8Array, and decode with TextDecoder, while checking for malformed surrogate pairs via isWellFormed or encodeURIComponent to avoid silent data loss.
Java 21 introduces six static methods in java.lang.Character that let developers reliably detect emojis, emoji presentations, modifiers, modifier bases, components, and extended pictographic characters using Unicode code points, with practical code examples and regex usage.
This article explains how a hidden non‑breaking space character copied from documentation caused the MySQL lower_case_table_names parameter to be unrecognizable, details the reproduction steps, and presents three methods (hexdump, od, editor) to detect and resolve such encoding issues.
This article explains the purpose and usage of several Python special methods—including __unicode__, __delattr__, __del__, __dict__, and __all__—and provides concrete code examples illustrating how each method works in practice.
This article explains the concepts of characters, character sets, and character encoding, describes how computers store and render text using methods like ASCII, GB2312, Unicode, and UTF‑8/16/32, and discusses why garbled text occurs across different languages and systems.
This guide explains the essential character set encodings—ANSI, ASCII, GB2312/GBK/GB18030, Unicode planes, UTF‑16, UTF‑32, and UTF‑8—and shows how they are used in MFC and Qt, providing code examples to avoid garbled text in software.
This article explains the fundamentals of character encoding, covering Unicode’s universal character set, the structure of its planes and surrogate areas, the variable‑length UTF‑8 and UTF‑16 encodings, Chinese‑specific GBK encoding, and practical iOS code examples for handling Unicode, emojis and regular‑expression based Chinese character detection.
This guide explains common character encodings such as ANSI, ASCII, GB2312/GBK/GB18030, Unicode, UTF‑8, UTF‑16, and UTF‑32, and shows how to handle them in MFC and Qt with practical code examples.
This article explains how emojis are represented in Unicode, covering basic code points, variation selectors, skin‑tone modifiers, zero‑width joiners, flag ligatures, tag sequences and keycap symbols, and shows how JavaScript can correctly process them using grapheme‑cluster techniques.
This article explains the fundamentals of character sets and encodings—including ASCII, GB2312, GBK, Unicode, UTF‑8, UTF‑16, and UCS‑2—shows how they affect JavaScript string length, emoji handling, and common pitfalls such as combining characters, endianness, and BOM issues.
This article walks through a Python word‑cloud issue where Chinese characters were not displayed, shows how to suppress irrelevant warnings, and provides the exact Matplotlib font settings and font‑installation steps needed to render the characters correctly.
The article reviews the latest TC39 meeting outcomes, explains the criteria for advancing proposals to higher stages, and dives into three key proposals—Array.fromAsync for async iteration, String.prototype.isWellFormed for Unicode scalar validation, and extractors for pattern matching—complete with code examples and usage notes.
This article explains why PHP 7.0's json_decode throws JSON_ERROR_UTF16 when encountering malformed Unicode surrogate pairs, demonstrates the problem with example code, and details the underlying re2c scanner logic that causes the error, offering insight for backend developers.
The article explains how to enforce a custom length limit in JavaScript input fields that counts English letters and numbers as half a unit, Chinese characters as one unit, and each emoji as one unit, using Unicode code‑point detection, regex extraction, and automatic truncation to prevent overflow.
This article explains why JavaScript's string length property returns unexpected values for Unicode characters like emojis, explores UTF‑16 encoding rules, and demonstrates modern ES6 techniques—including for‑of loops, spread syntax, and the \u{…} and /u regex flags—to correctly handle Unicode strings.
This article explains why Chinese characters appear as garbled symbols in Jupyter notebooks, reviews several ineffective Python‑based fixes, and presents a simple Windows console command (chcp 65001) that reliably resolves the encoding issue.
This article explains how the Cyrillic character е looks identical to the Latin e, why using it as a Python variable leads to NameError, demonstrates the Unicode code point differences, and warns about the potential bugs when unintentionally mixing these characters in code.
The article traces character encoding from early telegraph and Morse code through ASCII, ISO national sets and Chinese standards, explains Unicode’s unification and its UTF‑8/‑16/‑32 forms, and shows how modern languages—especially JavaScript—handle code points, highlighting the cultural and technical significance for developers.
This article explains the Unicode mapping between full‑width and half‑width characters, demonstrates simple Python functions for converting between them, and provides a flexible dictionary‑based approach for custom text conversion tasks.
This tutorial explains the origins and evolution of character encoding, covering early ASCII, Chinese GBK/GB18030, the universal Unicode standard, UTF‑8 variable‑length encoding, and practical differences between Python 2 and Python 3 with code examples.
This article explains why inserting emoji into MySQL fails with the default utf8 charset, demonstrates how switching to utf8mb4 resolves the issue, and provides a historical overview of MySQL's limited utf8 implementation and the proper use of true UTF‑8 encoding.
This article answers a fan’s question by clarifying the distinction between Python’s encode() function and the encoding parameter, explaining how encode() defaults to UTF‑8, how explicit encoding strings work, and when each is used in string handling and web crawling.
This article explains why a specially crafted Unicode string triggers an Android ANR by dissecting the Unicode line‑breaking (UAX #14) and bidirectional (UAX #9) algorithms, showing how runs ending with double zeros lead to an infinite loop in TextLine.getOffsetToLeftRightOf.
This article continues the investigation of a mysterious string that triggers Android app freezes, explaining Unicode line‑breaking and bidirectional algorithms, how runs end with double zeros, how to craft a reproducing string, and practical ways to avoid the resulting ANR.
This article walks through the evolution of character encoding—from the early ASCII standard, through Chinese extensions like GB2312 and GBK, to the universal Unicode system and its UTF‑32, UTF‑16, and UTF‑8 encodings—explaining their structures, usage, and common pitfalls.
This article explains how computers store text using binary, introduces ASCII, Unicode and UTF‑8 encoding rules, discusses the limitations of C‑style null‑terminated strings, and describes Redis's Simple Dynamic String (SDS) data structure, its old and new versions, advantages, and related APIs.
This article explains the history and purpose of Unicode, describes how character sets differ from encodings, details the storage formats of UTF‑8, UTF‑16, and UTF‑32, discusses byte order and BOM, and shows common encoding pitfalls in Redis and MySQL with practical solutions.
This article traces the evolution from ASCII to Unicode, explains how Unicode defines universal code points, compares the UTF‑8, UTF‑16 and UTF‑32 encoding schemes, discusses byte order and BOM, and shows practical fixes for common encoding problems in Redis and MySQL.
This article explains MySQL's Unicode character set support, compares the deprecated utf8mb3 with the modern utf8mb4, and provides step‑by‑step SQL commands for safely converting existing databases to the newer charset.
This article explains why characters appear as garbled text when encoding and decoding methods mismatch, explores how Excel defaults to GBK, shows how to convert files with iconv, and walks through the evolution from ASCII to GB2312, GBK, GB18030, and finally Unicode's UTF‑8 encoding.
This article introduces Python’s built‑in string class and explains ten commonly used string methods—including center, count, find, swapcase, startswith/endswith, split, case conversions, justification, strip, and zfill—detailing their syntax, parameters, and example usages while emphasizing that they return new strings without altering the original.
This step‑by‑step guide shows how to log into Toutiao, locate the fan list via browser dev tools, capture the AJAX request URLs, decode Unicode nicknames, paginate through all pages, and save the results as JSON using Python.
This article explains why strings containing emojis report different lengths in JavaScript, covering Unicode fundamentals, code points, UTF‑8 and UTF‑16 encodings, surrogate pairs, grapheme clusters, zero‑width joiners, and modern ES2015‑ESNext features that help handle Unicode correctly.
This article explains the process by which keyboard input is transformed into ASCII codes, how input methods map those codes to proprietary encodings, the role of code pages and fonts in rendering characters, and the use of Unicode, BOM, and endianness when saving text files.
This article explains why the garbled string “锟斤拷” appears in Chinese software, covering basic character encoding concepts, the Unicode replacement character, UTF‑8 decoding failures, and how GBK’s double‑byte scheme turns the placeholder bytes into the visible characters 锟、斤、拷.
This article explains how JavaScript processes Unicode characters, demonstrates the limitations of legacy APIs like charCodeAt and fromCharCode with supplementary characters, and introduces modern methods such as codePointAt, fromCodePoint, Unicode escape syntax, surrogate pairs, and polyfills for full Unicode support.
This article explains how character encoding works, using ASCII, Unicode, UTF‑8 and GBK examples to reveal why the garbled string “锟斤拷” appears when mismatched encodings are processed, and shows the underlying byte‑level transformations.
This guide explains how to add Unicode emoji rendering to Laravel projects by installing the Laravel Emoji package, configuring its service provider and alias, and using its facade methods to retrieve emojis by alias, name, or Unicode code point.
This article explains the various Java string methods for removing leading and trailing whitespace—including trim, strip, stripLeading, stripTrailing, replace, replaceAll, and replaceFirst—showing their differences, Unicode handling, and practical code examples.
This article explains how Java’s char type stores Unicode code units in UTF‑16, why its range of \u0000 to \uffff limits direct representation of newer Unicode characters, and how methods like String.length, getBytes, and code‑point APIs help handle multi‑byte characters such as emojis and rare Chinese glyphs.
The article explains a character‑encoding pitfall caused by a Python middle‑layer converting parameters to Unicode, provides a Java utility to decode escaped Unicode strings, and demonstrates how to use Spring's @Transactional annotation with rollbackFor to ensure database operations are rolled back on errors.
The article explains why MySQL’s legacy “utf8” character set only supports three‑byte sequences, causing errors with genuine four‑byte UTF‑8 characters, and shows how the proper “utf8mb4” charset resolves the issue, including historical background, practical examples, and migration guidance.
The article explains the process by which computers render characters, covering binary representation, character encoding standards such as Unicode and GBK, the role of input methods, font files, and the challenges posed by rare characters and system compatibility.
This article explains how to add Unicode emoji rendering to a Laravel application by installing the Laravel‑Emoji extension, configuring Composer, registering the service provider and alias, defining a route, and using the Emoji class in a controller to convert aliases, names, and Unicode codes into emoji graphics.
This article explains how characters are turned into binary, mapped through Unicode and character sets, processed by input methods and font files, and finally rendered on screen, while also covering the challenges of rare characters and recent Unicode updates.
This article explains Unicode's structure, encoding ranges, UTF-8/16/32 representations, byte order considerations, and provides Java code to detect emojis in strings, illustrating practical usage of Unicode concepts for text processing.
This article explains the Unicode standard, its code planes and ranges, the three UTF encoding forms (UTF-8, UTF-16, UTF-32), compares their storage characteristics, discusses byte order marks, and provides Java code for detecting emoji characters in strings.
This article explains the fundamentals of characters, character sets and encodings such as ASCII, GB2312, Unicode, UTF‑8/16/32, BOM, and demonstrates how Java, browsers, Tomcat and MySQL handle encoding conversion, endianness and string length issues with practical code examples.
This article explains how Java stores characters as UTF‑16 code units, why the char type cannot cover the entire Unicode range, how surrogate pairs work, and demonstrates the differences in length, byte length, and char array size for regular Chinese characters, emojis, and rare Chinese glyphs.
The article explains that MySQL’s legacy “utf8” charset only supports three‑byte characters, causing errors when storing true four‑byte UTF‑8 symbols like emojis, and shows why switching to the proper “utf8mb4” charset is essential for correct Unicode handling.
This article explains the fundamentals of character encoding, covering concepts such as information, symbols, character sets, various encoding schemes like ASCII, GB2312, UTF‑8, Unicode planes, common pitfalls, and practical examples to help developers avoid garbled text.
This article explains the fundamentals of character encoding—from Morse‑code analogies and ASCII to Unicode, UTF‑8/UTF‑16/UTF‑32, and Chinese encodings like GB2312, GBK, and GB18030—illustrating how mismatched encodings cause garbled text and how to avoid them.
This article explains why MySQL’s legacy "utf8" charset only supports three‑byte characters, why it’s not true UTF‑8, and provides a clear guide to migrating databases to the proper "utf8mb4" charset for full Unicode support.
This article explains the four new ES2018 regular‑expression features—lookbehind assertions, named capturing groups, the dotAll (s) flag, and Unicode property classes—showing how they work, when to use them, and providing clear JavaScript code examples.
This article explains the origins of character sets, the relationships among various encodings such as ASCII, GB2312, GBK, GB18030, Unicode, UTF-8, UTF-16, and UTF-32, and shows how JavaScript handles Unicode and emoji characters, including practical code examples and solutions for length‑limited input fields.
This article explains Unicode and UTF‑8 concepts, then demonstrates in Python 2 on Windows and Linux how encoding and decoding Chinese strings can cause errors, clarifies why str.encode('utf‑8') may still fail, and offers practical solutions for correct string handling.
This article explains why encoding problems frequently arise with Chinese characters in development, clarifies core concepts such as bytes, characters, ASCII, Unicode and UTF encodings, compares Python 2 and Python 3 handling of strings and bytes, and provides practical best‑practice recommendations to avoid encoding bugs.
The article explains how Python 3’s full Unicode integration introduces numerous filename and encoding pitfalls, especially with legacy data and non‑ASCII characters, and provides concrete steps—including using byte paths and sys.stdin/stdout buffers—to reliably handle Unix filenames.
This article explores how Unicode’s complex text layout lets characters like Thai letters stack multiple diacritics—creating “hats” and “shoes”—and examines the storage, rendering, and font challenges these bizarre glyphs pose for both developers and users.
This article explains the differences between JavaScript's escape, encodeURI and encodeURIComponent functions, the principles of percent‑encoding, the classification of reserved, unreserved and unsafe characters, and provides an overview of ASCII, Unicode and UTF‑8 character encodings.
This article investigates an Android JNI native crash caused by misusing NewString(), examines why a custom UTF‑8‑to‑UTF‑16 conversion was used instead of NewStringUTF(), compares Dalvik and ART string encodings, reveals a Dalvik UTF‑8 conversion bug fixed in ART, and advises developers on encoding nuances across Android versions.
This article explains the fundamental concepts of characters, character sets, and encodings, compares common encodings such as ASCII, Unicode, and UTF‑8, and shows how Python 2 and Python 3 handle default encodings, string types, and common Unicode errors with practical code examples.
MySQL 8.0 RC1 introduces major enhancements such as improved JSON handling, a full‑featured Document Store with transaction support, expanded Unicode and GIS capabilities, plus a suite of modern SQL features designed to better serve mobile‑first and cloud‑native applications.
This article reveals a collection of unconventional JavaScript techniques—including regex replacement, Unicode escapes, eval tricks, unusual operator combinations, custom getters/setters, and URL‑encoded payloads—that can evade common XSS filters and strengthen your understanding of web security.
This article explains the differences between JavaScript’s escape, encodeURI, and encodeURIComponent functions, detailing their encoding rules, percent‑encoding standards, reserved and unreserved characters, and how Unicode characters are transformed into UTF‑8 byte sequences, while also covering ASCII, Unicode, and UTF‑8 fundamentals.