How Computers Turn Words into Numbers: A Beginner’s Guide to Tokenization and Vector Similarity

Why Computers Need Numbers for Language

Although AI programs can perform impressive tasks, they cannot yet understand language like humans. To process text, computers store the meaning of words and phrases as numbers. Popular tools such as word2vec use shallow neural networks to create word embeddings, and memory‑network techniques can learn simple question‑answer patterns.

Challenges in Natural Language Processing

Compared with image recognition, NLP faces a scarcity of usable samples despite the abundance of textual data from books, blogs, and social media. For example, a high‑resolution photo (≈88 MB) contains far more raw data than the complete works of Shakespeare (≈4.4 MB), yet meaningful linguistic samples remain limited.

Tokens vs. Characters

In text analysis, the fundamental units are tokens (words or sub‑words), not individual characters. Tokens drive both analysis and generation, unlike pixels in images.

Building a Simple Vocabulary

To illustrate, we create a dictionary that maps each unique token to an index:

0 turn
1 on
2 the
3 lights
4 power
5 what
6 time
7 is
8 it
9 current

Each token’s index can be represented as a binary vector where the position of the token is 1 and all others are 0.

Encoding Example Sentences

Four example sentences are encoded as 10‑dimensional binary vectors: { 1, 1, 1, 1, 0, 0, 0, 0, 0, 0 } – “Turn on the lights” { 0, 1, 1, 1, 1, 0, 0, 0, 0, 0 } – “Power on the lights” { 0, 0, 0, 0, 0, 1, 1, 1, 1, 0 } – “What time is it?” { 0, 0, 1, 0, 0, 1, 1, 1, 0, 1 } – “What is the current time?”

Measuring Similarity with Dot Product

The dot product of two vectors yields a crude similarity score. For the first two sentences the dot product is 3, indicating some overlap, while the first and third sentences have a dot product of 0, showing no similarity.

These calculations demonstrate how token‑based vector representations can quantify textual similarity, albeit with a very limited vocabulary.

Limitations and Future Work

The example uses only ten tokens; real‑world applications require large vocabularies and richer embeddings. Future research aims to compress vectors and capture deeper semantic relationships for more accurate language understanding.