Recreating Google Ngram Trends for “Python” with PyTubes and NumPy

Google Ngram Viewer visualizes word usage over time using a massive n‑gram dataset. This article shows how to reproduce the “Python” usage chart with Python, NumPy and the new data‑loading library PyTubes.

Challenge

The 1‑gram dataset expands to 27 GB, containing 1.43 billion rows across 38 files. Processing such volume in pure Python is slow, but NumPy can handle it efficiently.

Loading the data

All examples run on a 2016 MacBook Pro with 8 GB RAM.

The 1‑gram files are tab‑separated with four fields: word, year, count, and number of books. We filter rows where the word equals “Python” and the year is after 1799.

import tubes
FILES = glob.glob(path.expanduser("~/src/data/ngrams/1gram/googlebooks*"))
WORD = "Python"
one_grams_tube = (tubes.Each(FILES)
    .read_files()
    .split()
    .tsv(headers=False)
    .skip_unless(lambda row: row.get(1).to(int).gt(1799))
    .multi(lambda row: (
        row.get(0).equals(WORD.encode('utf‑8')),
        row.get(1).to(int),
        row.get(2).to(int)
    )))

After filtering we obtain roughly 1.3 billion rows (only 3.7 % before 1800).

Yearly word totals

We compute the total number of words per year using np.histogram weighted by the count column.

last_year = 2008
YEAR_COL = '1'
COUNT_COL = '2'
year_totals, bins = np.histogram(
    one_grams[YEAR_COL],
    density=False,
    range=(0, last_year+1),
    bins=last_year+1,
    weights=one_grams[COUNT_COL]
)

Python’s yearly share

We accumulate the percentage of “Python” occurrences for each year.

word_rows = one_grams[IS_WORD_COL]
word_counts = np.zeros(last_year+1)
for _, year, count in one_grams[word_rows]:
    word_counts[year] += 100 * count / year_totals[year]

Plotting the result yields a curve similar to Google’s, though absolute percentages differ because the dataset includes variants such as “Python_VERB”.

Performance

Google’s chart renders in about one second, while the script takes roughly eight minutes on the same hardware. Pre‑computing yearly totals or indexing the data could reduce runtime dramatically.

Language war

Extending the analysis to compare “Python”, “Pascal” and “Perl” shows how case‑sensitive matching and baseline adjustments affect trends.

Future PyTubes improvements

Support for 1‑, 2‑ and 4‑bit integer types to cut memory usage.

More expressive filtering (combined AND/OR/NOT).

Enhanced string matching functions (startswith, endswith, contains, is_one).

Contributions are welcome.