How FlashText Cuts Keyword Search from Days to Minutes

Data cleaning is a primary challenge in many machine learning projects. The open‑source Python library FlashText provides fast large‑scale keyword search and replacement, turning a task that takes five days with regular expressions into about fifteen minutes.

Regex can become unbearably slow when the number of keywords exceeds a few hundred and the corpus contains millions of documents.

FlashText solves this problem by building a Trie (prefix‑tree) dictionary from the keyword list and using the Aho‑Corasick algorithm to scan the input text character by character. Because the search time does not depend on the number of keywords, performance remains constant even with hundreds of thousands of terms.

The library can both extract keywords and replace them in a single pass.

The above graph shows that while regex search time grows linearly with the number of keywords, FlashText’s search time stays flat.

Similarly, FlashText’s replacement speed far outperforms regex.

When the keyword count exceeds about 500, FlashText becomes noticeably faster than regex. However, FlashText does not support regex‑style patterns (e.g., ^, $, *), so it is best suited for exact‑match keyword extraction.

Typical usage involves creating a KeywordProcessor, adding keywords (optionally with replacement values), and then calling extract_keywords() or replace_keywords() on the target text.

# pip install flashtext
from flashtext.keyword import KeywordProcessor

keyword_processor = KeywordProcessor()
keyword_processor.add_keyword('Big Apple', 'New York')
keyword_processor.add_keyword('Bay Area')

keywords_found = keyword_processor.extract_keywords('I love Big Apple and Bay Area.')
# ['New York', 'Bay Area']

new_sentence = keyword_processor.replace_keywords('I love Big Apple and Bay Area.')
# 'I love New York and Bay Area.'

Internally, FlashText builds a Trie where each node represents a character. Special markers Start and EOT denote word boundaries, preventing partial matches such as matching "apple" inside "pineapple".

During processing, the algorithm walks the input string once, checking each character against the Trie. Even with a dictionary containing millions of keywords, the runtime remains unaffected, which is the core advantage of FlashText.

In summary, FlashText offers a highly efficient solution for keyword extraction and replacement in large text corpora, especially when dealing with hundreds or thousands of exact‑match terms.