Elasticsearch Wildcard Type: Advantages, Inverted Index, and Optimization Guide

Background: DataX and Wildcard Type Collection Issue

Data warehouse engineers using DataX to sync data to an offline warehouse discovered that DataX does not support collecting Elasticsearch index fields of wildcard type. The cloud vendor suggested changing those fields to keyword or text type, but that would increase memory consumption on ES data nodes and degrade query performance. The system architect confirmed that wildcard was chosen specifically to leverage inverted indexes and fuzzy matching for fast search, and the field type cannot be changed without risking business stability. An alternative collection method was adopted, and this article documents the core concepts of ES wildcard type.

Core Advantages of Wildcard Type

1.1 Storage Structure Optimized for Wildcard Queries

wildcard

type uses an inverted index storage method. Unlike text fields, it indexes the entire string value as a single term (similar to keyword). This allows wildcard queries (e.g., *.com, prefix*) to directly utilize the inverted index, avoiding full scans and significantly improving query speed.

1.2 No Need to Escape Special Characters

Standard keyword fields require manual escaping of special characters (e.g., \*) when executing wildcard queries. wildcard type automatically handles these special characters, simplifying query syntax.

1.3 Lower Memory Footprint

Compared to text fields that generate many sub-terms through tokenization, wildcard stores only the complete string as a single term in the inverted index, resulting in:

Fewer dictionary entries → smaller index size.

Less total data in posting lists → lower storage overhead.

No position metadata storage → reduced extra overhead.

More compact core in-memory structures (Term Index, posting lists) → lower memory usage.

What Is an Inverted Index?

An inverted index is the core data structure of full-text search systems (e.g., Elasticsearch). Its fundamental idea is a "content-to-document mapping" used to quickly locate documents containing a given keyword.

Forward Index vs. Inverted Index

Forward index: Document-centric. Maps Document ID → Document Content (like a book's table of contents).

Inverted index: Term-centric. Maps Keyword → List of Document IDs containing that keyword (like a book's index).

Core Components of an Inverted Index

Term Dictionary

Stores all unique terms extracted from documents, sorted (e.g., alphabetically) for fast lookup.

Posting List

For each term, stores a list of document IDs that contain the term, plus optional metadata:

Term Frequency (TF): Occurrence count in the document (used for relevance scoring).

Position: Exact positions in the document (used for phrase matching).

Document length, weights, etc. (for complex ranking algorithms).

Thus, the inverted index upgrades full-text search from "sequential document scanning" to "direct keyword positioning," forming the foundation of modern information retrieval.

Query Efficiency Comparison

Therefore, only when queries must use wildcards (especially non-prefix wildcards like *.com or a*c ) does wildcard type show significantly higher efficiency than keyword and text. For pure prefix queries (e.g., user_*), keyword with the keyword_prefixes parameter (ES 7.10+) is already highly efficient, making wildcard unnecessary.

Optimization Recommendations

1. Avoid Wildcard Queries When Possible

Alternatives:

Prefix queries: Use keyword field with keyword_prefixes parameter (ES 7.10+).

"my_field": {
  "type": "keyword",
  "keyword_prefixes": {}  // auto-generates prefix index
}

Regular expressions: For simple patterns, use regexp query (caution: can consume excessive resources).

Tokenization redesign: Split field into tokenizable parts (e.g., example.com → example and com) and use match queries.

2. Use Wildcard Type Judiciously

Applicable scenarios:

Non-prefix wildcards are mandatory (e.g., *.com, user?).

High query frequency with no alternative optimization.

Mapping configuration:

"my_wildcard_field": {
  "type": "wildcard"
}

Ensure indexing efficiency. Avoid overuse to prevent performance degradation. Plan carefully per specific requirements.

3. Narrow Query Scope

Reduce scanned documents by filtering first:

{
  "bool": {
    "filter": [
      {"term": {"category": "domain"}}  // filter by category first
    ],
    "must": [
      {"wildcard": {"my_wildcard_field": "*.com"}}  // then wildcard query
    ]
  }
}

4. Optimize Index Design

Shard count: Adjust based on data volume and query load to avoid oversized shards.

Cache warming: Cache hot wildcard query results (e.g., request_cache: true).

5. Monitoring and Tuning

Slow query logs: Enable ES slow query logs to identify expensive wildcard queries.

Index analysis: Use _analyze API to verify field indexing behavior.

Performance testing: Benchmark wildcard, keyword, and text in real scenarios to choose the optimal approach.

When to Use Wildcard Type

Recommended Scenarios

Business requires non-prefix wildcards (e.g., *.jpg, user_?).

Wildcard query frequency is extremely high and cannot be avoided via data modeling.

Not Recommended Scenarios

Only prefix queries needed (use keyword + keyword_prefixes for better efficiency).

Low query frequency where full scan cost is acceptable.

Summary

wildcard

type, by optimizing the index structure for wildcard queries, delivers significant performance gains in specific scenarios—especially non-prefix wildcards. However, in practice, priority should be given to data modeling and query optimization to reduce reliance on wildcards. Use wildcard only when necessary, and combine it with other optimization techniques to ensure system efficiency.