SLS Auto-Discovers Field Indexes: Eliminate Manual Index Maintenance for Dynamic Logs

In modern observability, log ingestion is flexible — new fields appear with every release — but analysis still depends on pre‑configured indexes. When a critical field lacks an index, queries fail, and adding indexes for historical data requires time‑consuming rebuilds. This gap widens with AI Agents that generate new fields and dynamic query paths at runtime, making manual index management impractical.

Pain Points of Manual Index Maintenance

Fields cannot be fully determined at onboarding: Application upgrades, new data sources, and conditional fields mean the initial log sample never covers all future fields.

Missing indexes are discovered only at query time: A new filter or grouping dimension fails because the field has no index. Adding the index then forces a wait; covering historical data may require a full rebuild, delaying analysis.

AI Agents make the problem acute: Agents write logs with task‑dependent fields and intermediate results. Their multi‑step “reason‑retrieve‑observe” loops dynamically decide which fields to query next. Pausing for index creation breaks the autonomous reasoning flow.

SLS Automatic Field Index Discovery

Alibaba Cloud Log Service (SLS) now offers automatic field index discovery . When enabled, SLS continuously inspects incoming logs, automatically creates a field index for any top‑level field that lacks one, and enables statistics for it — no manual intervention required.

Continuous Discovery: Auto‑Create Indexes and Enable Statistics

As logs are written, SLS detects previously unseen top‑level fields, builds their indexes, and updates the index configuration. Subsequent fields from new application versions, additional data sources, or previously unseen conditions are indexed on arrival. Developers save the sync‑maintenance step; analysts can immediately use new fields in queries and SQL aggregations.

Write‑Time Build: Seamless Coverage of Pre‑Config Data

Index construction starts the moment a field is written, independent of when the configuration becomes visible in the console. Data ingested during the configuration propagation window is already indexed, so once the config appears, that data is instantly queryable — no backfill or rebuild needed.

Unified Text Type: Compatibility with Uncertain Field Types

Because a field’s first value (e.g., 123) may misleadingly suggest a numeric type while later values are strings (e.g., abc), SLS indexes all auto‑discovered fields as text . This high‑compatibility choice avoids type conflicts and ensures every new field is indexed successfully.

SQL Adaptive Conversion: Transparent Numeric Operations

Although stored as text, auto‑discovered fields work naturally in numeric SQL expressions. When the engine encounters a numeric function like sum(x) on an auto‑detected field, it implicitly rewrites it to sum(try_cast(x as double)). Non‑numeric values are silently skipped. Users can write standard numeric SQL without manually altering field types.

Enabling and Using Automatic Discovery

Via Console

Open the target Logstore’s index configuration page.

Ensure full‑text index is enabled.

Check “Auto‑detect unspecified fields (requires full‑text index)” and save.

Console rollout is gradual; if unavailable, open a ticket to request whitelist access.

Viewing Auto‑Detected Fields

After writing new logs, discovered fields appear under “Auto‑detected fields” in the index configuration. They are all text type, share the full‑text index’s tokenizer, case‑sensitivity, and Chinese‑token settings, and have statistics enabled by default.

To customize a field’s properties, click the + icon to promote it to a manually configured field, then edit its attributes.

Query and Analysis with Auto‑Detected Fields

Once the field configuration is visible, the fields work identically to manually created indexes: field search, quick analysis, and full SQL are all supported. Because statistics are enabled by default, numeric SQL functions work transparently via the runtime cast.

Via SDK (Python ≥ 0.9.52)

SDK support is available in all regions without whitelist.

In the index configuration JSON, "auto_key_detect": true enables the feature, and "auto_text_keys" lists the fields that have been auto‑discovered.

{
  "line": {
    "auto_key_detect": true,
    "auto_text_keys": ["status", "code", "latency"]
  },
  "keys": {}
}

Python SDK example:

from aliyun.log import IndexConfig, IndexLineConfig

# Enable auto discovery
client.create_index(
    project, logstore,
    IndexConfig(line_config=IndexLineConfig(
        token_list=[" ", ",", ";", "
", "\t"],
        auto_key_detect=True,
    ))
)

# Retrieve auto discovery config
index = client.get_index_config(project, logstore).get_index_config()
print("Auto discovery enabled:", index.line_config.auto_key_detect)
print("Discovered fields:", index.line_config.auto_text_keys)

For detailed configuration limits and FAQs, refer to the official documentation at https://help.aliyun.com/zh/sls/automatic-field-indexing.

Summary

As AI Agents become active participants in planning and analysis, query demands shift from pre‑defined to runtime‑generated. Modern data systems must not only scale elastically and retrieve efficiently, but also automate prerequisite work — field recognition, index building — so that exploration proceeds without waits or human intervention. Automatic index discovery is SLS’s next step toward data self‑adaptation : elastic capacity handles volume changes, flexible schema handles structure evolution, and now auto‑discovery guarantees immediate usability of new fields. With configuration no longer a bottleneck, developers and AI Agents alike can focus purely on the business problem, achieving true out‑of‑the‑box, free‑form log analysis.