How to Accurately Track Document Write Time in Elasticsearch – 3 Practical Methods

Why Elasticsearch Lacks a Built‑In Write‑Time Field

Elasticsearch is a near‑real‑time search engine that generates an _id and version number for each document, but it does not automatically add a timestamp. To trace when a document was written, a timestamp must be added explicitly during the ingest stage.

Three Implementation Options Compared

Option 1: Ingest Pipeline (standard production solution, highly recommended)

Elasticsearch provides built‑in ingest nodes that can modify documents before indexing. A pipeline can be created to set a field (e.g., _source.create_time) to the ingest timestamp {{_ingest.timestamp}}. The pipeline is then referenced when indexing documents.

PUT _ingest/pipeline/add_timestamp
{
  "description": "Add write time field",
  "processors": [
    {
      "set": {
        "field": "_source.create_time",
        "value": "{{_ingest.timestamp}}"
      }
    }
  ]
}

PUT my-index/_doc/1?pipeline=add_timestamp
{
  "title": "Test document"
}

GET my-index/_search

Pros: No code changes required, centralized control, millisecond precision.

Note: Use a field name such as create_time to avoid confusion with the internal @timestamp field.

Option 2: Logstash/Beats Ingestion (source‑side solution)

When data is shipped via Logstash, a filter can add a write_time field that copies the event’s @timestamp value.

filter {
  mutate {
    add_field => { "write_time" => "%{[@timestamp]}" }
  }
}

Applicable Scenarios: Log data, messages consumed from queues such as Kafka.

Option 3: Application‑Side Timestamp (hard‑coded in code)

Clients in Java, Go, Python, etc., can manually add a timestamp field when constructing the document, e.g., create_time: new Date().

Cons: Strong dependency on business code; many applications may forget to add the field, leading to inconsistent timestamps.

Tracing Write Time for Historical Data

For existing documents that were indexed before any of the above solutions were applied, the write time cannot be recovered unless the upstream source already carried a timestamp (e.g., MySQL, Kafka) or external audit tools such as ES Auditbeat are used.

Upstream data sources that embed timestamps.

Log or audit plugins (e.g., ES Auditbeat) to assist in locating the time.

Conclusion

Immediate implementation: create and activate the chosen pipeline.

Long‑term plan: enforce a unified pipeline or index template that pre‑defines the timestamp field for all indices.