Unlock Loki v2.3.0: Custom Retention, Deletion & Recording Rules Explained

1. Introduction

Loki v2.3.0 adds notable new features, bug fixes, and performance improvements.

2. Custom Retention Rules

Loki’s retention was previously global for a cluster and delegated to the underlying object store. Now the Compactor component can handle retention, allowing per‑tenant and per‑stream retention periods, giving finer control over storage costs and compliance.

See the Loki storage retention documentation for more details.

3. Log Deletion

Loki now supports deleting selected log stream entries within a specified time range via the Compactor API. The initial implementation allows deletion of data up to 24 hours old (you cannot delete data from the past 24 hours). This experimental feature currently works only with the BoltDB‑shipped index storage.

4. Recording Rules

Loki supports Prometheus‑style recording rules, enabling periodic metric queries on logs and sending results to Prometheus or compatible systems such as Cortex and Thanos. This allows creating metrics solely from logs.

Using recording rules can reduce the number of series stored in Prometheus because each tenant generates only one series instead of one per bucket.

Refer to the recording rules documentation for more information.

5. Pattern Parser

The new pattern parser makes writing LogQL queries to extract labels from unstructured logs faster and easier, avoiding complex regular expressions.

Example NGINX log line:

<code>0.191.12.2 - - [10/Jun/2021:09:14:29 +0000] "GET /api/plugins/versioncheck HTTP/1.1" 200 2 "-" "Go-http-client/2.0" "13.76.247.102, 34.120.177.193" "TLSv1.2" "US" ""</code>

Pattern parser expression to parse the line:

<code>&lt;ip&gt; - - &lt;_&gt; "&lt;method&gt; &lt;uri&gt; &lt;_&gt;" &lt;status&gt; &lt;size&gt; &lt;_&gt; "&lt;agent&gt;" &lt;_&gt;</code>

Extracted fields:

<code>"ip"      => "0.191.12.2"
"method"  => "GET"
"uri"     => "/api/plugins/versioncheck"
"status"  => "200"
"size"    => "2"
"agent"   => "Go-http-client/2.0"
</code>

See the LogQL documentation for full syntax.

6. Sharding Ingestor

Sharding splits queries for parallel processing. Before sharding, recent data queries were slower. After enabling ingestion sharding, search speed increased from ~1 GB/s to ~15 GB/s.

Example query before sharding:

<code>❯ logcli query '{container="ingress-nginx"} |= "trytofindme!!!"' --since=1h --stats 2>&1 | grep Summary
Summary.BytesProcessedPerSecond  784 MB
Summary.LinesProcessedPerSecond  1263304
Summary.TotalBytesProcessed      88 GB
Summary.TotalLinesProcessed      141420028
Summary.ExecTime                 1m51.944531908s
</code>

After sharding:

<code>❯ logcli query '{container="ingress-nginx"} |= "trytofindme!!!"' --since=1h --stats 2>&1 | grep Summary
Summary.BytesProcessedPerSecond  15 GB
Summary.LinesProcessedPerSecond  24171034
Summary.TotalBytesProcessed      88 GB
Summary.TotalLinesProcessed      142086646
Summary.ExecTime                 5.878385063s
</code>

Ingestor sharding is enabled by default but only takes effect when a query‑frontend component creates the splits.

7. LogQL IP Address Matching

LogQL now includes powerful IP matching functions. Syntax:

ip("<pattern>")

Patterns can be a single IP, an IP range, or a CIDR block.

Single IP:

ip("192.0.2.0")

,

ip("::1")

Range:

ip("192.168.0.1-192.189.10.12")

,

ip("2001:db8::1-2001:db8::8")

CIDR:

ip("192.51.100.0/24")

,

ip("2001:db8::/32")

Examples:

<code>{container="nginx"} |= ip("192.168.4.5")</code>

Exclude a range:

<code>{container="nginx"} != ip("192.168.4.1-192.168.4.10")</code>

Match a subnet:

<code>{container="nginx"} |= ip("192.168.4.0/24")</code>

Refer to the LogQL documentation for more details.

8. Further Reading

The mica‑logging component of Spring Boot microservices supports JSON log collection to Loki and batch log sending.