How to Simplify Log Collection with Alibaba Cloud’s New AliyunPipelineConfig CRD

Background

Alibaba Cloud Log Service (SLS) provides log, trace and metric collection, processing, query, visualization, alerting and delivery. iLogtail is a lightweight, high‑performance collector that runs on VMs, physical machines or Kubernetes clusters. An iLogtail collection configuration (iLogtailLogConfig) defines inputs, processors and outputs.

Why a CRD?

Managing collection configurations through the SLS console is manual and error‑prone. In cloud‑native environments a declarative, version‑controlled resource is needed. A Custom Resource Definition (CRD) extends the Kubernetes API so that collection configurations become native objects that can be kept in sync with Pods, Services and Deployments.

Limitations of the legacy AliyunLogConfig CRD

All SLS resources (project, logstore, machineGroup, config) are flattened under spec, making the schema confusing.

Field names are inconsistent with the SLS API (e.g., logstoreHotTTL vs shardCount).

Only project and logstore are immutable; other fields have unclear update semantics.

Multiple CRs pointing to the same underlying config can cause conflicts, especially across clusters.

New CRD: AliyunPipelineConfig

The new cluster‑scoped CRD ( ClusterAliyunPipelineConfig) groups resources by type and aligns the schema with the SLS API.

Key Spec Fields

project : target SLS project (name, optional description, optional endpoint for cross‑region, optional uid for cross‑account).

logstores : list of Logstore definitions (name, ttl, shardCount, hotTtl, autoSplit, telemetryType, etc.).

machineGroups : optional list of machine‑group names; if omitted a default group is created.

config : collection configuration that mirrors the CreateLogtailPipelineConfig API (inputs, processors, aggregators, flushers, optional global settings, configTags). Only one input plugin and one flusher_sls are allowed, matching current API constraints.

Validation & Error Reporting

Kubernetes validates the CRD schema; a mutating/validating webhook performs deeper checks (e.g., metadata.name must be a valid iLogtail config name, spec.project.name must follow project naming rules, required plugins are present). Errors are written to the status field with detailed messages, timestamps and the last successfully applied configuration. The controller retries transient failures with exponential back‑off.

apiVersion: telemetry.alibabacloud.com/v1alpha1
kind: ClusterAliyunPipelineConfig
metadata:
  name: example-k8s-file
spec:
  project:
    name: my-project
  logstores:
    - name: k8s-file
  config:
    inputs:
      - Type: input_file
        FilePaths:
          - /data/logs/app_1/**/test.LOG
    flushers:
      - Type: flusher_sls
        Logstore: k8s-file
        Endpoint: cn-hangzhou.log.aliyuncs.com
        Region: cn-hangzhou
        TelemetryType: logs

Cross‑Region & Cross‑Account Support

To send logs from one region/account to another, add multiple config_server_address entries and a list of data_server_list entries in the iLogtail ilogtail_config.json. Deploy the file via a ConfigMap mounted into the logtail-ds DaemonSet and set the controller’s ALICLOUD_LOG_ACCOUNT_INFOS environment variable with the target account’s UID, AccessKeyID and AccessKeySecret.

Upgrade Path from AliyunLogConfig

If the old and new CRDs coexist in the same cluster, target the same project and use the same iLogtail config name, set enableUpgradeOverride: true on the AliyunPipelineConfig. The controller will replace the legacy CR and delete it after a successful apply.

apiVersion: telemetry.alibabacloud.com/v1alpha1
kind: ClusterAliyunPipelineConfig
metadata:
  name: k8s-nginx-ingress
enableUpgradeOverride: true
spec:
  project:
    name: my-project
  config:
    inputs:
      - Type: service_docker_stdout
        IncludeLabel:
          io.kubernetes.container.name: nginx-ingress-controller
    processors:
      - Type: processor_regex
        Regex: ^(\S+) ...
    flushers:
      - Type: flusher_sls
        Logstore: nginx-ingress
        Endpoint: cn-hangzhou.log.aliyuncs.com
        Region: cn-hangzhou
        TelemetryType: logs

Practical Example: Nginx‑Ingress Log Collection

The following manifest creates a Logstore named nginx-ingress, captures container stdout/stderr, parses fields with a regex processor and outputs to the Logstore via flusher_sls. The structure is concise and aligns with the SLS API.

apiVersion: telemetry.alibabacloud.com/v1alpha1
kind: ClusterAliyunPipelineConfig
metadata:
  name: k8s-nginx-ingress
spec:
  project:
    name: my-project
  logstores:
    - name: nginx-ingress
      productCode: k8s-nginx-ingress
  config:
    inputs:
      - Type: service_docker_stdout
        Stdout: true
        Stderr: true
        IncludeLabel:
          io.kubernetes.container.name: nginx-ingress-controller
    processors:
      - Type: processor_regex
        Keys: [client_ip, method, url, status]
        Regex: ^(\S+) ...
        SourceKey: content
    flushers:
      - Type: flusher_sls
        Logstore: nginx-ingress
        Endpoint: cn-hangzhou.log.aliyuncs.com
        Region: cn-hangzhou
        TelemetryType: logs

Spec Field Details

project

name (string, required): target project name.

description (string, optional): project description.

endpoint (string, optional): service endpoint for cross‑region scenarios.

uid (string, optional): account UID for cross‑account scenarios.

Note: project is immutable after creation; changing it requires a new CR.

config

sample (string, optional): log sample.

global (object, optional): global iLogtail settings.

inputs (list, required): exactly one input plugin.

processors (list, optional): any number of processing plugins.

aggregators (list, optional): aggregation plugins.

flushers (list, required): exactly one flusher_sls plugin.

configTags (map, optional): custom tags attached to the iLogtail config (not written to log records).

Only one input and one flusher are allowed because the underlying SLS API enforces this limitation.

logstores

name (string, required): Logstore name.

queryMode (string, optional): standard (default) or query.

ttl (int, optional, default 30): data retention days (1‑3650, 3650 means permanent).

hotTtl (int, optional): hot data retention (must be < ttl and ≥ 7).

shardCount (int, optional, default 2, range 1‑100).

maxSplitShard (int, optional, default 64, range 1‑256).

autoSplit (bool, optional, default true): enable automatic shard splitting.

telemetryType (string, optional): None (logs) or Metrics.

appendMeta (bool, optional, default true): record external IP.

enableTracking (bool, optional, default false): enable web tracking.

encryptConf (object, optional): encryption configuration.

meteringMode (string, optional): ChargeByFunction or ChargeByDataIngest.

All parameters map directly to the SLS CreateLogstore API. Only parameters relevant at creation time are applied; existing Logstores are not modified.

machineGroups

name (string, required): name of the machine group to associate with the collection config.

If omitted, a default identifier‑type machine group is created automatically. Existing machine groups are not overwritten.

Upgrade Override

Set enableUpgradeOverride: true to let the controller replace a matching AliyunLogConfig with the new AliyunPipelineConfig and delete the legacy CR after successful reconciliation.

Error Handling & Retry

The status sub‑resource contains: success: boolean indicating whether the CR was applied. message: detailed error payload from the SLS API (e.g., HTTP code, error code, message, request ID). lastUpdateTime: timestamp of the most recent status update. lastAppliedConfig: the configuration that was successfully applied (populated only after a successful reconciliation).

Transient failures (e.g., quota limits) are retried with exponential back‑off until they succeed.

Conflict Detection

When a CR creates a collection config, the controller labels the config with the originating cluster, namespace and CR name. If another CR attempts to create a config with the same name, the controller rejects the request and records the conflict in status.message, preventing accidental overwrites across clusters.

Summary

AliyunPipelineConfig

provides a clear, API‑aligned, cluster‑scoped way to manage SLS collection configurations in Kubernetes. It supports flexible project and Logstore definitions, built‑in schema validation, cross‑region/account capabilities, conflict detection via labeling, detailed status reporting and a seamless upgrade path from the legacy AliyunLogConfig CRD.