Mastering AliyunPipelineConfig: A Complete Guide to Modern Log Collection in Cloud Native Environments

Background

Alibaba Cloud Log Service (SLS) provides data collection, processing, querying, visualization, alerting, and delivery capabilities for development, operations, security, and other scenarios.

iLogtail is a lightweight, high‑performance collector that can be deployed on physical machines, VMs, or Kubernetes to gather logs, traces, and metrics.

An iLogtail collection configuration defines the pipeline for input, processing, and output.

Why Use a CRD for Collection Configurations

Managing configurations through the console is not fully automated. Manually updating many configurations for each release is error‑prone and cumbersome.

In the cloud‑native era we need a flexible, decoupled, centrally managed, and easily automated solution—enter Custom Resource Definitions (CRDs).

In Kubernetes, a CRD extends the API, allowing users to define and manage their own resources. Using a CRD, collection configurations become native Kubernetes objects, consistent with Pods, Services, and Deployments.

Existing CRD: AliyunLogConfig

AliyunLogConfig Example

apiVersion: log.alibabacloud.com/v1alpha1</code><code>kind: AliyunLogConfig</code><code>metadata:</code><code>  name: example-k8s-file</code><code>  namespace: kube-system</code><code>spec:</code><code>  project: k8s-log-xxxx</code><code>  logstore: k8s-file</code><code>  shardCount: 10</code><code>  lifeCycle: 30</code><code>  logstoreHotTTL: 7</code><code>  logtailConfig:</code><code>    inputType: file</code><code>    configName: example-k8s-file</code><code>    inputDetail:</code><code>      logType: common_reg_log</code><code>      logPath: /data/logs/app_1</code><code>      filePattern: test.LOG</code><code>      dockerFile: true</code><code>      advanced:</code><code>        k8s:</code><code>          K8sNamespaceRegex: ^(default)$</code><code>          K8sPodRegex: '^(demo-0.*)$'</code><code>          K8sContainerRegex: ^(demo)$</code><code>          IncludeK8sLabel:</code><code>            job-name: "^(demo.*)$"

AliyunLogConfig provides default project handling and supports various features, but it suffers from structural confusion and limited functionality.

Introducing AliyunPipelineConfig

Compared with AliyunLogConfig, AliyunPipelineConfig offers a clearer format, lower configuration difficulty, and better observability.

Type

AliyunPipelineConfig (new)

AliyunLogConfig

ApiGroup

telemetry.alibabacloud.com/v1alpha1

log.alibabacloud.com/v1alpha1

CRD Resource Name

ClusterAliyunPipelineConfig

AliyunLogConfig

Scope

Cluster

Default Cluster

Configuration Format

Equivalent to LogtailPipelineConfig[1]

Equivalent to LogtailConfig[2]

Cross‑Region

Supported

Supported

Cross‑Account

Supported

Supported

Webhook Validation

Supported

Not Supported

Conflict Detection

Supported

Not Supported

Configuration Difficulty

Low

High

Observability

Status includes error details, update time, last successful config, etc.

Status includes error code and message only.

AliyunPipelineConfig Structure

apiVersion: telemetry.alibabacloud.com/v1alpha1</code><code>kind: ClusterAliyunPipelineConfig</code><code>metadata:</code><code>  name: test-config</code><code>spec:</code><code>  project:</code><code>    name: k8s-your-project</code><code>    uid: 11111</code><code>    endpoint: cn-hangzhou-intranet.log.aliyuncs.com</code><code>  logstores:</code><code>    - name: your-logstore</code><code>      ttl: 30</code><code>      shardCount: 10</code><code>  machineGroups:</code><code>    - name: machine-group-1</code><code>    - name: machine-group-2</code><code>  config:</code><code>    inputs:</code><code>      ...</code><code>    processors:</code><code>      ...</code><code>    aggregators:</code><code>      ...</code><code>    flushers:</code><code>      ...</code><code>    enableUpgradeOverride: false

Key characteristics:

Clear structure grouped by SLS resource type.

Only essential fields are mandatory (project.name, config.inputs, config.flushers).

Field names align with SLS API.

Field Details

project

Defines the target project. Once created, it cannot be modified; a new CR is required for changes.

name (string, required): Project name.

description (string, optional): Project description.

endpoint (string, optional): Service endpoint for cross‑region.

uid (string, optional): Account UID for cross‑account.

config

sample (string, optional): Log sample.

global (object, optional): Global settings.

inputs (object list, required): Input plugins (exactly one).

processors (object list, optional): Processing plugins.

aggregators (object list, optional): Aggregation plugins.

flushers (object list, required): Output plugins (must be flusher_sls).

configTags (map, optional): Custom tags for the iLogtail config.

The config fields map directly to the SLS CreateLogtailPipelineConfig API, supporting all features.

Validation and Error Handling

The CRD schema is validated by Kubernetes, and a webhook checks fixed parameters such as metadata.name, spec.project.name, and the presence of input plugins.

apiVersion: telemetry.alibabacloud.com/v1alpha1</code><code>kind: ClusterAliyunPipelineConfig</code><code>metadata:</code><code>  finalizers:</code><code>    - finalizer.pipeline.alibabacloud.com</code><code>  name: example-k8s-file</code><code>spec:</code><code>  config:</code><code>    flushers:</code><code>      - Endpoint: cn-hangzhou.log.aliyuncs.com</code><code>        Logstore: k8s-file</code><code>        Region: cn-hangzhou</code><code>        TelemetryType: logs</code><code>        Type: flusher_sls</code><code>    inputs:</code><code>      - EnableContainerDiscovery: true</code><code>        FilePaths:</code><code>          - /data/logs/app_1/**/test.LOG</code><code>        Type: input_file</code><code>project:</code><code>  name: k8s-log-clusterid</code><code>status:</code><code>  success: false</code><code>  message: |-</code><code>    {</code><code>        "httpCode": 401,</code><code>        "errorCode": "Unauthorized",</code><code>        "errorMessage": "The project does not belong to you.",</code><code>        "requestID": "xxxxxx"</code><code>    }</code><code>  lastUpdateTime: '2024-06-19T09:21:34.215702958Z'</code><code>  lastAppliedConfig: {}

Status provides detailed error messages, timestamps, and the last successfully applied configuration, enabling quick troubleshooting and automatic retry with exponential back‑off.

Conflict Resolution

AliyunPipelineConfig tags each configuration with cluster, namespace, and type. Only the creator can modify a tagged config; other CRs receive a rejection with conflict details recorded in Status.

Practical Examples

Collecting Nginx‑Ingress Logs

apiVersion: telemetry.alibabacloud.com/v1alpha1</code><code>kind: ClusterAliyunPipelineConfig</code><code>metadata:</code><code>  name: k8s-nginx-ingress</code><code>spec:</code><code>  project:</code><code>    name: <your-project-name></code><code>  logstores:</code><code>    - name: nginx-ingress</code><code>      productCode: k8s-nginx-ingress</code><code>  config:</code><code>    inputs:</code><code>      - Type: service_docker_stdout</code><code>        Stdout: true</code><code>        Stderr: true</code><code>        IncludeLabel:</code><code>          io.kubernetes.container.name: nginx-ingress-controller</code><code>    processors:</code><code>      - Type: processor_regex</code><code>        Keys: [client_ip, x_forward_for, remote_user, time, method, url, version, status, body_bytes_sent, http_referer, http_user_agent, request_length, request_time, proxy_upstream_name, upstream_addr, upstream_response_length, upstream_response_time, upstream_status, req_id, host, proxy_alternative_upstream_name]</code><code>        NoKeyError: true</code><code>        NoMatchError: true</code><code>        Regex: ^(\S+)\s-\s\[([^]]+)]\s-\s(\S+)\s\[(\S+)\s\S+\s"(\w+)\s(\S+)\s([^"]+)"\s(\d+)\s(\d+)\s"([^\"]*)"\s"([^\"]*)"\s(\S+)\s(\S+)+\s\[([^]]*)]\s(\S+?(?:,\s\S+?)*)\s(\S+?(?:,\s\S+?)*)\s(\S+?(?:,\s\S+?)*)\s(\S+?(?:,\s\S+?)*)\s(\S+)\s*(\S*)\s*\[*([^]]*)\]*.*</code><code>        SourceKey: content</code><code>    flushers:</code><code>      - Type: flusher_sls</code><code>        Logstore: nginx-ingress</code><code>        Endpoint: <your-endpoint></code><code>        Region: <your-region></code><code>        TelemetryType: logs

Cross‑Region & Cross‑Account Setup

Modify ilogtail_config.json to include multiple regions and endpoints, mount it via a ConfigMap, and update the controller’s environment variables with the target account’s access keys.

{</code><code>    "config_server_address" : "http://logtail.cn-hangzhou.log.aliyuncs.com",</code><code>    "config_server_address_list": [</code><code>        "http://logtail.cn-shanghai.log.aliyuncs.com",</code><code>        "http://logtail.cn-hangzhou.log.aliyuncs.com"</code><code>    ],</code><code>    "data_server_list" : [</code><code>        {"cluster" : "cn-shanghai", "endpoint" : "cn-shanghai.log.aliyuncs.com"},</code><code>        {"cluster" : "cn-hangzhou", "endpoint" : "cn-hangzhou.log.aliyuncs.com"}</code><code>    ]</code><code>}

After updating the ConfigMap and controller deployment, restart the relevant DaemonSet and Deployment to apply the changes.

Upgrade Path from AliyunLogConfig

When the following conditions are met, set enableUpgradeOverride: true in AliyunPipelineConfig to replace an existing AliyunLogConfig:

Both CRDs exist in the same cluster.

They target the same project and iLogtail config name.

The upgrade flag is enabled.

The controller will apply the new pipeline config, then delete the old AliyunLogConfig upon success.

Conclusion

AliyunPipelineConfig simplifies log collection in Kubernetes, supports cross‑region and cross‑account scenarios, provides robust validation and observability, and offers a clear migration path from AliyunLogConfig.