Quickly Set Up One‑Click Data Ingestion Pipelines in Tencent Cloud Elasticsearch

Background

Managed Elasticsearch services on major cloud platforms provide monitoring and log viewing, but users must manually connect each component (source, collector, cache, processor, destination) to build a complete data pipeline, which increases integration and maintenance effort.

One‑Click Data Ingestion Service

Tencent Cloud Elasticsearch Service (ES) integrates with CKafka to offer a visual, one‑step data‑link creation service. The service reduces configuration and maintenance costs and accelerates pipeline creation. Supported sources include CVM, TKE, MySQL, and others.

Usage Steps

Log in to the ES console and select Data Ingestion to view the data‑link list.

Click New Data Link to open the creation wizard.

Basic Configuration

Enter a data‑link name; an ID is generated automatically.

Select a region; all subsequent resources are created in this region.

Choose a data source (e.g., CVM or TKE for logs, MySQL for metrics). Additional sources can be added via documentation.

Click Start Creation to proceed to component configuration.

Component Configuration Guidelines

All components should reside in the same VPC to ensure network connectivity. If they are in different VPCs, configure routing or VPC peering manually.

Select the Data Link Mode to switch between predefined component sets.

On first use, authorize the ES service to access Automation Assistant, CVM, TKE, and CKafka. This grants the required IAM role.

Example Pipeline: CVM → Filebeat → CKafka → Logstash → ES

Data Source

Choose a VPC.

Select a Linux CVM instance that has Automation Assistant installed.

Data Collection

Configure Filebeat collection parameters; output fields are auto‑filled upon submission.

Data Cache (CKafka)

Select a CKafka instance.

Choose a VPC that is already routed to the CKafka instance.

Create or select a Topic. If creating without a name, the system generates {linkID}_topic_{random}.

Data Processing (Logstash)

Select a VPC.

Choose a Logstash instance.

The service creates a pipeline named {linkID}_lgpipe_{random}. Further processing can be edited in the Logstash console after creation.

Data Destination (Elasticsearch)

Select a VPC.

Choose an ES instance.

Specify the index type. Autonomous indices are supported on clusters created after 2022‑06‑01 with version 7.14.2 or later; older clusters must be upgraded.

Provide an index name. If omitted, the system generates {linkID}_index_{random}. For autonomous indices, you may pre‑define field mappings; for existing autonomous indices, ensure the time field matches exactly.

Example Pipeline: TKE → Filebeat → CKafka → Logstash → ES

Data Source

Choose a VPC.

Select a TKE cluster.

Data Collection

Configure namespace inclusion, optional Pod label filters, container name (optional), log content filters, and advanced parsing options. The collector gathers logs from containers that match the criteria.

Data Cache, Processing, and Destination

The CKafka, Logstash, and ES steps are identical to the CVM example, with automatic generation of Topic, pipeline, and index names.

Notes

All components must share the same VPC unless custom network routes are configured.

IAM role authorization is required only once per ES account.

Autonomous index support requires ES version 7.14.2 or later; older versions need an upgrade.

The service auto‑fills output fields and generates IDs/names when users leave those fields blank.