Deploy Elastic Stack Beats on Tencent Cloud for Scalable Monitoring

Overview

Elastic Stack Beats modules automate log collection and metric gathering. By using Tencent Cloud’s Elasticsearch service as a backend, you can build a secure, scalable monitoring pipeline for large cloud environments.

Create Elasticsearch Cluster

In the Tencent Cloud console, open the Elasticsearch service and create a new cluster with the following recommended settings:

Region: Beijing

Version: 7.5.1 (Platinum)

Deployment: Single availability zone

Storage mode: Cold‑hot

Leave other parameters at their defaults unless specific workload requirements dictate changes. After creation, note the internal Kibana URL (e.g., http://es-xxxx.internal.kibana.tencentelasticsearch.com:5601) and the public Kibana URL (e.g., https://es-xxxx.kibana.tencentelasticsearch.com:5601). Set a super‑user name and password for the cluster.

Create Beats Write Role and User

In Kibana’s Management → Security → Roles, create a role that grants only write permissions to the Beats indices. Then create a user (for example beats-writer) and assign the new role, using a strong password. This minimal‑privilege account will be used by all Beats instances.

Initialize Beats on a Setup Host

On a dedicated Linux VM, install the Beat RPM packages (Filebeat and Metricbeat). Run the following one‑time setup commands to load index templates, ILM policies, and Kibana dashboards. Replace placeholders with your actual Elasticsearch host, username, password, and Kibana address.

filebeat setup -e \
  -E output.logstash.enabled=false \
  -E output.elasticsearch.hosts=['<strong>ES_HOST:9200</strong>'] \
  -E output.elasticsearch.username=elastic \
  -E output.elasticsearch.password=YourPassword \
  -E setup.kibana.host=<strong>INTERNAL_KIBANA_HOST:5601</strong>

metricbeat setup -e \
  -E output.elasticsearch.hosts=['<strong>ES_HOST:9200</strong>'] \
  -E output.elasticsearch.username=elastic \
  -E output.elasticsearch.password=YourPassword \
  -E setup.kibana.host=<strong>INTERNAL_KIBANA_HOST:5601</strong>

These commands need to be executed only once per VM. After they complete, the Kibana Dashboard menu will show a set of pre‑loaded visualizations.

Deploy Beats to Target Nodes

Clone the helper repository that contains an installation script and run it on each target host:

git clone https://github.com/martinliu/elastic-stack-lab.git
cd elastic-stack-lab/tencent
sh add-agent.sh

Verify that the Beat services are running: sudo systemctl status filebeat The script removes unrelated default configurations, applies a minimal best‑practice set, and leaves placeholders for additional modules.

Key Configuration Snippets

filebeat.yml

# =========================== Filebeat inputs ===========================
filebeat.inputs:
- type: log
  enabled: false
  paths:
    - /var/log/*.log

# =========================== Modules ===========================
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 60s

# --------------------------- Elasticsearch output ---------------------------
output.elasticsearch:
  hosts: ["${INT_ES_SRV}"]
  username: ${BEATS_WRITER_USERNAME}
  password: ${BEATS_WRITER_PW}

# =========================== Processors ===========================
processors:
  - add_host_metadata:
      netinfo.enabled: true
      cache.ttl: 5m
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
  - add_fields:
      target: ''
      fields:
        service.name: 'Joint Lab'
        service.id: 'es-qq'

# =========================== Best practice ===========================
setup.ilm.check_exists: false
logging.level: error
queue.spool: ~
monitoring:
  enabled: true

metricbeat.yml follows the same structure, adjusting the modules path and retaining the identical Elasticsearch output and processor sections.

Troubleshooting

filebeat setup hangs or fails

If the setup command does not finish, wait a few minutes and retry. Ensure that both the Elasticsearch endpoint and the internal Kibana URL are reachable from the host.

Configuration errors prevent service start

Temporarily comment out the line logging.level: error and any custom output sections, then run filebeat -e to view detailed startup logs. The output will pinpoint syntax or reference errors in the configuration file.

References

All scripts and configuration files are available in the public repository:

https://github.com/martinliu/elastic-stack-lab.git