Why a CK+Kafka+Filebeat Stack Beats ELK for Log Analytics

Background

For SaaS services facing data security and compliance requirements, the team needed a private‑deployed analytics stack that would improve operational insight without the heavy server overhead of a full big‑data platform.

Elasticsearch vs ClickHouse

Write throughput : a single ClickHouse node can ingest 50‑200 MB/s, exceeding 600 k records per second, more than five times Elasticsearch’s rate.

Query speed : ClickHouse claims 2‑30 GB/s from page‑cache; tests show it is 5‑30× faster than Elasticsearch, especially when data is not cached.

Cost : ClickHouse’s compression ratio is 1/3‑1/30 of Elasticsearch, reducing disk usage and I/O. It also consumes less memory and CPU, allowing a roughly 50% reduction in server cost for log processing.

Cost Analysis

Using Alibaba Cloud pricing without discounts, the ClickHouse‑based solution cuts the estimated server cost by about half compared with an Elasticsearch deployment.

Environment Deployment

1. Zookeeper cluster

yum install java-1.8.0-openjdk-devel.x86_64
/etc/profile # set environment variables
yum install ntpdate
ntpdate asia.pool.ntp.org

mkdir zookeeper
mkdir ./zookeeper/data
mkdir ./zookeeper/logs
wget --no-check-certificate https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/zookeeper-3.7.1/apache-zookeeper-3.7.1-bin.tar.gz
tar -zvxf apache-zookeeper-3.7.1-bin.tar.gz -C /usr/zookeeper
export ZOOKEEPER_HOME=/usr/zookeeper/apache-zookeeper-3.7.1-bin
export PATH=$ZOOKEEPER_HOME/bin:$PATH
cd $ZOOKEEPER_HOME/conf
vi zoo.cfg

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/usr/zookeeper/data
dataLogDir=/usr/zookeeper/logs
clientPort=2181
server.1=zk1:2888:3888
server.2=zk2:2888:3888
server.3=zk3:2888:3888

echo "1" > /usr/zookeeper/data/myid
echo "2" > /usr/zookeeper/data/myid
echo "3" > /usr/zookeeper/data/myid

cd $ZOOKEEPER_HOME/bin
sh zkServer.sh start

2. Kafka cluster

mkdir -p /usr/kafka
chmod 777 -R /usr/kafka
wget --no-check-certificate https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/3.2.0/kafka_2.12-3.2.0.tgz
tar -zvxf kafka_2.12-3.2.0.tgz -C /usr/kafka

# broker configuration (example)
broker.id=1
listeners=PLAINTEXT://ip:9092
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dir=/usr/kafka/logs
num.partitions=5
num.recovery.threads.per.data.dir=3
offsets.topic.replication.factor=2
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=3
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=zk1:2181,zk2:2181,zk3:2181
zookeeper.connection.timeout.ms=30000
group.initial.rebalance.delay.ms=0

nohup /usr/kafka/kafka_2.12-3.2.0/bin/kafka-server-start.sh /usr/kafka/kafka_2.12-3.2.0/config/server.properties > /usr/kafka/logs/kafka.log 2>&1 &

# basic commands
$KAFKA_HOME/bin/kafka-topics.sh --list --bootstrap-server ip:9092
$KAFKA_HOME/bin/kafka-console-consumer.sh --bootstrap-server ip:9092 --topic test --from-beginning
$KAFKA_HOME/bin/kafka-topics.sh --create --bootstrap-server ip:9092 --replication-factor 2 --partitions 3 --topic xxx_data

3. Filebeat

sudo rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
# create /etc/yum.repos.d/elastic.repo with:
[elastic-8.x]
name=Elastic repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

yum install filebeat
systemctl enable filebeat
chkconfig --add filebeat

# /etc/filebeat/filebeat.yml (excerpt)
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /root/logs/xxx/inner/*.log
  json:
    keys_under_root: true
output.kafka:
  hosts: ["kafka1:9092", "kafka2:9092", "kafka3:9092"]
  topic: 'xxx_data_clickhouse'
  partition.round_robin:
    reachable_only: false
    required_acks: 1
    compression: gzip
processors:
- drop_fields:
    fields: ["input", "agent", "ecs", "log", "metadata", "timestamp"]
    ignore_missing: false

nohup ./filebeat -e -c /etc/filebeat/filebeat.yml > /user/filebeat/filebeat.log &

4. ClickHouse

yum install yum-utils
rpm --import https://repo.clickhouse.tech/CLICKHOUSE-KEY.GPG
yum-config-manager --add-repo https://repo.clickhouse.tech/rpm/stable/x86_64
yum list | grep clickhouse
yum -y install clickhouse-server clickhouse-client

# edit /etc/clickhouse-server/config.xml, set <level>information</level>
# start/stop commands
sudo clickhouse stop
sudo clickhouse start

# performance tuning (example)
echo 'performance' | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
echo 0 | tee /proc/sys/vm/overcommit_memory
echo 'never' | tee /sys/kernel/mm/transparent_hugepage/enabled

# create data directory
mkdir -p /data/clickhouse
# add hosts entries for clickhouse nodes
# ... (example IPs omitted)

Common Issues and Solutions

Kafka engine table direct select

Direct select is not allowed. To enable use setting stream_like_engine_allow_direct_select.

Solution: start the client with the flag:

clickhouse-client --stream_like_engine_allow_direct_select 1 --password xxxx

Macro configuration error

Code: 62. DB::Exception: No macro 'shard' in config while processing substitutions.

Solution: define distinct

<macros><shard>01</shard><replica>example01-01-1</replica></macros>

on each node.

Replica already exists

Code: 253. DB::Exception: Replica ... already exists.

Solution: delete the corresponding Zookeeper node and recreate the ReplicatedReplacingMergeTree table.

Distributed table authentication failure

Code: 516. DB::Exception: Authentication failed: password is incorrect.

Solution: configure correct <user>default</user> and <password>xxxx</password> in <remote_servers> section.

Materialized View

CREATE MATERIALIZED VIEW default.view_bi_inner_log ON CLUSTER clickhouse_cluster TO default.bi_inner_log_all AS
SELECT log_uuid, date_partition, event_name, activity_name, credits_bring, activity_type, activity_id
FROM default.kafka_clickhouse_inner_log;

Conclusion

The CK+Kafka+Filebeat combination provides higher write throughput, faster queries, and lower server cost than a traditional ELK stack. By following the detailed deployment steps and addressing the listed pitfalls, teams can build a reliable, cost‑effective private analytics platform.