How to Build a Serverless Log Processing Pipeline with OpenFunction and Kafka

Overview

When container logs are sent to a message server, processing them with a dedicated workload can be costly and hard to scale. This article presents a serverless log‑processing solution that reduces cost and improves flexibility.

Design Steps

Deploy a Kafka cluster in the Kubernetes cluster to receive logs.

Deploy OpenFunction to provide serverless capabilities.

Write a log‑handling function that extracts 404 events and sends alerts.

Configure Notification Manager to forward alerts to Slack.

Kafka is installed using the Strimzi Kafka Operator with an ephemeral storage configuration and a logs topic.

helm repo add strimzi https://strimzi.io/charts/
helm install kafka-operator -n default strimzi/strimzi-kafka-operator
# create Kafka cluster and topic
cat <<EOF | kubectl apply -f -
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka-logs-receiver
  namespace: default
spec:
  kafka:
    version: 2.8.0
    replicas: 1
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      log.message.format.version: '2.8'
      inter.broker.protocol.version: "2.8"
    storage:
      type: ephemeral
  zookeeper:
    replicas: 1
    storage:
      type: ephemeral
  entityOperator:
    topicOperator: {}
    userOperator: {}
---
apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaTopic
metadata:
  name: logs
  namespace: default
  labels:
    strimzi.io/cluster: kafka-logs-receiver
spec:
  partitions: 10
  replicas: 3
  config:
    retention.ms: 7200000
    segment.bytes: 1073741824
EOF

After the Kafka cluster is ready, enable the KubeSphere logging component and add the Kafka receiver through the UI.

Deploy OpenFunction

Install OpenFunction and its dependencies with the prerequisite script, then deploy the function using a Function CRD that references the Kafka receiver as input and the Notification Manager as output.

sh hack/deploy.sh --with-shipwright --with-openFuncAsync --poor-network
kubectl apply -f https://raw.githubusercontent.com/OpenFunction/OpenFunction/main/config/strategy/openfunction.yaml
kubectl apply -f https://github.com/OpenFunction/OpenFunction/releases/download/v0.3.0/bundle.yaml

The Go function logs‑handler parses log entries, matches 404 responses, extracts namespace, pod name, HTTP method and path, builds a Prometheus Alertmanager payload and sends it to the notification‑manager endpoint.

package logshandler
import (
    "encoding/json"
    "fmt"
    "log"
    "regexp"
    "time"
    ofctx "github.com/OpenFunction/functions-framework-go/openfunction-context"
    alert "github.com/prometheus/alertmanager/template"
)

func LogsHandler(ctx *ofctx.OpenFunctionContext, in []byte) int {
    // implementation omitted for brevity – parses the log, matches 404, builds alert, sends via ctx.SendTo
    return 200
}

Configure the Function CRD to use the OpenFuncAsync runtime, bind the Kafka topic logs as input and the HTTP binding of the Notification Manager as output. KEDA scales the function based on the Kafka lag threshold.

apiVersion: core.openfunction.io/v1alpha1
kind: Function
metadata:
  name: logs-handler
spec:
  version: "v1.0.0"
  image: openfunctiondev/logs-async-handler:v1
  imageCredentials:
    name: push-secret
  build:
    builder: openfunctiondev/go115-builder:v0.2.0
    srcRepo:
      url: "https://github.com/OpenFunction/samples.git"
      sourceSubPath: "functions/OpenFuncAsync/logs-handler-function/"
  serving:
    runtime: "OpenFuncAsync"
    openFuncAsync:
      dapr:
        inputs:
          - name: kafka-receiver
            type: bindings
        outputs:
          - name: notification-manager
            type: bindings
            params:
              operation: "post"
              type: "bindings"
        components:
          - name: kafka-receiver
            type: bindings.kafka
            metadata:
              - name: brokers
                value: "kafka-logs-receiver-kafka-brokers:9092"
              - name: topics
                value: "logs"
              - name: consumerGroup
                value: "logs-handler"
          - name: notification-manager
            type: bindings.http
            metadata:
              - name: url
                value: "http://notification-manager-svc.kubesphere-monitoring-system.svc.cluster.local:19093/api/v2/alerts"
      keda:
        triggers:
          - type: kafka
            metadata:
              topic: logs
              bootstrapServers: kafka-logs-receiver-kafka-brokers.default.svc.cluster.local:9092
              consumerGroup: logs-handler
              lagThreshold: "10"

When a 404 request is generated by the WordPress demo, the function sends an alert to Slack; scaling up and down can be observed by enabling or disabling the Kafka log receiver.

Conclusion

Serverless with OpenFunction simplifies log‑processing pipelines, reduces operational cost, and provides automatic scaling via KEDA, while keeping the configuration declarative and portable.