Master Real-Time Log Collection with LogHub: Strategies for E‑Commerce Platforms

LogHub provides real‑time log collection and consumption, supporting more than 30 ingestion methods; this article introduces the collection approaches for various scenarios.

Background

"I want to order takeout" is a platform‑type e‑commerce site with users, restaurants, and delivery personnel. Users can place orders via web, app, WeChat, Alipay, etc.; merchants receive orders, start preparation, and automatically notify nearby couriers, who deliver the food.

Operational Requirements

Acquiring users is costly; advertising spend cannot be evaluated per channel.

Customers complain about slow delivery, but the bottleneck (order taking, dispatch, cooking) is unclear.

Promotional activities (coupons) lack measurable impact.

During peak periods, merchants need advance stock preparation and more couriers in specific areas.

Customer service needs to trace order‑failure causes and system errors.

Data Collection Challenges

In data‑driven operations, the first step is consolidating scattered log data, which faces several challenges:

Multiple channels (advertising, flyers, etc.).

Various terminals (web, public accounts, mobile, browsers).

Heterogeneous networks (VPC, self‑built IDC, Alibaba Cloud ECS).

Multiple programming languages (Java core system, Nginx front‑end, C++ payment backend).

Diverse devices (X86, ARM).

LogHub can now unify log collection, replacing many manual steps.

Unified Log Management & Configuration

Create a log project, e.g., myorder .

Create logstores for each data source, such as:

wechat-server (WeChat server access logs)

wechat-app (WeChat app logs)

wechat-error (error logs)

alipay-server

alipay-app

deliver-app (courier app status)

deliver-error (error logs)

web-click (H5 page clicks)

server-access (server access logs)

server-app (application logs)

coupon (coupon usage logs)

pay (payment logs)

order (order logs)

If raw data needs cleaning or ETL, create intermediate logstores (see data cleaning & ETL).

(More operations can be found in the quick start/management console.)

User Promotion Log Collection

To acquire new users, two common methods are used:

Issue coupons directly during website registration.

Provide QR codes on flyers or web pages for users to scan and register.

Define a registration server address and generate a QR code. When a user scans and registers, the source is recorded in the log.

http://examplewebsite/login?source=10012&ref=kd4b

When the server receives the request, it outputs a log line like:

2016-06-20 19:00:00 e41234ab342ef034,102345,5k4d,467890

time: registration time

session: browser session for behavior tracking

source: channel identifier (e.g., 10001 for campaign A, 10002 for flyer)

ref: referral code, empty if none

params: other parameters

Collection methods:

Application writes logs to disk, collected by Logtail.

Application writes logs via SDK (see SDK documentation).

Server Data Collection

Alipay/WeChat public account programming follows a typical web model with three log types (illustrated in the image).

Methods:

Write logs to local files and configure Logtail with regex to route to specific logstores.

In Docker, use the container service’s integrated log service.

Java programs can use Log4j Appender (no disk write) or LogHub Producer Library for high‑concurrency writes.

C#, Python, Java, PHP, C, etc., can use the respective SDKs.

Windows servers can collect logs via Logstash.

End‑User Log Ingestion

Mobile: use mobile SDKs for iOS, Android, or MAN (Mobile Analytics) to ingest logs.

ARM devices: native C cross‑compiled.

Merchant platform devices: X86 devices use SDKs; ARM devices use native C.

Web/M Site User Behavior

User behavior collection on pages falls into two categories:

Interactions with backend servers (order, login, logout, etc.).

Front‑end only actions (scrolling, closing page, etc.).

Methods:

For the first category, follow server‑side collection methods.

For the second, use a tracking pixel or JavaScript library (see Tracking Web API).

Server Log Operations

(Illustrative diagram of server log operations.)

Data Collection in Different Network Environments

LogHub provides access points in each region with three connection options:

Intranet (classic): best bandwidth within the same region (recommended).

Internet (classic): accessible from anywhere; performance depends on link quality; use HTTPS for security.

Private network (VPC): access within the region’s VPC.

Refer to network access documentation for the most suitable option.

Other

See the full LogHub collection methods.

Refer to real‑time consumption, covering stream processing, data cleaning, data warehousing, and indexing.

Author: Jian Zhi, Senior Alibaba Cloud Computing Expert, specialized in log analysis and processing. Original article: https://yq.aliyun.com/articles/59921