Mastering Modern Operations: Trends, Skill Maps, and Big Data Monitoring Strategies

Operations is a multidisciplinary technology field that combines networking, systems, development, security, application architecture, storage, and more. It has progressed from basic network management to specialized roles such as system operations engineer, network operations engineer, security operations engineer, and DevOps engineer, with increasing demands for comprehensive skills.

1. Operations Position Development and Trends

Based on domain, technical aspect, and process, the 2019 trends for operations positions are outlined.

1) By Domain

Infrastructure operations: IDC/network, server/storage maintenance

System operations: middleware, cloud platform maintenance

Data operations: database and big‑data platform maintenance

Application operations: application software systems

Cloud platform operations: public‑cloud platform maintenance

Container operations: operations based on container services

2) By Technical Aspect

Security operations

Performance operations

Data operations

Integration operations

3) By Process

Build/CI/release

Installation, deployment, upgrade, migration, merge, expansion

Configuration, initialization, configuration changes

Backup, transfer, recovery

Log, monitoring, alerting

Diagnosis, troubleshooting, optimization

2. System Operations Skill Map

System operations form the foundation of all operations work; mastering these skills is essential for further learning.

3. Web Operations Skill Map

Web operations is the most common and well‑paid operations role, requiring a broad set of knowledge.

4. Big Data Operations Skill Map

Since 2017, big data has become pervasive, and in 2019 its industry support continues to grow, making big‑data operations a frontier skill.

5. Container Operations Skill Map

Container technology sparked a revolution around 2015‑2016 and has since become mainstream in Chinese enterprises, with a thriving ecosystem of vendors, open‑source communities, and public clouds.

6. Data as the Foundation

Just as a skyscraper needs a solid foundation, operations data underpins all management activities. Operations data includes CMDB, logs, production databases, and knowledge bases.

CMDB (Configuration Management Database) stores asset and configuration information. Log data captures system, device, and database logs, representing core enterprise data. Database data covers production, test, and development databases. Knowledge base records events, problems, solutions, and best practices.

Maintaining and managing this data—especially logs—is crucial for diagnosing issues, tracing root causes, and predicting potential failures.

1) System Logs

System logs (e.g., /var/log) include operation, security, and cron logs, forming the audit basis for security monitoring. They often require custom retention policies and protection against deletion.

2) Application Logs

Application logs record service health and business operations, providing data for performance analysis and audit trails.

3) Database Logs

Database logs reflect the state of DBMSs such as Oracle (v$ views) or MySQL (performance_schema), enabling proactive monitoring and issue resolution.

4) Device Logs

Device logs from switches, firewalls, and network security appliances reveal hardware and network health, essential for preventing widespread outages.

Collecting, filtering, analyzing, and visualizing logs can be achieved with the ELK stack (Elasticsearch, Logstash, Kibana) and Beats tools.

Elasticsearch: distributed search engine for storing and querying log data. Logstash: pipeline for collecting, filtering, and forwarding logs. Kibana: web UI for visualizing and reporting on log data.

Beats (Filebeat, Packetbeat, Topbeat, Winlogbeat) extend log collection to files, network traffic, system metrics, and Windows event logs.

Three common ELK architectures are presented:

This simple architecture places Logstash on each node, sending filtered data directly to Elasticsearch. It is easy to set up but consumes significant resources and lacks buffering.

The second architecture introduces a message queue (Kafka or Redis) between Logstash agents and a central Logstash server, providing fault tolerance and better load distribution.

The third architecture replaces Logstash agents with Filebeat, uses a Kafka cluster for buffering, and runs both Logstash and Elasticsearch in clustered mode, suitable for large‑scale, high‑throughput environments.

7. Applying Big Data Thinking to Operations Monitoring

Big data analysis originated from operations log analysis and now drives business insights. Metrics derived from big data include business‑level indicators (e.g., transaction rates), application‑level indicators (error counts, latency), system‑resource indicators (CPU, memory, disk), and network indicators (packet loss, latency).

These metrics enable health monitoring, root‑cause analysis, performance tuning, and security tracking.

The three‑step big‑data‑driven monitoring process is: acquire required data, filter anomalies and set alert thresholds, and trigger alerts via third‑party monitoring platforms.

Ultimately, consolidating log analysis into a unified platform allows operations teams to define analysis logic once and reap continuous, automated insights.