Design and Implementation of a Custom HttpAgent for a Complex Test Environment

The new test environment deploys a dedicated HttpAgent on each test server, managed by the eig testing platform, to receive, parse, and respond to commands, enabling concurrent scheduling, asynchronous script execution, and extensive customization for the highly complex multi‑business setup.

Overview

The agent functions as a lightweight web service built with Python 2.7 and Tornado, exposing a unified listening port. It receives execution requests from the platform, parses them into subtasks, runs the corresponding scripts, and returns results and logs. Additionally, it gathers server performance metrics, monitors service status, and collects JaCoCo coverage data.

Advantages

Reduces connection overhead compared to SSH, which requires key authorization and Kerberos trust.

HTTP‑based command reception offers better concurrency control and easier management.

Supports self‑updates, automatic restarts, and process guarding for reliability.

Tightly integrates with servers to perform information collection and scheduled tasks, providing higher customizability.

Agent Architecture

The agent is divided into install, agent, config, modules, script, util, console, and test components. This article focuses on the HTTP web server portion, laying the groundwork for future deep dives into lower‑level modules.

Initial Installation, Pre‑installation, and Self‑Update

1. Pre‑installation : An install.sh script performs a one‑click installation, bundling the main program, all dependencies, third‑party packages, and initializing scheduled tasks, registration, and process guarding. Supervisor is started within the script to monitor the registration process.

2. Self‑Update and Auto‑Restart : The agent is a Git‑managed project with a cron job that checks for updates every minute. Upon detecting new commits, the update is triggered, all running agent processes are terminated, and the supervisor automatically restarts them.

Agent Workflow

The agent supports both synchronous and asynchronous invocations:

Heavy tasks such as deployment, initialization, or recycling are executed in separate threads after the agent returns an execution status, with results and logs sent upon completion.

Lightweight tasks like authorization, deletion, or JaCoCo data collection are executed immediately after parsing, and results are returned instantly.

It also handles high concurrency by employing a thread pool that processes queued synchronization scripts, preventing task loss during burst executions.

Process Guarding

Supervisor monitors the agent and log‑upload processes, automatically restarting them if they crash. The configuration is as follows:

[program:tornado-9889]
command=python2.7 /opt/deploy/agent/HttpAgent.py --port=9889 ; process start command
autorestart=true ; restart on failure
redirect_stderr=true
stdout_logfile_maxbytes=50MB ; max log size
stdout_logfile_backups=20 ; number of log backups
stderr_logfile=/opt/log/supervisord/tornado-stderr.log ; supervisor log
stdout_logfile=/opt/log/supervisord/http_agent.log ; agent log
loglevel=info

Agent Monitoring

Multi‑layer monitoring ensures agent reliability:

Supervisor provides basic process guarding.

A cron‑based check verifies supervisor status each minute during auto‑updates, restarting it if necessary.

The AgentMonitor process runs scheduled tasks to check the status of HttpAgent and CronUpload processes, sending email alerts on failures.

AgentMonitor stores timestamps in Redis to detect its own failures.

AgentMonitor also collects server performance data stored in Redis, including CPU usage (from top ), memory info, disk I/O, and network rates (from /proc/meminfo , /proc/diskstats , /proc/net/dev ).

Future Work

This article covered only the upper‑level operation mode of the agent. Upcoming posts will explore the underlying modules responsible for deployment, synchronization, and configuration file templating.