How to Build a Lightweight CI/CD Service Automation with DevOps and Docker

1. Introduction and Background

1.1 Evolution of Development Models

(1) Traditional Development Model

In the traditional model, development, operations, and physical machines are tightly coupled; after development, operations deploy the project to a single physical machine.

Because services depend heavily on the physical machine environment, swapping machines requires reinstalling dependencies, which is cumbersome.

(2) The Container Revolution

Virtual machines were introduced to solve this, but were quickly replaced by lightweight OS container products like Docker.

Docker allows packaging service dependencies into an image and running them in containers, decoupling services from physical machines. Developers can script operations in a Dockerfile, breaking down the Dev‑Ops barrier and reducing deployment cost.

(3) Agile Development Innovation

Agile splits product delivery into short cycles, iterating quickly to meet market needs and reduce risk.

(4) DevOps Cultural Movement

Container, agile, and CI/CD technologies fostered the DevOps movement, emphasizing communication among development, testing, and operations. More teams now invest in DevOps, using CI/CD for automated, high‑quality, and standardized development.

2. Characteristics of the Evolution

Development, testing, and operations are no longer isolated; they are increasingly intertwined, driving automation in the development process.

3. Why Write This Article

Current work faces efficiency and quality issues. Existing CI/CD mechanisms are costly and hard to extend, prompting a DevOps‑driven, CI‑based server automation solution that closes the loop for development, testing, deployment, monitoring, and delivery.

4. What This Article Covers

It focuses on CI/CD as the core, explaining its benefits, why a custom CI server automation is needed, and how to design it.

5. Reading Precautions

CI/CD will be referred to as CI in the text.

Typos or misunderstandings are welcome for correction.

2. Problem Status

1) Fast Project Iteration

Agile shortens development cycles, compressing requirements, testing, and other phases, leading to rapid iteration.

Example: a feature may be released without testing after product acceptance.

2) Heavy Third‑Party Tools

Many third‑party tools are bulky, offering many features but requiring unnecessary resources, unsuitable for fast‑iteration projects.

Example: API documentation generators often need a private server.

3) Tight Testing Schedule

Testing time is limited, especially with multi‑platform testing and bug verification, making it hard to ensure quality.

Example: QA discovers issues late because developers omitted unit tests.

4) Slow Requirement Fulfilment

Even with fast agile cycles, meeting QA, Ops, or SRE requests can lag behind iteration speed.

Example 1: Automated test support requires QA scheduling. Example 2: Configuration checks for panic prevention need SRE scheduling.

3. Solution

Without a complete mechanism, handling fast agile iteration is difficult and quality suffers. Applying DevOps and CI, we built a comprehensive server automation that breaks departmental silos and meets speed and quality demands.

4. Practice

1) Design Scheme

(1) Lightweight Design

The primary goal is lightweight—removing unnecessary details while meeting requirements with minimal cost and rapid delivery.

Only rely on existing GitLab Runner machines; no extra servers.

No extra backend management system; use GitLab repository and store API docs in /apidoc.md and Wiki.

Third‑party caches are kept under 50 MB, most tools stay below 10 MB.

(2) Multi‑Project Sharing

In micro‑service architecture, each codebase has its own CI scripts, causing large coordination overhead when any CI changes. We moved CI logic to a shared CommonCI repository and launch it via start.sh .

(3) Plug‑in Tasks

All tasks are plug‑in, extensible, and fully controllable, enabling complex task orchestration.

(4) Third‑Party Cache

GitLab CI’s built‑in cache and artifacts have limitations, so we built a custom cache library to store compiled binaries and speed up CI.

2) Technologies and Ideas

(1) Technologies Used

GitLab stack: GitLab CI and API for pipeline automation.

Shell scripting: for task plug‑in and orchestration.

Go: for configuration checks, dependency checks, API doc generation, etc.

Container stack: Docker for future containerized service management.

(2) DevOps Thought

Why build server automation? What does it achieve? How to implement? All driven by DevOps.

3) Overall Architecture

The CI‑based server automation consists of three layers:

Code Repository Layer – business and common repositories.

Runner Layer – GitLab Runner servers that execute CI.

CI Automation Layer – abstracted plug‑in tasks forming the pipeline.

4) Code Repository Layer

Besides business code, a common CommonCI repository provides shared CI scripts:

/.gitlab – common methods and CI task directories.

/start.sh – launches CI by executing all *.sh scripts under /.gitlab .

5) CI Automation Layer

(1) Pre‑Installation

Pre‑install scripts are placed in .gitlab/pre_install.sh .

<code>echo -e "------------ 开始运行预安装 ------------"
EveryStageCommonOperation
# check and install inkedep tool
if ! which "inkedep" >/dev/null 2>&1; then
  echo -e "inkedep不存在, 正在安装 ..."
  go get git.inke.cn/BackendPlatform/inkedep && cd $GOPATH/src/git.inke.cn/BackendPlatform/inkedep && go install && inkedep ver && cd $currentDir
else
  echo -e "inkedep已安装"
fi
# copy project to $PROJECT_PATH
mkdir -p $PROJECT_PATH && cp -r ./ $PROJECT_PATH && cd $PROJECT_PATH
# build project
cd $APP_PATH && inkedep save && inkedep build && cd $PROJECT_PATH
# fetch other dependencies
inkedep get code.inke.cn/lingxi/server/metis/metis.common.tools
echo -e "------------ 预安装运行结束 ------------"
</code>

(2) Code Check

We use golangci-lint to catch most code errors.

<code>echo -e "------------ 开始运行代码检查 ------------"
if [ -z "$CHECK_CODE_SWITCH" ] || [ "$CHECK_CODE_SWITCH" = "off" ]; then
  echo -e "因关闭代码检查, 已跳过check_code.sh"
  exit 0
fi
EveryStageCommonOperation
# install golangci-lint if missing
if ! which "$GOPATH/bin/golangci-lint" >/dev/null 2>&1; then
  echo -e "golangci-lint不存在, 正在安装 ..."
  curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(go env GOPATH)/bin v1.46.2
else
  echo -e "golangci-lint已安装"
fi
# run lint
$(go env GOPATH)/bin/golangci-lint run --timeout=10m
if [ $? -ne 0 ]; then
  FAILURE_REASON="代码检查失败"
  SendFailureNotice
  exit 1
fi
echo -e "------------ 代码检查运行结束 ------------"
</code>

(3) Config Check

A lightweight config‑check tool compares test and online configs to prevent missing configurations that cause panics.

<code>echo -e "------------ 开始运行配置检查 ------------"
if [ -z "$UNIT_TEST_SWITCH" ] || [ "$UNIT_TEST_SWITCH" = "off" ]; then
  echo -e "因关闭配置检查, 已跳过config_check.sh"
  exit 0
fi
# install cdiff if missing
if ! which "./cdiff" >/dev/null 2>&1; then
  echo -e "cdiff工具不存在，正在安装 ..."
  go get code.inke.cn/ConfigDiff && cd $GOPATH/src/code.inke.cn/ConfigDiff && go build -o cdiff main.go
else
  echo -e "cdiff工具已安装"
fi
result=$(./cdiff --testConfigPath=$TEST_CONFIG_FILE --onlineConfigPath=$ONLINE_CONFIG_FILE | grep 'Unfortunately!  There are some difference as follows')
if [ -n "$result" ]; then
  ./cdiff --testConfigPath=$TEST_CONFIG_FILE --onlineConfigPath=$ONLINE_CONFIG_FILE
  FAILURE_REASON="配置检查失败"
  SendFailureNotice
  exit 1
fi
echo -e "------------ 配置检查运行结束 ------------"
</code>

(4) Dependency Check

Parses Godeps.json to find wrong dependencies across projects.

<code>echo -e "------------ 开始运行依赖检查 ------------"
if [ -z "$UNIT_TEST_SWITCH" ] || [ "$UNIT_TEST_SWITCH" = "off" ]; then
  echo -e "因关闭依赖检查, 已跳过godep_check.sh"
  exit 0
fi
# install checkdep if missing
if ! which "./checkdep" >/dev/null 2>&1; then
  echo -e "checkdep工具不存在，正在安装中 ..."
  go get code.inke.cn/checkGodeps && cd $GOPATH/src/code.inke.cn/checkGodeps && go build -o checkdep main.go
else
  echo -e "checkdep工具已安装"
fi
result=$(./checkdep --goDepsFile=$GODEPS_FILE --pattern $GODEPS_PATTERN | grep '==import wrong project==')
if [ -n "$result" ]; then
  ./checkdep --goDepsFile=$GODEPS_FILE --pattern $GODEPS_PATTERN
  FAILURE_REASON="依赖检查失败"
  SendFailureNotice
  exit 1
fi
echo -e "------------ 依赖检查运行结束 ------------"
</code>

(5) Unit Test

Extensive unit tests are written and integrated into CI to avoid hidden bugs.

<code>echo -e "------------ 开始运行单元测试 ------------"
if [ -z "$UNIT_TEST_SWITCH" ] || [ "$UNIT_TEST_SWITCH" = "off" ]; then
  echo -e "因关闭单元测试, 已跳过unit_test.sh"
  exit 0
fi
EveryStageCommonOperation
echo "1. 执行单元测试"
eval $UNIT_TEST_TRIGGER_CMD
if [ $? -ne 0 ]; then
  FAILURE_REASON="单元测试失败"
  SendFailureNotice
  exit 1
fi
echo -e "------------ 单元测试运行结束 ------------"
</code>

(6) Local Build

Builds the project locally, packaging and uploading artifacts.

<code>cd $PROJECT_PATH
currentDir=$(pwd)
. $currentDir/.gitlab/include/function.sh
if [ -z "$LOCAL_BUILD_SWITCH" ] || [ "$LOCAL_BUILD_SWITCH" = "off" ]; then
  echo -e "因关闭本地构建, 已跳过local_build.sh"
  exit 0
fi
EveryStageCommonOperation
eval $LOCAL_BUILD_TRIGGER_CMD
if [ $? -ne 0 ]; then
  FAILURE_REASON="本地构建失败"
  SendFailureNotice
  exit 1
else
  echo -e "本地构建成功"
fi
echo -e "------------ 本地构建运行结束 ------------"
</code>

(7) Health Check (Startup Self‑Check)

After deployment, the service is started and health‑checked repeatedly until success.

<code>if [ -z "$HEALTH_CHECK_SWITCH" ] || [ "$HEALTH_CHECK_SWITCH" = "off" ]; then
  echo -e "因关闭启动自检, 已跳过health_check.sh"
  exit 0
fi
i=0
while true; do
  ((i++))
  res=$(eval $HEALTH_CHECK_TRIGGER_CMD)
  if [ "$res" = "$HEALTH_CHECK_SUCCESS" ]; then
    echo -e "健康检查成功"
    break
  fi
  if [ $i -ge 100 ]; then
    FAILURE_REASON="启动自检失败: $res"
    SendFailureNotice
    exit 1
  fi
  sleep 1
done
echo -e "------------ 启动自检运行结束 ------------"
</code>

(8) API Documentation Generation

A lightweight Go tool generates API docs from models, storing them in GitLab Wiki without a heavy server.

<code>echo -e "------------ 开始运行接口文档自动生成 ------------"
if [ -z "$APIDOC_SWITCH" ] || [ "$APIDOC_SWITCH" = "off" ]; then
  echo -e "因关闭文档自动生成, 已跳过apidoc_gen.sh"
  exit 0
fi
EveryStageCommonOperation
eval $APIDOC_TRIGGER_CMD && cd $currentDir
content=$(cat $APIDOC_FILE)
if [ -z "$content" ]; then
  echo -e "接口文档内容为空或生成失败"
  FAILURE_REASON="接口文档自动生成失败"
  SendFailureNotice
  exit 1
fi
curl --request PUT --data "format=markdown&content=$content&title=APIDoc" \
  --header "PRIVATE-TOKEN: $GITLAB_API_TOKEN" \
  "https://code.inke.cn/api/v4/projects/$PROJECT_ID/wikis/APIDoc"
echo -e "------------ 接口文档自动生成运行结束 ------------"
</code>

(9) API Automated Testing

Go‑based API test suite runs after deployment to ensure end‑to‑end functionality.

<code>if [ -z "$API_TEST_SWITCH" ] || [ "$API_TEST_SWITCH" = "off" ]; then
  echo -e "因关闭接口自动化测试, 已跳过api_test.sh"
  exit 0
fi
EveryStageCommonOperation
eval $API_TEST_TRIGGER_CMD
if [ $? -ne 0 ]; then
  FAILURE_REASON="接口自动化测试失败"
  SendFailureNotice
  exit 1
else
  echo -e "接口自动化测试成功"
fi
echo -e "------------ 接口自动化测试运行结束 ------------"
</code>

(10) Adding/Removing Tasks

To add a new task, place its script (e.g., auto_deploy.sh ) and reference it in start.sh .

6) Project Integration

Previously, CI integration required complex .gitlab-ci.yml scripts. To lower entry cost, we provide a minimal template that only needs a simple configuration file.

<code>image: golang:1.17
variables:
  GITLAB_HOST: ""
  GITLAB_API_TOKEN: ""
  PROJECT_NAME: "apimock-example"
  PROJECT_ID: 48845
  DING_KEYWORD: "apimock-example"
  DING_ACCESS_TOKEN: ""
  DING_NOTICE_SWITCH: "off"
  CHECK_CODE_SWITCH: "on"
  UNIT_TEST_TRIGGER_CMD: "cd mock && go test -v . && cd .. && cd service && go test -v . && cd .."
  UNIT_TEST_SWITCH: "on"
  APIDOC_TRIGGER_CMD: "cd mock && go test -v . && cd .."
  APIDOC_FILE: "apidoc.md"
  APIDOC_SWITCH: "off"
  LOCAL_BUILD_TRIGGER_CMD: "go mod download && go build -o project && nohup ./project &"
  LOCAL_BUILD_SWITCH: "on"
  HEALTH_CHECK_TRIGGER_CMD: "curl -X GET 127.0.0.1:8000/ping"
  HEALTH_CHECK_SUCCESS: "ok"
  HEALTH_CHECK_SWITCH: "on"
before_script:
  - echo '====== CIManager Start Running ======'
after_script:
  - echo '====== CIManager Stopped Successfully ======'
stages:
  - CIManager
CIManager:
  stage: CIManager
  script:
    - git clone -b testing https://github.com/wgrape/CIManager.git ; cp -an ./CIManager/. ./ ; rm -rf ./CIManager ; bash start.sh
</code>

7) Cache‑Driven Performance Boost

Third‑party cache reduced CI runtime from 5‑10 minutes to about 1 minute, a five‑fold speedup.

5. Goals

After practice, we set two main goals:

1) Stability

Avoid missing configurations that cause panics.

Prevent uninitialized resources.

Ensure code changes don’t introduce regressions.

Catch low‑level errors before deployment.

2) Efficiency

Eliminate testing bottlenecks.

Improve front‑end/back‑end coordination via reliable API docs.

6. Review and Summary

We reviewed the need for a CI‑based server automation in fast agile cycles, the DevOps‑driven solution, lightweight design, and how we tackled multi‑project sharing, speed, and extensibility. The presented approach offers a practical, low‑cost way to boost development speed and quality.