How to Build Low-Cost Automated Operations with Prometheus, Ansible, and Jenkins

First, Set Up Monitoring and Alerts

Monitoring and alerting should be the initial focus even if it slows down feature development, because knowing the current state is essential for planning.

Among many monitoring solutions (Zabbix, Open‑Falcon, Prometheus), the author chose Prometheus for its pull model, text‑based configuration that supports versioning, extensive plugins, and alignment with Google SRE recommendations.

It uses a pull model.

Configuration is text‑based, facilitating version control.

Plenty of ready‑made exporters for various metrics.

Learning Prometheus aligns with SRE best practices.

Prometheus deployment is automated with Ansible + Git. The architecture includes:

Prometheus Server for data collection and storage.

Prometheus Alertmanager for rule‑based alerts, integrable with many channels.

node‑exporter to expose host metrics via HTTP.

Installation uses the prometheus‑ansible role.

Grafana is added for visualization, integrating tightly with Prometheus.

Alerting is configured via Prometheus' built‑in channels; for DingTalk integration, the open‑source prometheus-webhook-dingtalk component is used.

Configuration Versioning from the Start

All configuration files are stored in a separate Git repository, enabling versioned deployment. Ansible’s hierarchical variable system provides flexibility.

├── environments/
│   ├── dev/
│   │   ├── group_vars/
│   │   │   ├── all
│   │   │   ├── db
│   │   │   └── web
│   │   └── hosts
│   ├── prod/
│   │   ├── group_vars/
│   │   │   ├── all
│   │   │   ├── db
│   │   │   └── web
│   │   └── hosts
│   └── stage/
│       ├── group_vars/
│       │   ├── all
│       │   ├── db
│       │   └── web
│       └── hosts

Future migration to Consul as a configuration center is straightforward because Ansible 2.0+ includes native Consul support.

Jenkins Integration: Automating Builds

Jenkins automates project packaging. An existing Ansible role ( ansible-role-jenkins) handles Jenkins installation and plugin management via a variable jenkins_plugins list.

---
- hosts: all
  vars:
    jenkins_plugins:
      - blueocean
      - ghprb
      - greenballs
      - workflow-aggregator
    jenkins_plugin_timeout: 120
  pre_tasks:
    - include_tasks: java-8.yml
  roles:
    - geerlingguy.java
    - ansible-role-jenkins

After Jenkins is up, it is linked to an existing GitLab instance.

Two ways to define build steps are offered; the author prefers using a Jenkinsfile for version control and flexibility.

pipeline {
    agent any
    stages {
        stage('Build') {
            steps {
                sh './gradlew clean build'
                archiveArtifacts artifacts: '**/target/*.jar', fingerprint: true
            }
        }
    }
}

The Jenkinsfile resides alongside the source code of each project.

Running Ansible from Jenkins

Jenkins executes Ansible playbooks via the Ansible plugin. Credentials (SSH keys, Vault passwords) are injected using the Credentials Binding plugin.

withCredentials([sshUserPrivateKey(keyFileVariable:"deploy_private",credentialsId:"deploy"),
                file(credentialsId: 'vault_password', variable: 'vault_password')]) {
    ansiblePlaybook vaultCredentialsId: 'vault_password',
                    inventory: "environments/prod",
                    playbook: "playbook.yaml",
                    extraVars:[
                        ansible_ssh_private_key_file: [value:"${deploy_private}", hidden:true],
                        build_number: [value:"${params.build_number}", hidden:false]
                    ]
}

Explanation: ansiblePlaybook runs the playbook similarly to the CLI. withCredentials provides secure access to sensitive data.

Secrets are encrypted with Ansible Vault.

Organizing Ansible Scripts

Each project contains its own ansible directory with playbooks. During the build phase the directory is zipped; during deployment it is unpacked and executed.

Generating Boilerplate with Cookiecutter

To avoid repetitive manual work, the author uses cookiecutter to scaffold Jenkinsfiles and Ansible scripts for new projects.

Summary

The recommended progression for a small team is:

Implement basic monitoring.

Adopt GitLab for source control.

Deploy Jenkins and integrate with GitLab.

Use Jenkins for automated build and packaging.

Execute Ansible deployments via Jenkins.

From this foundation, teams can evolve toward enterprise‑grade capabilities such as CMDB generation (using ansible‑cmdb), blue‑green deployments, auto‑scaling via Prometheus alerts, and ChatOps.

Related links: https://github.com/prometheus/node_exporter https://github.com/ernestas-poskus/ansible-prometheus https://github.com/timonwong/prometheus-webhook-dingtalk https://www.digitalocean.com/community/tutorials/how-to-manage-multistage-environments-with-ansible http://docs.ansible.com/ansible/latest/modules/consul_module.html https://github.com/geerlingguy/ansible-role-jenkins https://jenkins.io/doc/book/pipeline/jenkinsfile/ https://wiki.jenkins.io/display/JENKINS/Ansible+Plugin https://jenkins.io/doc/pipeline/steps/credentials-binding/ http://docs.ansible.com/ansible/2.5/user_guide/vault.html https://github.com/audreyr/cookiecutter https://github.com/fboender/ansible-cmdb https://showme.codes/2017-10-08/chatops-in-action/