Master Git Ops: From Zero to Production-Ready IaC Version Control

Git Operations Practical: From Zero to Production-Ready IaC Version Control

Applicable Scenarios & Prerequisites

Applicable Scenarios : configuration management, IaC (Terraform/Ansible), script versioning, CI/CD pipelines, multi‑person ops collaboration

Operating Systems : RHEL/CentOS 7.x‑9.x, Ubuntu 18.04‑24.04

Permission Requirements : regular user (recommended), Git server permissions for repository management

Dependent Tools : git 2.20+, SSH client, GPG (optional signing)

Prerequisite Knowledge : basic Linux commands, text editor usage

Environment & Version Matrix

Component versions for RHEL/CentOS and Ubuntu/Debian, with recommended versions:

Git 2.40+ (LTS) recommended; RHEL/CentOS 2.31+/2.43+, Ubuntu/Debian 2.34+/2.43+

OpenSSH 8.0+ (RHEL/CentOS) / 8.2+ (Ubuntu/Debian); latest stable recommended

GPG 2.2.20+ / 2.2.27+; optional 2.3+

GitLab CE as hosting platform; GitLab 16.x recommended; Gitea 1.21+ as alternative

Quick Checklist

Installation & Initial Configuration (identity, editor, alias)

SSH Key Authentication (password‑less push)

Create Ops Repository (directory structure)

Common Workflow (commit/branch/merge)

Sensitive Data Management (.gitignore + git‑crypt)

Multi‑Environment Branch Strategy (dev/staging/prod)

Configuration File Templating (variable substitution)

Automated Deployment Integration (Git Hooks)

Backup & Restore Mechanism (bare repository backup)

Audit & Compliance (signed commits + log audit)

Implementation Steps

Step 1: Installation & Initial Configuration

RHEL/CentOS installation:

# RHEL 8/9 includes Git 2.31+
sudo yum install -y git

# CentOS 7 requires EPEL for newer version
sudo yum install -y epel-release
sudo yum install -y git222  # Git 2.22

Ubuntu/Debian installation:

# Ubuntu 22.04+ includes Git 2.34+
sudo apt update
sudo apt install -y git

# Install latest via PPA
sudo add-apt-repository ppa:git-core/ppa -y
sudo apt update
sudo apt install -y git

Verify installation:

git --version

Global configuration (required):

# User identity (affects commit logs)
git config --global user.name "Ops Team"
git config --global user.email "[email protected]"

# Default editor
git config --global core.editor vim

# Color output
git config --global color.ui auto

# Default branch name for new repos
git config --global init.defaultBranch main

# Pull strategy (rebase to avoid unnecessary merge commits)
git config --global pull.rebase true

View configuration:

git config --list --show-origin

Step 2: SSH Key Authentication

Generate SSH key pair:

# Recommended Ed25519 (high performance / security)
ssh-keygen -t ed25519 -C "[email protected]" -f ~/.ssh/id_ed25519_git

# Fallback RSA 4096 (compatible with older systems)
ssh-keygen -t rsa -b 4096 -C "[email protected]" -f ~/.ssh/id_rsa_git

Configure SSH client:

cat >> ~/.ssh/config <<'EOF'
Host gitlab.example.com
  HostName gitlab.example.com
  User git
  IdentityFile ~/.ssh/id_ed25519_git
  IdentitiesOnly yes
  StrictHostKeyChecking accept-new
EOF
chmod 600 ~/.ssh/config

Upload public key to Git platform:

# Display public key (copy to GitLab/GitHub settings)
cat ~/.ssh/id_ed25519_git.pub

Validate connection:

ssh -T [email protected]

Expected output:

Welcome to GitLab, @ops!

Step 3: Create Ops Repository

Standard directory layout:

mkdir -p ~/ops-repo && cd ~/ops-repo
git init
mkdir -p {ansible/{inventories,playbooks,roles},scripts/{backup,monitoring},terraform/{modules,environments},docs}
cat > README.md <<'EOF'
## Directory Overview
- `ansible/` : Ansible configuration management
- `scripts/` : Ops scripts (backup/monitoring/deployment)
- `terraform/` : Infrastructure as code
- `docs/` : Documentation & runbooks
EOF

# .gitignore (see Step 5)
cat > .gitignore <<'EOF'
*.key
*.pem
*.p12
*.env
*_rsa*
secrets.yml
vault-password.txt
*.log
*.tmp
*.swp
.terraform/
.terraform.lock.hcl
.vscode/
.idea/
EOF

git add .
git commit -m "Initial commit: initialize ops repository structure"
git remote add origin [email protected]:ops/ops-repo.git
git push -u origin main

Step 4: Common Workflow

Scenario 1: Modify Nginx configuration

Workflow:

# 1. Pull latest code
cd ~/ops-repo
git pull

# 2. Create feature branch
git checkout -b feature/nginx-tuning

# 3. Edit configuration file
vim ansible/playbooks/nginx-config.yml

# 4. Review changes
git diff

# 5. Stage changes
git add ansible/playbooks/nginx-config.yml

# 6. Commit with clear description
git commit -m "feat: optimize Nginx worker connections
- set worker_connections to 4096
- enable TCP_NODELAY
- target: production web cluster"

# 7. Push branch
git push -u origin feature/nginx-tuning

Code review & merge:

# Create Merge Request in GitLab, approve, then merge to main
git checkout main
git pull
git branch -d feature/nginx-tuning

Scenario 2: Hotfix

# 1. Create hotfix branch from prod
git checkout prod
git checkout -b hotfix/disk-alert-threshold

# 2. Quick fix
echo "DISK_THRESHOLD=85" > scripts/monitoring/disk-check.conf
git add scripts/monitoring/disk-check.conf
git commit -m "fix: lower disk alert threshold from 95% to 85%"

# 3. Merge back to prod
git checkout prod
git merge --no-ff hotfix/disk-alert-threshold

# 4. Sync to main
git checkout main
git merge --no-ff hotfix/disk-alert-threshold

# 5. Clean up
git branch -d hotfix/disk-alert-threshold

Scenario 3: View history & revert

# Graphical log
git log --oneline --graph --all --decorate

# File‑specific history
git log -p -- ansible/playbooks/nginx-config.yml

# Show specific commit
git show <commit-hash>

# Soft reset (keep changes)
git reset --soft HEAD~1

# Hard reset (dangerous)
git reset --hard <commit-hash>

# Revert a commit (creates opposite commit)
git revert <commit-hash>

Step 5: Sensitive Data Management

Option 1: .gitignore (prevention)

cat > .gitignore <<'EOF'
# Keys & certificates
*.key
*.pem
*.crt
*.p12
*.jks
*_rsa
*_ecdsa
id_*

# Passwords & tokens
*.env
.env.*
secrets.yml
credentials.yml
vault-password*
*_token
*_secret
*.password

# Cloud credentials
.aws/credentials
.azure/credentials
.kube/config

# Terraform state
*.tfstate
*.tfstate.*
.terraform/
.terragrunt-cache/

# Ansible Vault exceptions
! *vault*.yml

# Logs & temp files
*.log
*.tmp
*.bak
*.swp
*.swo
*~
.DS_Store

# IDE
.vscode/
.idea/
*.iml
EOF

Check for accidental sensitive files:

git status --ignored

Option 2: git‑crypt (encryption)

Install git‑crypt:

# RHEL/CentOS
sudo yum install -y git-crypt

# Ubuntu/Debian
sudo apt install -y git-crypt

Initialize encryption:

cd ~/ops-repo
git-crypt init

# .gitattributes to mark encrypted files
cat > .gitattributes <<'EOF'
secrets.yml filter=git-crypt diff=git-crypt
ansible/vault-password.txt filter=git-crypt diff=git-crypt
*.env filter=git-crypt diff=git-crypt
EOF

git add .gitattributes
git commit -m "chore: enable git-crypt for sensitive files"

Export decryption key (backup!):

git-crypt export-key /secure/path/git-crypt-key
chmod 400 /secure/path/git-crypt-key

Team members unlock:

git clone [email protected]:ops/ops-repo.git
cd ops-repo
git-crypt unlock /secure/path/git-crypt-key

Step 6: Multi‑Environment Branch Strategy

Recommended simplified GitFlow:

main (development)
 ├── staging (pre‑release)
 └── prod (production)

Create branches:

# From main create staging
git checkout -b staging main
git push -u origin staging

# From staging create prod
git checkout -b prod staging
git push -u origin prod

Workflow example:

# Development → Testing → Staging → Production
# 1. Merge dev into staging
git checkout staging
git merge --no-ff main
git push

# 2. After validation, merge into prod and tag
git checkout prod
git merge --no-ff staging
git tag -a v1.2.3 -m "Release 1.2.3: Nginx performance tuning"
git push --tags

Protect branches in GitLab (example): prod: only maintainers can push, MR approval required staging: developers can push main: all commits require MR

Step 7: Configuration File Templating

Use Jinja2 templates for environment‑specific parameters.

# ansible/templates/nginx.conf.j2
worker_processes {{ nginx_worker_processes }};
worker_connections {{ nginx_worker_connections }};

upstream backend {
    {% for host in backend_servers %}
    server {{ host }}:8080;
    {% endfor %}
}

Environment variable files:

# ansible/inventories/production/group_vars/all.yml
nginx_worker_processes: 8
nginx_worker_connections: 4096
backend_servers:
  - 10.0.1.10
  - 10.0.1.11
  - 10.0.1.12

# ansible/inventories/staging/group_vars/all.yml
nginx_worker_processes: 2
nginx_worker_connections: 1024
backend_servers:
  - 10.0.2.10

Version‑control points:

Commit template files to repository

Separate environment variable files per directory

Encrypt production secrets with Ansible Vault

Step 8: Automated Deployment Integration (Git Hooks)

Server‑side post‑receive hook (trigger on prod push):

# /var/opt/gitlab/git-data/repositories/ops/ops-repo.git/hooks/post-receive
#!/bin/bash
while read oldrev newrev refname; do
  if [[ "$refname" == "refs/heads/prod" ]]; then
    echo "Detected prod branch update, triggering deployment..."
    DEPLOY_DIR="/tmp/deploy-$(date +%s)"
    git clone /var/opt/gitlab/git-data/repositories/ops/ops-repo.git "$DEPLOY_DIR"
    cd "$DEPLOY_DIR"
    git checkout prod
    ansible-playbook -i ansible/inventories/production ansible/playbooks/deploy.yml
    rm -rf "$DEPLOY_DIR"
  fi
done

Client‑side pre‑commit hook (checks before commit):

# ~/ops-repo/.git/hooks/pre-commit
#!/bin/bash
# Prevent committing secret files
FORBIDDEN_FILES=$(git diff --cached --name-only | grep -E '\.(key|pem|env)$')
if [ -n "$FORBIDDEN_FILES" ]; then
  echo "Error: Sensitive files detected, commit rejected!"
  echo "$FORBIDDEN_FILES"
  exit 1
fi

# YAML syntax check
for file in $(git diff --cached --name-only | grep '\.ya\?ml$'); do
  yamllint "$file" || exit 1
done

exit 0

Activate hook:

chmod +x .git/hooks/pre-commit

Step 9: Backup & Restore Mechanism

Recommended bare‑repository backup:

# Mirror clone
git clone --mirror [email protected]:ops/ops-repo.git /backup/ops-repo.git

# Cron job for incremental updates
cat > /usr/local/bin/git-backup.sh <<'EOF'
#!/bin/bash
BACKUP_DIR="/backup/ops-repo.git"
LOG_FILE="/var/log/git-backup.log"
cd "$BACKUP_DIR" && git remote update --prune
if [ $? -eq 0 ]; then
  echo "$(date '+%Y-%m-%d %H:%M:%S') - backup succeeded" >> "$LOG_FILE"
else
  echo "$(date '+%Y-%m-%d %H:%M:%S') - backup failed" >> "$LOG_FILE"
fi
EOF
chmod +x /usr/local/bin/git-backup.sh
echo "0 2 * * * /usr/local/bin/git-backup.sh" | crontab -

Restore to a new Git server:

git clone --mirror /backup/ops-repo.git /tmp/restore-repo.git
cd /tmp/restore-repo.git
git push --mirror [email protected]:ops/ops-repo.git

Export specific tag:

git archive --format=tar.gz --prefix=ops-repo-v1.2.3/ v1.2.3 -o /backup/ops-repo-v1.2.3.tar.gz

Step 10: Audit & Compliance

Enable signed commits (GPG):

# Generate GPG key (RSA 4096)
gpg --full-generate-key   # choose RSA 4096

# List key ID
gpg --list-secret-keys --keyid-format LONG

# Configure Git
git config --global user.signingkey <YOUR_GPG_KEY_ID>
git config --global commit.gpgsign true
git config --global tag.gpgsign true

# Export public key to GitLab
gpg --armor --export <YOUR_GPG_KEY_ID>

Signed commit example:

git commit -S -m "feat: add Redis high‑availability config"

Verify signature:

git log --show-signature
git verify-commit HEAD

Audit log queries:

# All operators
git log --pretty=format:"%h - %an <%ae> : %s" --all

# File‑specific history
git log --follow --all -- ansible/playbooks/nginx-config.yml

# Recent 30‑day audit report
git log --since="30 days ago" --pretty=format:"%ai | %an | %s" > /tmp/git-audit-$(date +%F).log

Monitoring & Alerting (Ready‑to‑Use)

GitLab CI/CD Integration Monitoring

Pipeline configuration (.gitlab-ci.yml):

stages:
  - validate
  - deploy

validate_ansible:
  stage: validate
  image: alpine:latest
  before_script:
    - apk add --no-cache ansible yamllint
  script:
    - yamllint ansible/
    - ansible-playbook --syntax-check ansible/playbooks/*.yml
  only:
    - merge_requests
    - main

deploy_production:
  stage: deploy
  script:
    - ansible-playbook -i ansible/inventories/production ansible/playbooks/deploy.yml
  only:
    - prod
  when: manual

Git operation monitoring script

#!/bin/bash
REPO_PATH="/var/opt/gitlab/git-data/repositories/ops/ops-repo.git"
THRESHOLD=50   # commits per hour
COMMITS=$(git --git-dir="$REPO_PATH" rev-list --since="1 hour ago" --count --all)
if [ "$COMMITS" -gt "$THRESHOLD" ]; then
  echo "Alert: $COMMITS commits in last hour exceed $THRESHOLD" | mail -s "Git Alert" [email protected]
fi

Performance & Capacity (Copy‑Ready)

Repository Size Optimization

Identify large objects:

git rev-list --objects --all |
  git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' |
  awk '/^blob/ {print substr($0,6)}' |
  sort -k2nr | head -20

Clean large files with git‑filter‑repo:

pip3 install git-filter-repo
git filter-repo --path-glob '*.iso' --invert-paths
git push origin --force --all
git push origin --force --tags

Garbage collection:

git gc --aggressive --prune=now
du -sh .git

Clone Acceleration

Shallow clone (CI/CD):

git clone --depth 1 [email protected]:ops/ops-repo.git
git fetch --unshallow

Partial clone (Git 2.22+):

git clone --filter=blob:none [email protected]:ops/ops-repo.git

Security & Compliance (Minimum Required)

Access Control

SSH key restrictions (authorized_keys):

command="/usr/bin/git-shell -c \"$SSH_ORIGINAL_COMMAND\"",no-port-forwarding,no-X11-forwarding,no-agent-forwarding ssh-ed25519 AAAA...

Repository permission audit via GitLab API:

curl --header "PRIVATE-TOKEN: <your-token>" "https://gitlab.example.com/api/v4/projects/123/members"

Compliance Checklist

[ ] All production changes require MR approval (≥1 reviewer)

[ ] Protect sensitive branches (prod/staging)

[ ] Enable GPG‑signed commits (recommended)

[ ] Encrypt sensitive data with git‑crypt or Ansible Vault

[ ] Daily backup of bare repository to offline storage

[ ] Monthly export of commit logs for audit

Common Issues & Troubleshooting

Symptom

Diagnostic Command

Possible Root Cause

Quick Fix

Permanent Fix

Push rejected (non‑fast‑forward)

git log --graph --all

Remote has new commits

git pull --rebase && git push

Always pull before push

Merge conflicts

git status

Concurrent edits

Resolve conflicts, git add, commit

Finer branch granularity

Accidental sensitive file commit

git log --all -- *.key

.gitignore missing

git filter-repo --path *.key --invert-paths

Proper .gitignore

Repository too large

git count-objects -vH

Large binaries in history

git gc --aggressive

Use LFS for large files

Slow clone

time git clone

Long history

git clone --depth 1

Archive old history regularly

Branch chaos

git branch -a

Unused branches linger

git remote prune origin

Branch management policy

Change & Rollback Playbooks

Change Process

Development validation (main) : functional testing + Ansible syntax check

Pre‑release (staging) : full testing for 24 h

Production release (prod) :

Maintenance window: Tue/Thu 22:00‑23:00

Canary rollout: single node → 10 % → 50 % → 100 %

Health check each stage:

ansible -m ping all

Rollback Operations

Scenario 1: Revert latest commit

git revert HEAD
git push origin prod

Scenario 2: Roll back to a specific tag

git checkout prod
git reset --hard v1.2.2
git push --force origin prod   # notify team first

Scenario 3: Recover deleted branch

git reflog
git checkout -b recovered-branch <commit-hash>

Best Practices (10 Key Points)

Commit granularity: one logical change per commit, use Conventional Commits

Branch naming: feature/, bugfix/, hotfix/ + short description

Frequent pushes: at least once per day to avoid local backlog

Protect main branches: production branches disallow direct pushes, require MR

Tag releases with semantic versioning (v1.2.3)

Automate hooks: pre‑commit checks, post‑receive deployment

Isolate sensitive data: store keys in Vault/KMS, keep only encrypted blobs in repo

Regular cleanup: quarterly archive unused branches, purge large historic files

Backup strategy: daily incremental, weekly full to off‑site

Team training: new members must complete Git fundamentals and hands‑on exercises