How to Perform Offline Ceph Octopus Deployment with cephadm on Ubuntu

Creating Offline Installation Package

First cache required Debian packages and Docker images in an online environment.

Install docker-ce

curl -sSL https://get.daocloud.io/docker | sh
systemctl daemon-reload
systemctl restart docker
systemctl enable docker

Install cephadm

Use curl to fetch the latest standalone script; if the network is poor, copy it from GitHub.

Edit /etc/resolv.conf to set nameserver to 114.114.114.114.

curl --silent --remote-name --location https://github.com/ceph/ceph/raw/octopus/src/cephadm/cephadm
chmod +x cephadm

Install cephadm

./cephadm add-repo --release octopus
./cephadm install

Bootstrap new cluster

Create directory /etc/ceph and run:

mkdir -p /etc/ceph
./cephadm bootstrap --mon-ip 192.168.10.2

Enable ceph CLI

cephadm add-repo --release octopus
cephadm install ceph-common

Deploy OSD

Device is considered available if it meets all conditions: no partitions, no LVM state, not a system device, no filesystem, not a Ceph BlueStore OSD, and size greater than 5 GB.

Create OSD on specific host/device

ceph orch daemon add osd node1:/dev/sdb

Export Docker Images

Save required images to tar files for the offline package:

docker save -o ceph.tar quay.io/ceph/ceph:v15
docker save -o prometheus.tar quay.io/prometheus/prometheus:v2.18.1
docker save -o grafana.tar quay.io/ceph/ceph-grafana:6.7.4
docker save -o alertmanager.tar quay.io/prometheus/alertmanager:v0.20.0
docker save -o node-exporter.tar quay.io/prometheus/node-exporter:v0.18.1

Export Deb Packages

Copy cached deb files from /var/cache/apt/archives to a new folder and create a Packages.gz index.

apt-get install dpkg-dev -y
mkdir /offlinePackage
cp -r /var/cache/apt/archives /offlinePackage
dpkg-scanpackages /offlinePackage/ /dev/null | gzip > /offlinePackage/Packages.gz
tar zcvf offlinePackage.tar.gz /offlinePackage/

Modify cephadm Script

Change the pull command to use local images instead of pulling from the internet.

Start Offline Deployment

Prerequisites

cephadm supports Octopus v15.2.0 and later.

Requires container runtime (podman or docker) and Python 3.

Time synchronization.

Basic Configuration

All three Ubuntu 20.04 nodes must run the following steps.

Configure hosts resolution

cat >> /etc/hosts <<EOF
192.168.10.2 node1
192.168.10.3 node2
192.168.10.4 node3
EOF

Set hostnames

hostnamectl set-hostname node1
hostnamectl set-hostname node2
hostnamectl set-hostname node3

Configure local apt source

tar zxvf offlinePackage.tar.gz -C /
mv /etc/apt/sources.list /etc/apt/sources.list.bak
vi /etc/apt/sources.list
deb file:/// offlinePackage/
apt update

Install Docker from offline deb packages

cd /offlinedeb/archives
dpkg -i containerd.io_1.4.11-1_amd64.deb
dpkg -i docker-ce-cli_5...amd64.deb
dpkg -i docker-ce-rootless-extras_5...amd64.deb
dpkg -i docker-ce_5...amd64.deb
systemctl start docker
systemctl enable docker

Load Docker images

docker load -i node-exporter.tar
docker load -i alertmanager.tar
docker load -i prometheus.tar
docker load -i ceph.tar
docker load -i grafana.tar

Install cephadm

chmod +x cephadm
cp cephadm /usr/bin/
apt install cephadm --allow-unauthenticated
# if errors, run apt --fix-broken install

Bootstrap new cluster

cephadm bootstrap --mon-ip 192.168.174.128

This creates monitor and manager daemons, generates SSH keys, writes ceph.conf, admin keyring, and public key.

Add Hosts to Cluster

ssh-copy-id -f -i /etc/ceph/ceph.pub node2
ssh-copy-id -f -i /etc/ceph/ceph.pub node3
ceph orch host add node2
ceph orch host add node3

Adding hosts automatically expands monitor and manager services.

Deploy OSD

List storage devices:

ceph orch device ls

Add OSDs on each node:

ceph orch daemon add osd node1:/dev/sdb
ceph orch daemon add osd node1:/dev/sdc
... (repeat for other devices and nodes)

Deploy MDS

ceph orch apply mds *fs-name* --placement="3 node1 node2 node3"

Create pools for CephFS and then the filesystem:

ceph osd pool create cephfs_data 64 64
ceph osd pool create cephfs_metadata 64 64
ceph fs new cephfs cephfs_metadata cephfs_data

Deploy RGW

ceph orch apply rgw myorg cn-east-1 --placement="3 node1 node2 node3"
# or use radosgw-admin to create realm, zonegroup, zone, and period