Manipulate OpenStack Glance Images Directly on Ceph: A Step‑by‑Step Guide

1. Prepare

In Ceph‑backed OpenStack deployments, image paths are typically {pool_name}/{uuid} . Ensure the Glance location field matches the Ceph RBD UUID by enabling detailed RBD paths in glance-api.conf :

enable_v2_api = True
show_multiple_locations = True
filesystem_store_metadata_file = filePath

Define a libvirt secret for Ceph authentication:

cat > secret.xml <<EOF
<secret ephemeral='no' private='no'>
  <usage type='ceph'>
    <name>client.libvirt secret</name>
  </usage>
</secret>
EOF

Register the secret with libvirt:

virsh secret-define --file secret.xml
# Note the generated UUID
cat /etc/ceph/ceph.client.glance.keyring
sudo virsh secret-set-value --secret 453cf8e2-02b4-4744-9455-5544601bb043 --base64 AQBkGYZX/tuDMxAANVznwk/+MkrhsEBlZAJlLg==

2. Two Approaches to Image Operations

Approach A – Direct RBD manipulation

Generate a new UUID and convert the existing qcow2 image to raw directly in Ceph:

# uuidgen
b0b6bb99-7693-4614-b6f1-01b602d69f54
# Convert qcow2 to raw
qemu-img convert -O raw rbd:images/e6c01758-b7ad-4097-9400-570cc1ceed43 rbd:images/b0b6bb99-7693-4614-b6f1-01b602d69f54

Resize the raw image in Ceph:

qemu-img resize rbd:images/b0b6bb99-7693-4614-b6f1-01b602d69f54 50G

Boot the VM once to let resize2fs adjust the filesystem, then clean up unnecessary files.

Approach B – Attach the image via libvirt

Define a libvirt <disk> element that points to the Ceph RBD image:

<disk type='network' device='disk'>
  <driver name='qemu' type='raw' cache='writeback' discard='unmap'/>
  <auth username='glance'>
    <secret type='ceph' uuid='453cf8e2-02b4-4744-9455-5544601bb043'/>
  </auth>
  <source protocol='rbd' name='rbd:images/b0b6bb99-7693-4614-b6f1-01b602d69f54'>
    <host name='10.10.10.84' port='6789'/>
    <host name='10.10.10.102' port='6789'/>
    <host name='10.10.10.110' port='6789'/>
  </source>
  <target dev='vda' bus='virtio'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</disk>

After the image is ready, create a new Glance entry that points to the RBD snapshot:

glance image-create --id {NEW_UUID} --name {NEW_IMAGE_NAME} \
  --store rbd --disk-format raw --container-format bare \
  --location rbd://$(sudo ceph fsid)/images/b0b6bb99-7693-4614-b6f1-01b602d69f54/snap-20160922

3. In‑place Editing of Glance Images

If configuration files inside the image need modification, use guestfish :

guestfish
<fs> add-drive images/b0b6bb99-7693-4614-b6f1-01b602d69f54 username:glance protocol:rbd format:raw secret:AQBkGYZX/tuDMxAANVznwk/+MkrhsEBlZAJlLg==
<fs> run
<fs> list-filesystems
<fs> mount /dev/sda1 /
<fs> edit /etc/fstab   # modify as needed
<fs> exit

After editing, repeat the snapshot creation and Glance redirection steps described above.

4. Summary

Using Ceph as the backend makes image creation, modification, resizing, and format conversion completely remote, often faster than local‑disk workflows. Although the procedure looks involved, it showcases several practical patterns that can be combined or automated to fit specific production environments.