How to Upgrade OpenStack from Kilo to Mitaka: Risks, Steps, and Lessons Learned

OpenStack Usage and Background

The HULK team maintains a private OpenStack cloud called "HULK" that provides web and cloud‑server services for internal business. The cloud runs on more than 1,000 physical nodes across nine data centers, hosting over 4,000 virtual machines. Two virtualization modes are used: local‑disk + VLAN and shared‑storage + VXLAN.

OpenStack is accessed via HTTP APIs; a custom management UI was built instead of the native dashboard to better fit HULK’s workflows.

Risk and Benefit Assessment

Upgrading OpenStack is inevitable due to rapid releases, but the team first evaluated why to upgrade and the cost‑benefit ratio. The primary driver was the need for DPDK support, which is only available from the Mitaka (M) release onward. Additional motivations included staying close to the upstream version to avoid a widening gap.

Pre‑Upgrade Preparation

The team selected the Mitaka (M) release because it is one version ahead of their current Kilo (K) version, offering a smaller upgrade jump and a more stable, bug‑fixed codebase. Preparation steps included:

Reviewing official release notes and feature deprecations.

Checking API version changes.

Backing up configuration files.

Comparing existing configs with the new version’s defaults.

Key resources:

https://releases.openstack.org/

https://releases.openstack.org/mitaka/index.html

Configuration comparison was performed using the Mitaka config reference:

The recommended component upgrade order (from official docs) was followed, as shown below:

# Backup configuration
cp -a /etc/nova /etc/nova-kilo-backup
# Upgrade packages
yum upgrade *nova*
# Edit configuration as needed
vim /etc/nova/nova.conf
# Ensure database config is correct
# Upgrade DB schema
openstack-db --service nova --update
# Sync nova API DB
nova-manage api_db sync
# Restart services

Upgrade Steps and Strategy

The upgrade was first rehearsed on a test cluster, then applied to production region by region. The strategy emphasized minimal impact on running VMs by ensuring libvirt, QEMU, and OVS remained compatible. Short maintenance windows were scheduled during low‑traffic periods, but daytime work was preferred to allow collaborative troubleshooting.

Component upgrade order in production:

Keystone

Glance

Neutron

Nova

Cinder

After each component upgrade, functional tests (image pull, network creation, VM launch) were performed. Logs were inspected for deprecation warnings and configuration changes, e.g.:

Option "verbose" from group "DEFAULT" is deprecated for removal. (Boolean) DEPRECATED: If set to false, the logging level will be set to WARNING instead of the default INFO level.

WARNING oslo_log.versionutils [-] Deprecated: keystone.contrib.revoke.backends.sql.Revoke is deprecated as of Mitaka in favor of sql and may be removed in O.

Necessary configuration adjustments (e.g., keystone‑wsgi‑admin/public) were applied.

Issues Encountered During Upgrade

Common problems and resolutions included:

Component dependency conflicts: Simultaneous upgrade of all services on a controller node avoided library version mismatches.

Nova endpoint API version change to v2.1: Updated Keystone endpoints accordingly.

Neutron configuration errors in nova.conf: Added required parameters (user_domain_name, project_domain_name, project_name).

Database migration failures: Ensured proper DB permissions and avoided manual schema changes that broke Alembic versioning.

OVS‑agent restart causing temporary network loss: Verified RabbitMQ and Neutron services were healthy before restarting.

RabbitMQ memory growth: Resolved by aligning Erlang versions.

python‑novaclient bug with Keystone V3: Patched and rebuilt the client package.

Q&A Highlights

Typical questions covered monitoring tools (custom Wonder based on OpenFalcon), API usage, network type support, containerization considerations (Kolla), and upgrade timing.

Conclusion

The HULK team completed the OpenStack upgrade in about one month, with most issues resolved through careful planning, automated Ansible playbooks, and incremental region‑by‑region rollout, providing a practical reference for similar cloud‑infrastructure migrations.