Unlocking Cloudify: A Deep Dive into Open-Source Cloud Orchestration

Preface

We are pleased to share technical insights about a PaaS‑related open‑source platform—Cloudify. After an initial encounter at IBM and a recent deep dive using Python, we present a thorough discussion of Cloudify’s overview, business and technical positioning, architecture, and core components, as well as its integration with OpenStack and Docker.

1. Cloudify Overview

Cloudify is an open‑source cloud application orchestration system that automates deployment of applications across diverse clouds.

Developed by GigaSpaces, an Israel‑based middleware company headquartered in New York, Cloudify’s roadmap has been influenced by OpenStack.

Before version 3.0 Cloudify was built on a Java stack using Groovy scripts. Starting with 3.0 the stack shifted almost entirely to Python, reflecting the growing Python ecosystem and the maturity of Python‑based cloud management platforms such as OpenStack.

2. What Does Cloudify Do?

From Blueprint to Production With Cloudify you can deploy the same application in your own data center or on the cloud of your choice using your favorite automation and configuration management tools. This enables you to build a robust continuous delivery pipeline by standardizing deployment, orchestration and build processes to create smoother transitions between development and production environments.

Cloudify allows operators and developers to describe IT topology in a single language and deploy it to any environment, ensuring consistency across environments.

Automation & Orchestration Through the Entire App Lifecycle With built‑in auto‑everything rules—scaling, failover, or custom metrics—Cloudify provides post‑deployment support, infrastructure upgrades, and fine‑tuning via blueprints and custom commands.

From an automation perspective, Cloudify focuses on application automation, a layer above middleware and infrastructure automation, positioning it as a PaaS platform for application orchestration.

3. Typical Application Scenarios

Scenario 1: Infrastructure Creation and Update

Supports integration with major IaaS platforms (VMware, OpenStack, CloudStack, AWS, SoftLayer) and with Docker and Kubernetes. Version 3.3 adds NVF integration for network component automation, enabling hybrid‑cloud topology deployment and cross‑cloud migration.

Scenario 2: Provisioning

Cloudify blueprints define relationships between IT components, offering functionality similar to Docker Compose but with broader orchestration standards (TOSCA).

Scenario 3: IT Automation Workflow

Enables composition of complex automation tasks, comparable to Tencent BlueKing, and supports plugins for Puppet, Chef, SaltStack, Ansible, as well as SSH script execution.

Other Scenarios

Built‑in monitoring, logging, and workflow engines allow fault auto‑recovery, service degradation, and other operational use cases.

4. Cloudify Architecture

The architecture consists of a client side (GUI and CLI), the Cloudify Manager, and the Application VM side managed by Cloudify agents. The manager can be deployed via Docker for easier installation.

5. Core Components

Nginx – HTTP and reverse‑proxy server forwarding GUI requests to the REST API (Swagger). Elasticsearch – Full‑text search for blueprints, instances, metrics, logs, and events. Logstash – Log collection and analysis. RabbitMQ – Asynchronous execution of metrics, events, and tasks. Riemann – Java‑based metrics stream processor and rule engine. InfluxDB – Distributed time‑series database for metrics storage. Grafana – Dashboard for visualizing InfluxDB data. Flask – Python microframework providing RESTful APIs via Swagger. Gunicorn – Python WSGI HTTP server. Celery – Distributed task engine using RabbitMQ for workflow execution. Fabric – SSH library. Diamond – Python monitoring component providing a monitoring agent.

Component interaction diagrams:

Internal manager flow:

Deployment and installation.

Workflow execution on specific nodes.

Monitoring and reporting.

Log and event handling.

Four Call‑Scenario Diagrams

1) Deployment and Installation

2) Workflow Execution

3) Monitoring and Reporting

4) Log and Event Handling

In summary, the client invokes backend components via REST APIs; the backend uses RabbitMQ for asynchronous task execution and agents (monitoring and workflow) to manage Application VMs, which may be VMs, Docker containers, or physical machines.

6. Integration with Docker and OpenStack

Cloudify’s plugin architecture includes built‑in Docker and OpenStack plugins, as well as plugins for AWS, SoftLayer, Puppet, Chef, etc.

The OpenStack plugin supports the Kilo release and provides Nova, Neutron, Cinder, and Keystone clients, enabling Blueprint‑driven provisioning of compute, storage, and network resources directly under OpenStack management.

Example OpenStack integration diagram:

Beyond basic provisioning, Cloudify can orchestrate complex topologies and supports the cloudify‑murano‑integration plugin for OpenStack Murano (application catalog) from version 3.2 onward.

For Docker, Cloudify can manage the entire Docker‑managed infrastructure and perform lifecycle operations such as container creation, start, stop, and removal.

These capabilities complete the end‑to‑end provisioning workflow comparable to OpenStack Heat, but with a unified Cloudify approach.