Building Hypervisor Infrastructure with Proxmox and Ceph

Sep 30, 2022

Written by Rajendra Acharya, Platform Engineer

Why Proxmox and Ceph?

Proxmox Virtual Environment (PVE) is a powerful, open-source platform for managing virtual machines and containers. Combined with Ceph, a distributed storage system, you can achieve:

High Availability (HA): Minimized downtime with automatic failover.
Scalability: Seamless expansion of compute and storage resources.
Cost-Effectiveness: Open-source and community-driven solutions.

This combination is ideal for enterprises, startups, and enthusiasts looking to build a production-grade hypervisor infrastructure.

Setting Up Proxmox Infrastructure

Step 1: Install Proxmox VE

Download ISO: Download the Proxmox VE ISO.
Install: Burn the ISO to a USB drive or use PXE boot, and install it on your server hardware.
Initial Configuration:
- Set up the hostname, IP address, and root password.
- Ensure proper network configuration for management.

Step 2: Network Configuration

Use a bonded interface for redundancy.
Create separate VLANs for:
- Management traffic.
- Ceph storage traffic.
- VM data traffic.

Setting Up Ceph for Distributed Storage

Step 1: Prepare Proxmox Nodes

Ensure at least three Proxmox nodes for a reliable Ceph cluster.
Install additional disks on each node for Ceph’s OSDs (Object Storage Daemons).

Step 2: Install and Configure Ceph

Enable Ceph:
- Navigate to Datacenter → Ceph in the Proxmox web interface.
- Initialize the Ceph cluster.
Deploy OSDs:
- Add the dedicated disks to the Ceph cluster via Ceph → OSD.
Setup Monitors (MONs):
- Add at least three monitor nodes for quorum and cluster reliability.
Configure Ceph Pools:
- Create pools for storage, defining redundancy settings like replication or erasure coding.

Integrating Ceph with Proxmox

Step 1: Add Ceph Storage to Proxmox

Go to Datacenter → Storage, and add Ceph RBD as a new storage type.
Test connectivity and ensure proper pool access.

Step 2: Migrate VM Storage

Move VM disks to the Ceph RBD pool for centralized storage and HA.

High Availability and Maintenance

Configure HA in Proxmox

Use HA Groups to define failover priorities.
Ensure fencing devices or quorum settings are configured to avoid split-brain scenarios.

Regular Maintenance

Monitor the health of the Ceph cluster (ceph status).
Use the Proxmox dashboard to monitor and manage VM performance.

The Take-Away

Building a hypervisor infrastructure with Proxmox and Ceph is a rewarding experience that blends the power of open-source tools with enterprise-grade features. While the initial setup requires thoughtful planning, the result is a resilient, scalable, and cost-effective environment.

By understanding and implementing these systems, you not only gain valuable experience but also the satisfaction of running an infrastructure that rivals commercial offerings in performance and reliability.