What’s the best virtualization solution for your homelab? The other day a friend asked me this. I replied with two options: VMware ESXi or Proxmox VE.
What is a hypervisor? In case you’ve been in the dark for the last decade, a hypervisor is a server that allows you to run multiple virtual machines (VMs) on physical hardware. VMs are useful for isolation (good for security and also for learning since you can wipe out and recreate VMs at will). But virtualization is also an essential foundation for any sort of IT infrastructure, including a homelab. A virtual environment gives you the flexibility to install and run almost anything down the road, even if you can’t foresee it today.
VMware ESXi or Proxmox VE
Two popular Hypervisor options are VMware ESXi and Proxmox VE. These are both Virtualization platforms and allow for running many VMs or containers at high density. Both perform well (I have not done formal testing, but I can’t tell the difference in CPU performance).
VMware is the king of virtualization. You will not find many companies not using VMware until you get to large corporations where open source solutions based on Xen or KVM take over. But anywhere in between from small to medium businesses, you will find VMware everywhere. Also, if you will use any other enterprise-class solution, be it a SIEM, storage, monitoring tool, etc. the vendor will most likely support running on VMware or have integration with it if applicable.
While VMware is proprietary, it is robust and fully featured. ESXi’s vSAN and distributed switching, high availability, and fault-tolerance are unparalleled.
VMware ESXi free is the hypervisor of choice for many homelabs, and VMUG Advantage ($200/year) allows you to use all the features (with up to 6 CPUs) in a non-production environment for no cost.
Another option I like is Proxmox VE. In fact, I like it better than VMware ESXi.
Over the last 16 months, I have migrated most of my VMware VMs over to Proxmox in my Homelab. But I don’t recommend this switch to everyone. If you are learning virtualization to work in an enterprise environment, you want to stick with VMware. Why? I know of hundreds of businesses that use VMware. I know of 1 using Proxmox.
Why Did I Switch to Proxmox VE? Ansible Integration
So why did I switch to Proxmox? In my case, I wasn’t learning VMware. I was learning Ansible. I was trying to automate some of my tasks like deploying VMs. When I tried to set up Ansible with VMware ESXi free in my homelab, I found out the free version of ESXi doesn’t have a REST API! There was no way Ansible could communicate with it. My first thought was to get VMUG, but I doubt my lab qualifies for VMUG since I run some VMs for business purposes. Also, I didn’t want to spend $200/year just for software. So I tried Proxmox to get Ansible integration. I decided if it worked out well, I would switch over to it. 16 months later I migrated my final VM to Proxmox.
Migrating from VMware to Proxmox
Proxmox has a guide to migrate from VMware. I couldn’t quite get that to work. However, what worked for me was mounting the FreeNAS NFS share for VMware from Proxmox. In Proxmox I created a new VM, but then I would delete the virtual disk file and use this command to import the .vmdk file from VMware:
cd /mnt/nfs/vmware/minecraft.b3n.org qm importdisk 123 minecraft.b3n.org.vmdk pool1 --format vmdk # ^ 123 is the Proxmox VM ID
My Thoughts on Proxmox
Proxmox is excellent. You get Clustering, Live-migration, High Availability. All for free. You can use ZFS on a single box without the complexity of running FreeNAS on VMware All-in-One. If you have two Proxmox servers, they can use ZFS to mirror each other. With three or more Proxmox servers (technically you only need two with a Raspberry Pi to maintain the quorum), Proxmox can configure a Ceph cluster for distributed, scalable, and high-available storage. From one small server to a cluster, Proxmox can handle a variety of scenarios.
Proxmox builds on open-source technology. KVM for VMs, LXC for Containers, ZFS or Ceph for storage, and bridged networking or Open vSwitch for networking. You could run the open-source components in an ad hoc manner yourself (before I tried Proxmox I had experimented with an Ubuntu LXD server), but Proxmox provides a nice single pane of glass. Besides the GUI, it adds some features like live-migration and integrated backups and alerts on failures or warnings.
My favorite feature in Proxmox that VMware doesn’t have is containers! These aren’t docker containers, but full system containers. Containers allow you to get a higher density (in terms of memory) than VMs because containers share the host kernel. Ubuntu reports a 14.5 higher density using containers instead of VMs. I love the way containers work. You can download whatever templates you want (they include all popular Linux distributions) and then instantly deploy containers from those templates.
Getting performance information on the cluster, individual nodes, VMs, or containers is simple. At a glance, you can see where your bottleneck is and see which VMs and containers are consuming what resources.
Why I would choose VMware ESXi or Proxmox VE
Reasons to use VMware ESXi
- Distributed virtual switching
- Fault Tolerance (the nearest you can get to zero-downtime on legacy systems).
- A good solution when uptime is important or 24/7 support or handholding is needed.
- Integration with other Enterprise-class solutions
- More Robust
- VT-d passthrough is easier to set up (Proxmox can’t set up a passthrough device in the GUI).
- Live network changes (Proxmox requires a reboot for network changes).
Reasons to use Proxmox VE
- All features are free (Ansible integration, High Availability, distributed storage, etc.)
- I’m already familiar with Ubuntu and Debian so Proxmox is easy to manage.
- Can install common open-source tools and packages easily since it’s based on Debian
- Better performance insights and history using tools like Zabbix or Netdata than I can get out of ESXi free.
- Integrated ZFS (useful for a single box using local storage without a RAID card)
- Better suited when licensing costs are important or open-source solutions are required.
- Built on and uses open source technologies
- LXC Containers–instant deploy allows for higher density than VMs.
- Updates are much easier than VMware since Proxmox is just apt-get update;
apt-get upgradeapt-get dist-upgrade (thanks for the correction Ivan).
- Each Proxmox node is a complete management node so you have full control of your cluster. You aren’t concerned with anything like vCenter that can go down.
- Upgrades are a lot easier. I’ve seen VMware experts struggle at VMware upgrades. Proxmox provides straight forward upgrade procedures.
- Simple to automate SSL certs to the management interface with something like Let’s Encrypt
VMware ESXi and Proxmox VE feature comparison Chart
|Hypervisor||VMware ESXi Free||VMware ESXi VMUG||Proxmox VE CE||Proxmox VE Enterprise|
|HA-Storage||No||Yes, vSAN||Yes, Ceph||Yes, Ceph|
|Live-Migration||No||Yes||VMs only||VMs only|
|VM Backups||3rd Party: Veeam||3rd Party: Veeam||Yes||Yes|
|GUI for Virtual Networking||Yes||Yes||No||No|
|Storage Compatible||iSCSI, NFS, FC, FCoE, local||iSCSI, NFS, vSAN, FC, FCoE, SAN||iSCSI, NFS, ZFS, Ceph, CIFS, GlusterFS||iSCSI, NFS|
ZFS, Ceph, CIFS, GlusterFS
|Enterprise SAN integration||N/A||Yes||No||No|
|Minimum Nodes for HA||N/A||2 (3 preferred)||3 (or 2 + witness device)||3 (or 2 + witness device)|
|Per VM Firewall||No||No||Yes||Yes|
|License allows renting out VMs/Containers||No||VSPP Required||Yes||Yes|
|License allows business use||Yes||No||Yes||Yes|
|API (Integration with Ansible, etc.)||No||Yes||Yes||Yes|
Alternatives to ESXi and Proxmox VE
If you enjoy doing everything off the CLI, you can just use KVM for VMs and LXD for containers. I prefer to have some sort of management system. I find it helpful because it automatically does a lot of things for me like running backups, alerting on the system or drive failures, and monitoring performance. oVirt may be a good option, I passed it by because the list of supported operating systems is outdated so I don’t know how well maintained the project is. Bhyve is another option based on FreeBSD; however, last I tested, it wasn’t entirely stable and resulted in a few crashes under load. I tried to get Hyper-V up and running once, but it wanted to be in an Active Directory environment, and I didn’t want to hack it to work standalone. Xenserver is another option, but it has been discontinued. I do think the best options for homelab use are VMware and Proxmox. Of course, there’s also Kubernetes, but that’s for another post.
The Best Hypervisor for Your Goals
If your goal is to learn hypervisors or enterprise solutions that integrate with VMware for your career, the best hypervisor to learn is VMware ESXi. You cannot match the robustness and ubiquitousness of VMware. Otherwise, for homelab use, I prefer Proxmox. For a zero dollar budget, it offers a lot more freedom and features in the community edition than the ESXi free version.
Cost and licensing restrictions side, from a technical and feature perspective; I would rank the solutions as follows: ESXi VMUG > Proxmox VE Enterprise > Proxmox VE CE > ESXi Free.
One area that Proxmox excels is for those running a single box without hardware RAID. VMware ESXi free provides no software storage solution for a single hypervisor. Proxmox has built-in ZFS making for a much simpler design than the VMware and FreeNAS All-in-one.
My current homelab setup: I have Proxmox VE installed on two of my Supermicro servers. All of the VMs and containers are running there. And both Proxmox servers are backing up to my VMware+FreeNAS all-in-one box.
Which hypervisor do you use in your homelab, and why?