The “vmhive.com” domain is my personal hobby and project to build and maintain my own private “mini cloud” for the dual purpose of hosting my various Internet domains (including this one) as well as to act as a platform for my own experimentation. It may also become the basis of a start-up company if I decide at some point to go that route.
The site is built on a single Intel Core i7-6850K CPU with 6 cores and 12 threads, overclocked to 4GHZ. It has 128 GB of RAM and 40 TB of storage. It’s a pretty nice setup for a personal home-based “white box” server. About half of the storage is dedicated to hosting rips of my movies and other media for my OpenElec/Kodi home media servers. The rest is available for setting up and tearing down experimental configurations of things like Ceph clusters, and various VM and Cloud technologies like OpenStack, ProxMox, VMWare, and so on.
I’m currently running VMWare’s free ESXi 6.0 hypervisor on the bare metal. Everything is virtualized except for the Comcast router and a 10 port Cisco smart switch configured for the several external VLANs. The firewalls and even the storage subsystem are implemented as virtual machines. Since VMWare doesn’t support the built-in RAID controller, I simply used PCI passthrough to pass the SATA HBA directly to a FreeNAS VM, which then manages the storage as a ZFS pool and exposes it via internal VMWare networking via iSCSI. It took a little tweaking to ensure things boot up in the right order upon power failure or reboot, but it actually works quite well. Performance is not completely optimal, but it’s quite good for a “home server” and is “good enough” for my purposes. And it saved me having to spend another $1000 on my currently tight budget. My firewalls are pfSense VMs that act as both firewalls and routers between the virtual switches they are attached to. Backups of VMs are easy. I periodically export each VM to an OVA file and copy it to separate offline media. I have a bunch of old hard drives that make perfect backup media.
Why VMWare? It’s simple, and unfortunate at the same time. I originally tried to set up the server using ProxMox, but I need to be able to run “nested” hypervisors for my experimentation and research. Unfortunately, VMWare in particular runs extremely poorly when nested under KVM on Linux. It seems that Linux’s nested EPT support isn’t quite up to par with VMWare’s. I’m perfectly able to run Linux KVM (i.e. ProxMox) and other hypervisors under VMWare with relatively small performance hit. I can even run another ESX hypervisor under ESX. So, for my purposes, ESXi has to run on the bare metal for now. I have to admit, the VMWare hypervisor is a quite good product.
Yes. I realize that using only a single server would not be a wise choice if I were running a business critical cloud. When and if I do decide to launch a start-up or consulting company, I’d certainly move my servers to a co-location service, or the public cloud. At this point I’m still at the “garage” stage, so I must make some conscious trade-offs. Obviously, the equipment cost is one consideration, but another is electric power, and yet another is noise. As long as I’m making backups I can easily recover and rebuild my system within a week even if my entire server went up in flames. Well unless my house also burns down, but then I’ll have more important problems to deal with. 🙂
Internally, I run two Samba4 Active Directory domain controllers to both manage internal DNS and to manage user identities both for Linux hosts and my Windows desktop. It works quite well in that it’s basically LDAP and using Active Directory Users and Computers as well as the Microsoft DNS manager, I’m able to easily manage. NAS services are fed by an internal Linux VM (not the FreeNAS providing iSCSI directly from the HBA) and provide various shared storage volumes for things like development, media storage, etc.
It really is a sweet setup. I can easily launch a full VMWare or OpenStack deployment including a full Ceph cluster entirely within a single physical machine and it actually runs at a slower but still quite usable performance level. It’s ideal for experimentation, testing and development. With VMWare’s VM snapshot feature it’s perfect for setting up a repeatable test environment. It’s definitely a far cry from when I had to constantly be reinstalling bare metal in the lab back in my start-up days in the 90s. I definitely don’t miss that.