Here’s my latest VMware + ZFS VSAN all in one server build based on a Supermicro platform… this is borrowing from the napp-it all-in-one concept where you pass the SAS/SATA controller to the ZFS VM using VT-d and then share a ZFS dataset back to VMware using NFS or iSCSI. For this particular server I’m using FreeNAS as the VSAN.
Chassis – 2U Supermicro 6 Bay Hotswap
For the case I went with the Supermicro CSE-822T-400LPB. It comes with rails, 6 hotswap bays, a 400W PSU (80 plus energy efficiency), and a 5.25 bay which works great for installing a mobile rack.
The 3400 RPM Nidec UltraFlo fans are too loud for my house (although quiet for a rack-mount server) and there is no way to keep the speed low in Supermicro’s BIOS so I replaced them with low noise Antec 80mm fans which are a couple of dollars each.
Unfortunately these fans only have 3 wire (don’t have the RPM sensor) so I used freeipmi (apt-get install freeipmi) to disable monitoring the fan speeds:
Motherboard – Supermicro X10SL7-F with LSI-2308
Supermicro’s X10SL7-F has a built-in LSI 2308 controller with 8 SAS/SATA ports. Using this across the two hotswap bays works perfectly. I used the instructions on the FreeNAS forum post to flash the controller into IT mode. Under VMware the LSI-2308 is passed to the ZFS Server which is connected to all 6 x 3.5″ hotswap bays plus the first two hotswap bays of the mobile rack. This allows for 6 drives in a RAID-Z2 configuration, plus one SSD for ZIL and another for L2ARC. The final 2.5″ drives in the mobile rack are connected into the two 6Gbps ports on the motherboard and used by VMware. I usually enable VMware’s host cache on SSDs.
Memory: Crucial 8GB DDR3L (low volt) ECC Server Memory I got the 16GB kit.
CPU Xeon E3-1231v3 Quad Core
I think the Xeon E3-1231v3 is about the best bang for the dollar for an E3 class CPU right now. You get 4 3.4GHz cores (with 3.8GHz turbo), VT-d, ECC support, and hyper-threading. It’s about $3 more than the Xeon E3-1230v3 but at 100MHz faster. I’m actually using the Xeon E3-1240v3 (which is identical to the 1231v3) because I saw a good price on a used one a while ago.
Mobile Rack 2.5″ SSD bay
ICY DOCK ToughArmor 4 x 2.5″ Mobile Rack. The hotswap bays seem pretty high quality metal.
Hard Drives: See my Hard Drives for ZFS post. I’m running 2TB Seagate 7200 RPM Drives, although they may not be the most reliable drives they’re cheap and I’m confident with RAID-Z2 (and I also always have two off-site backups and one onsite backup). For SSDs I use Intel’s DC S3700 (here’s some performance testing I did on a different system, hopefully, I’ll get a chance to retest on this build)
Racking with Rails on a Lackrack
Awhile ago I discovered the Lackrack (these are $7 at Ikea)… I figured since I have rails I may as well install them. The legs were slightly too close together so I used a handsaw to cut a small notch for the rails and mounted the rails using particle board screws. One thing to note is the legs are hollow just below where the top screw of the rail mount goes so only that top screw is going to be able to bear any load.
Here’s the final product:
What VMs runs on the server?
- ZFS – provides CIFS and NFS storage for VMware and Windows, Linux, and FreeBSD clients. Basically all our movies, pictures, videos, music, any anything of importance goes on here.
- Backup server – backs up some of my offsite cloud servers.
- Kolab – Mail / Groupware server (Exchange alternative)
- Minecraft Server
- Owncloud (Dropbox replacement)
- VPN Server
- Squid Proxy Cache (I have limited bandwidth where I’m at so this helps a lot).
- DLNA server (so Android clients can watch movies).