Supermicro 2U ZFS Server Build

Here’s my latest VMware + ZFS VSAN all in one server build based on a Supermicro platform… this is borrowing from the napp-it all-in-one concept where you pass the SAS/SATA controller to the ZFS VM using VT-d and then share a ZFS dataset back to VMware using NFS or iSCSI.  For this particular server I’m using FreeNAS as the VSAN.

Chassis – 2U Supermicro 6 Bay Hotswap

15161878016_af4bd6c2fd_k

For the case I went with the Supermicro CSE-822T-400LPB.  It comes with rails, 6 hotswap bays, a 400W PSU (80 plus energy efficiency), and a 5.25 bay which works great for installing a mobile rack.

The 3400 RPM Nidec UltraFlo fans are too loud for my house (although quiet for a rack-mount server) and there is no way to keep the speed low in Supermicro’s BIOS so I replaced them with low noise Antec 80mm fans which are a couple dollars each.

14998128759_079d8fe1ab_k

14998338517_aa3f5ff041_k

Unfortunately these fans only have 3 wire (don’t have the RPM sensor) so I used freeipmi (apt-get install freeipmi) to disable monitoring the fan speeds:

#  ipmi-sensors-config -h 10.2.0.228 -u ADMIN -p ADMIN –filename=ipmi.config –checkout
Under the FAN sections change this to “No”…
Enable_Scanning_On_This_Sensor                                              No
# ipmi-sensors-config -h 10.2.0.228 -u ADMIN -p ADMIN –filename=ipmi.config –commit
Probably a better solution is to use 4-wire fans and lower the sensor thresholds but then you’re looking at around $10/fan.
The server makes about as much noise as my desktop computer now.

Motherboard – Supermicro X10SL7-F with LSI-2308

Supermicro’s X10SL7-F has a built in LSI 2308 controller with 8 SAS/SATA ports.  Using this across the two hotswap bays works perfectly.  I used the instructions on the FreeNAS forum post to flash the controller into IT mode.  Under VMware the LSI-2308 is passed to the ZFS Server which is connected to all 6 x 3.5″ hotswap bays plus the first two hotswap bays of the mobile rack.  This allows for 6 drives in a RAID-Z2 configuration, plus one SSD for ZIL and another for L2ARC.    The final 2.5″ drives in the mobile rack are connected into the two 6Gbps ports on the motherboard and used by VMware.  I usually enable VMware’s hostcache on SSDs.

vmware_zfs_vtd_supermicro_lsi2308

Memory: Crucial

Memory: Crucial 8GB DDR3L (low volt) ECC Server Memory I got the 16GB kit.

CPU Xeon E3-1231v3 Quad Core

I think the Xeon E3-1231v3 is about the best bang for the dollar for an E3 class CPU right now.  You get 4 3.4GHz cores (with 3.8GHz turbo), VT-d, ECC support, and hyper-threading.  It’s about $3 more than the Xeon E3-1230v3 but at 100MHz faster.   I’m actually using the Xeon E3-1240v3 (which is identical to the 1231v3) because I saw a good price on a used one awhile ago.

Mobile Rack 2.5″ SSD bay


15184899715_d71b7ad69a_k
ICY DOCK ToughArmor
4 x 2.5″ Mobile Rack.  The hotswap bays seem pretty high quality metal.

Hard Drives: I’m running 2TB Seagate 7200 RPM Drives, although they may not be the most reliable drives they’re cheap and I’m confident with RAID-Z2 (and I also always have two off-site backups and one onsite backup).  For SSDs I use Intel’s DC S3700 (here’s some performance testing I did on a different system, hopefully I’ll get a chance to retest on this build)

Racking with Rails on a Lackrack

15152114936_a863dcbece_o (1)
Awhile ago I discovered the Lackrack (these are $7 at Ikea)… I figured since I have rails I may as well install them.  The legs were slightly too close together so I used a handsaw to cut a small notch for the rails and mounted the rails using particle board screws.  One thing to note is the legs are hollow just below where the top screw of the rail mount goes so only that top screw is going to be able to bear any load.

14998127309_3741d99834_k

Here’s the final product:

supermicro_server_lackrack

What VMs runs on the server?

  • ZFS – provides CIFS and NFS storage for VMware and Windows, Linux, and FreeBSD clients.  Basically all our movies, pictures, videos, music, any anything of importance goes on here.
  • Backup server – backs up some of my offsite cloud servers.
  • Kolab – Mail / Groupware server (Exchange alternative)
  • Minecraft Server
  • Owncloud (Dropbox replacement)
  • VPN Server
  • Squid Proxy Cache (I have limited bandwidth where I’m at so this helps a lot).
  • DLNA server (so Android clients can watch movies).