The other day I got a little frustrated with my Gen 8 Microserver, I was trying to upgrade ESXi to 5.5 but the virtual media feature kept disconnecting in the middle of the install due to not having an ILO4 license–I actually bought an ILO4 enterprise license but I have no idea where I put it! What’s the point of IPMI when you get stopped by licensing? I hate having to physically plug in a USB key to upgrade VMware so much that I decided I’d just build a new server–which I honestly think is faster than messing around with getting an ISO image on a USB stick.
Warning: I’m sorry to say that I cannot recommend this motherboard that I reviewed earlier: I ended up having to RMA this board twice to get one that didn’t crash. The Marvell SATA Controller was never stable long term under load even after multiple RMAs so I ran it without using those ports which sort of defeated the reason I got the board in the first place. Then in 2017 the board died shy of 3 years old, the shortest I have ever had a motherboard last me. Generally I have been pretty happy with ASRock desktop boards but this server board isn’t stable enough for business or home use. I have switched to Supermicro X10SDV Motherboards for my home server builds.
- SilverStone DS-380 ($150 at Amazon) 8 bay hot-swap
- ASRock Avoton C2750D4I ($400)
- Seagate Barracudas 7200 RPM (2 TB) and Seagate 600 Pro SSD (I already had the 2TB drives and I got the SSD at TigerDirect but they’re out now).
- Kingston 2 x KVR16E11/8I DDR3-1600 8GB ECC ($85)
- Power Supply: FSP300-60GH 300W ($45)
ASRock C2750D4I Motherboard / CPU
Update: 2014-05-11. Here’s a great video review on the motherboard…
12 SATA ports! This motherboard is perfect for ZFS which loves having direct access to JBOD disks. The Marvell SATA controllers did not show up in VMware initially, however Andreas Peetz provides a package for adding unsupported drivers in VMware, and this worked perfectly. It took me a couple minutes to realize all you need to do is run these three commands:
Update November 16, 2014 .. it turned out the below issue was caused by a faulty Marvell controller on the motherboard, I ran FreeBSD (a supported OS) and the fault also occurred there so I RMAed the motherboard … I ended up getting a bad motherboard again but after a second RMA everything is stable in VMware… so you can disregard the below warning.
Update March 12, 2015. My board continues to function okay, but some people are having issues with the drives working under VMware ESXi. Read the comments for details.
Update August 23, 2014 ** WARNING Read this before you run the command below ** I had stability issues using the below hack to get the Marvell controllers to show up. VMware started hanging as often as several times a day requiring a system reboot. This is the entry in the motherboard’s event log: Critical Interrupt – I/O Channel Check NMI Asserted. I swapped the Kingston memory out for Crucial on ASRock’s HCL list but the issue still persisted so I can’t recommend this drive for VMware. After heavy I/O tests ZFS also detected data corruption on two drives connected to the Marvell controllers. I am pretty sure this is because VMware does not officially support these drives so this issue likely doesn’t exist for operating systems that officially support the Marvell controller.
esxcli software acceptance set --level=CommunitySupported
esxcli network firewall ruleset set -e true -r httpClient
esxcli software vib install -d http://vibsdepot.v-front.de -n sata-xahci
Avoton Octa-Core processor. Normally I don’t even look at Atom processors, but this is not your grandfather’s Atom. The Avoton processor supports VT-x, ECC memory, AES instructions, and is a lot more powerful and at only 20 W TDP. This CPU Boss benchmark says it will probably perform similarly to the Xeon E3-1220L. The Avoton can also go up to 64GB memory where the E3 series is limited to 32GB making it a good option for VMware or for a high performance ZFS NAS. The Avoton does not support VT-d so there is no passing devices directly to VMs.
My only two disappointments are no internal USB header on the board (I always install VMware on a USB stick so right now there’s a USB stick hanging on the back) and I wish they had used SFF-8087 mini-SAS connectors instead of individual SATA ports on the board to cut down on the number of SATA cables.
Overall I am very impressed with this board and it’s server-grade features like IPMI.
Instead of going into more detail here, I’ll just reference Patrick’s review of the ASRock C2750D4I
Alternative Avoton Boards
There are a few other options worth looking at. The ASRock C2550D4I is the same board but Quad core instead of Octa Core. I actually almost bought this one except I got the 2750 at a good price on SuperBiiz.
Also the SuperMicro A1SAi-2750F (Octa core) and A1SAi-2550F (Quad core) are good options if you don’t need as many SATA ports or you’re going to use a PCI-E SATA/SAS controller. Supermicro’s motherboards have the advantage of Quad GbE ports, an internal USB header (not to mention USB 3.0), while sacrificing the number of SATA ports–only 2 SATA3 ports and 4 SATA2 ports. These Supermicro boards use the smaller SO-DIMM memory.
Silverstone DS-380: 8 hot-swap bay chassis
The DS-380 has 8 hot-swap bays, plus room for four fixed 2.5″ drives for up to 12 drives. As I started building this server I found the design was very well thought out. Power button lockout (a necessity if you have kids), locking door, dust screens on fan intakes, etc. The case is practical in that the designers cut costs where they could (like not painting the inside) but didn’t sacrifice anything of importance.
A little larger than the HP Gen8 Microserver, but it can hold more than twice as many drives. Also the Gen8 Microserver is a bit noisier.
You’ll notice above from the top there is a set of two drives, then one drive by itself, and a set of five drives. This struck me as odd at first, but this is actually that way by design. If you have a tall PCI card plugged into your motherboard (such as a video card) you can forfeit the 3rd drive from the top to make room for it.
The drive trays are plastic, obviously not as nice as a metal tray but not too bad either. One nice feature is screw holes on the bottom allow for mounting a 2.5″ drive such as an SSD! That’s well thought out! Also there’s a clear plastic piece that runs alongside the left of each tray that carries the hard drive activity LED light to the front of the case (see video below).
Here’s the official Silverstone DS-380 site, and here’s a very detailed review of the DS-380 with lots of pictures by Lawrence Lee.
Using 4TB drives 8 bays would get you to 24TB using RAID-Z2 or RAID-6. Plus have 4 2.5″ fixed bays left for SSDs.
I run a virtualized ZFS server on OmniOS following Gea’s Napp-in-one guide. I deviate from his design slightly because I run on top of VMDKs instead of Passing the controllers to the guest VM (because I don’t have VT-d on the Avoton).
ZIL – Seagate SSD Pro
120GB Seagate Pro SSD. The ZIL (ZFS Intent Log) is the real trick to high performance random writes, by being able to cache writes on capacitor backed cache the SSD can guarantee a write to the requesting application before it is transferred out of RAM and onto spindles.
I’m pretty happy with the custom build. I think the Gen 8 HP Microserver looks more professional compared to the DS-380 which looks more like a DIY server. But what matters is on the inside, and having access to IPMI when I need it without having to worry about licensing is worth something in my book.