2015-01-07: I’ve updated this post to to reflect changes in FreeNAS 9.3.
I’ve been using OpenIndiana since late 2011 and switched to OmniOS in 2013. Lately I started testing FreeNAS, what drove me to do this is I use CrashPlan to backup my pool but recently Code 42 announced they’ll be discontinuing Solaris support for Crashplan so I needed to look for an alternative OS or an alternative backup solution. I looked at FreeNAS because it has a CrashPlan plugin that runs in a jail using Linux emulation. After testing it out for a while I will likely stay on OmniOS since it suits my needs better and instead switch out CrashPlan for ZnapZend for my backup solution. But after running FreeNAS for a few months here are my thoughts on both platforms and their strengths and weaknesses as a ZFS storage server.
Update: 2015-01-07: After a lot of testing ZnapZend ended up not working for me, this is not it’s fault, but because I have limited and bandwidth, the snapshots don’t catch up and it gets further and further behind so for now I’m continuing with Crashplan on OmniOS. I am also testing FreeNAS and may consider a switch at some point.
CIFS / SMB Performance for Windows Shares
FreeNAS has a newer implementation of SMB, supporting SMB3, I think OmniOS is at SMB1. FreeNAS can function as an Active Directory Domain Controller.
OmniOS is slightly faster, writing a large file over my LAN gets around 115MBps vs 98MBps on FreeNAS. I suspect this is because OmniOS runs NFS SMB at the kernel level and FreeNAS runs it in user space. I tried changing the FreeNAS protocol to SMB2, and even SMB1 but couldn’t get past 99MBps. This is on a Xeon E3-1240V3 so there’s plenty of CPU power, Samba on FreeNAS just can’t keep up.
CIFS / SMB Snapshot Integration with Previous Versions
Previous Versions Snapshot Integration with Windows is far superior in OmniOS. I always use multiple snapshot jobs to do progressive thinning of snapshots. So for example I’ll setup monthly snaps with a 6 month retention, weekly with two-month retention, daily with two week, hourly with 1 week, and every 5 minutes for two days. FreeNAS will let you setup the snap jobs this way, but in Windows Previous Versions it will only show the snapshots from one of the snap jobs under Previous Versions (so you may see your every 5 minute snaps but you can’t see the hourly or weekly snaps). OmniOS handles this nicely. As a bonus Napp-It has an option to automatically delete empty snapshots sooner than their retention expiration, so I don’t see them in Previous Versions unless some data changed.
Enclosure Management
Both platforms struggle here, FreeNAS has a bit of an edge here… probably the best thing to do is record the serial number of each drive with the slot number. In FreeNAS drives are given device names like da0, da1, etc. but unfortunately the numbers don’t seem to correspond to anything and they can even change between reboots. FreeNAS has the ability to label drives so you could insert one drive at a time and label them with the slot they’re in.
OmniOS drives are given names like c3t5000C5005328D67Bd0 which isn’t entirely helpful.
For LSI controllers, the sas2irc utility (which works on FreeBSD or Solaris) will map the drives to slots.
Fault Management
The ZFS fault management daemon will automatically replace a failed drive with a hot spare… but it hasn’t been ported to FreeBSD yet, so FreeNAS only has warm spare capability. Update: FreeNAS added hot spare capability on Feb 27, 2015. To me this is a minor concern… if you will use RAID-Z with a hot spare why not just configure the pool with RAID-Z2 or RAID-Z3 to begin with? However, I can see how the fault management daemon on OmniOS would reduce the amount of work if you had several hundred drives and failures were routine.
SWAP issue on FreeNAS
While I was testing, I had a drive fail (this is why 3-year old Seagate drives are great to test with) and FreeNAS crashed! The NFS pool dropped out from under VMware. When I looked at the console I saw “swap_pager: I/O error – pagein failed” I had run into FreeNAS Bug 208 (removed dead link) which was closed a year ago but never resolved. The default setting in FreeNAS is to create a 2GB swap partition on every drive which acts like striped swap space (I am not making this up, this is the default setting). So if any one drive fails, it can take FreeNAS down. The argument from FreeNAS is that you shouldn’t be using swap—and perhaps that’s true but I had a FreeNAS box with 8GB memory and running only one jail with CrashPlan bring my entire system down because a single drive failed. That’s not an acceptable default setting. Fortunately, there is a way to disable automatically creating swap partitions on FreeNAS, it’s best to disable the setting before initializing any disks.
In my three years of running an OpenSolaris / Illumos based OS I’ve never had a drive failure bring the system down
Running under VMware
FreeNAS is not supported running under a VM (this is now a supported configuration) but OmniOS is. In my testing both OmniOS and FreeNAS work well under VMware under the best practices of passing an LSI controller flashed into IT mode to the VM using VT-d. I found that OmniOS does a lot better virtualized on slower hardware than FreeNAS. On an Avaton C2750 FreeNAS performed well on bare metal, but when I virtualized it using vmdks on drives instead of VT-d FreeNAS suffered in performance but OmniOS performed well under the same scenario.
Both platforms have VMXNET3 drivers, neither has a Paravirtual SCSI driver.
Encryption
Unfortunately, Oracle did not release the source for Solaris 11, so there is no encryption support on OpenZFS directly.
FreeNAS can take advantage of FreeBSD’s GELI based encryption. FreeBSD’s implementation can use the AES instruction set, last I tested Solaris 11 the AES instruction set was not used so FreeBSD/FreeNAS probably has the fastest encryption implementation for ZFS.
There isn’t a good encryption option on OmniOS.
ZFS High Availability
Neither systems supports ZFS high availability out of the box. OmniOS can use a third-party tool like RSF-1 (paid) to accomplish this. The commercially supported TrueNAS uses RSF-1 so it should also work in FreeNAS.
ZFS Replication & Backups
FreeNAS can easily setup replication as often as every 5 minutes which is a great way to have a standby host to failover to. Replication can be done over the network. If you will replicate over the internet, I’d say you want a small data set or a fast connection—I ran into issues a couple of times where the replication got interrupted and it needed to start all over from scratch. On OmniOS Napp-It does not offer a free replication solution, but there is a paid replication feature, however there are also numerous free ZFS replication scripts that people have written such as ZnapZend.
I did get the CrashPlan plugin to work under FreeNAS, however I found that after a reboot the CrashPlan jail sometimes wouldn’t auto-mount my main pool so it ended up not being a reliable enough solution for me to be comfortable with. I wish FreeNAS made it so it wasn’t in a jail.
Memory Requirements
FreeNAS is a little more power hungry than OmniOS. For my 8TB pool a bare minimum for FreeNAS is 8GB while OmniOS is happy with 4GB, although I run it with 6GB to give it a little more ARC.
Hardware Support
FreeNAS supports more hardware than OmniOS. I virtualize my ZFS server so it doesn’t matter too much to me but if you’re running bare metal and on obscure or newer hardware, there’s a much better chance that FreeNAS supports it. Also in 9.3 you can configure IPMI from the web interface.
VAAI (VMware vSphere Storage API’s — Array Integration)
FreeNAS now has VAAI support for iSCSI. OmniOS has no VAAI support. As of FreeNAS 9.3 and Napp-It 0.9f4 both control panels can enable VMware snapshot integration / ESXi hot snaps. The way this works is before every ZFS snapshot is taken FreeNAS has VMware snap all the VMs, then the ZFS snapshot is taken, then the VMware snapshots are released. This is nice and allows for proper consistent snapshots.
GUI
The FreeNAS GUI looks a little nicer and is probably a little easier for a beginner. The background of the screen turns red whenever you’re about to do something dangerous. I found you can set up just about everything from the GUI, where I had to drop into the command line more often with OmniOS. The FreeNAS web interface seems to hang for a few seconds from time to time compared to Napp-It, but nothing major. I believe FreeNAS will have an asynchronous GUI in version 10.
One frustration I have with FreeNAS is it doesn’t quite do things compatible with CLI. For example, if you create a pool via CLI FreeNAS doesn’t see it, you have to import it using the GUI to use it there. Napp-it is essentially an interface that runs CLI commands so you can seamlessly switch back and forth between managing things on CLI and Napp-It. This is a difference in philosophy. Napp-It is just a web interface meant to run on top of an OS, where FreeNAS is more than just a web app on top of FreeBSD, FreeNAS is its own OS.
I think most people experienced with the ZFS command line and Solaris will be a little more at home with Napp-It’s control panel, but it’s easy enough to figure out what FreeNAS is doing. You just have to be careful what you do in the CLI.
On both platforms I found I had to switch into CLI from time to time to do things right (e.g. FreeNAS can’t set sync=always from the GUI, Napp-It can’t setup networking).
As far as managing a ZFS file system both have what I want.. email alerts when there’s a problem, scheduling for data scrubs, snapshots, etc.
FreeNAS has better security, it’s much easier to set up an SSL cert on the management interface, in fact you can create an internal CA to sign certificates from the GUI. Security updates are easier to manage from the web interface in FreeNAS as well.
Community
FreeNAS and OmniOS both have great communities. If you post anything at HardForum chances are, you’ll get a response from Gea and he’s helpful. There is a lot of info on the FreeNAS forums and the FreeNAS Jira project is open so you can see all the issues, it’s a great way to see what bugs and feature requests are out there and when they were or will be fixed. OmniOS has an active OmniOS Discuss mailman list and Gea, the author of Napp-It is active on various forums. He has answered my questions frequently over at HardForum’s Data Storage subforum. I’ve found the HardForum community a little more helpful…I’ve always gotten a response there while several questions I posted on the FreeNAS forums went unanswered.
Documentation
FreeNAS documentation is great, like FreeBSD’s. Just about everything is in the FreeNAS Guide
OmniOS isn’t as organized. I found some howtos here, not nearly as comprehensive as FreeNAS. Most of what I find from OmniOS I find in forums or the Napp-It site.
Mirrored ZFS boot device / rpool
OmniOS can boot to a mirrored ZFS rpool.
FreeNAS does not have a way to mirror the ZFS boot device. FreeBSD does have this capability but it turns out FreeNAS is built on NanoBSD. The only way to get FreeNAS to have redundancy on the boot device that I know of is to set it up on a hardware RAID card.
FreeNAS 9.3 can now install to a mirrored ZFS rpool!
Features / Plugins / Extensions
Napp-It’s extensions include:
- AMP (Apache, MySQL, PHP stack)
- Baikal CalDAV / CardDAV Server
- Logitech MediaServer
- MediaTomb (DLNA / UPnP server)
- Owncloud (Dropbox alternative)
- PHPvirtualbox (VirtualBox interface)
- Pydio Sharing
- FTP Server
- Serviio Mediaserver
- Tine Groupware
FreeNAS plugins:
- Bacula (Backup Server)
- BTSync (Bittorrent Sync)
- CouchPotato (NZB and Torrent downloader)
- CrashPlan (Backup client/server)
- Cruciblewds (Computer imaging / cloning)
- Firefly (media server for Roku SoundBridge and Apple iTunes)
- Headphones (automatic music downloader for SABnzbd)
- HTPC-Manager
- LazyLibrarian (follow authors and grab metadata for digital reading)
- Maraschino (web interfrace for XBMC HTPC)
- MineOS (Minecraft control panel)
- Mylar (Comic book downloader)
- OwnCloud (Dropbox alternative)
- PlexMediaServer
- s3cmd
- SABnzbd (Binary newsreader)
- SickBeard (PVR for newsgroup users)
- SickRage (Video file manager for TV shows)
- Subsonic (music streaming server)
- Syncthing (Open source cluster synchronization)
- Transmission (BitTorrent client)
- XDM (eXtendable Download Manager)
All FreeNAS plugins run in a jail so you must mount the storage that service will need inside the jail… this can be annoying but it allows for some nice security—for example CrashPlan can mount the storage you want to back up as read-only.
Protocols and Services
Both systems offer a standard stack of AFP, SMB/CIFS, iSCSI, FTP, NFS, RSYNC, TFTP
FreeNAS also has WebDAV and few extra services like Dynamic DNS, LLDP, and UPS (the ability to connect to a UPS unit and shutdown automatically).
Performance Reporting and Monitoring
Napp-It does not have reports and graphs in the free version. FreeNAS has reports and you can look back as far as you want to see historical performance metrics.
As a Hypervisor
Both systems are very efficient, running guests of the same OS. OmniOS has Zones, FreeNAS can run FreeBSD Jails. OmniOS also has KVM which can run any OS. I suspect that FreeNAS 10 will have Bhyve. Also both can run VirtualBox.
Stability vs Latest
Both systems are stable, OmniOS/Napp-It seems to be the most robust of the two. The OmniOS LTS updates are very minimal, mostly security updates and a few bug fixes. Infrequent and minimal updates are what I like to see in a storage solution.
FreeNAS is pushing a little close to the cutting edge. They have frequent updates pushed out—sometimes I think they are too frequent to have been thoroughly tested. On the other hand, if you come across an issue or feature request in FreeNAS and report it chances are they’ll get it in the next release pretty quickly.
Because of this, OmniOS is behind FreeNAS on some things like NFS and SMB protocol versions, VAAI support for iSCSI, etc.
I think this is an important consideration. With FreeNAS you’ll get newer features and later technologies while OmniOS LTS is the better platform for stability. The commercial TrueNAS solution is also going to robust. For FreeNAS you could always pick a stable version and not update often—I really wish FreeNAS had an LTS, or at least a slower moving stable branch that maybe only did quarterly updates except for security fixes.
ZFS Integration
OmniOS has a slight edge on ZFS integration. As I mentioned earlier OmniOS has multi-tiered snapshot integration into the Windows Previous versions feature where FreeNAS can only pick one snap-frequency to show up there. Also, in OmniOS NFS and SMB shares are stored as properties on the datasets so you can export the pool, import it somewhere else and the shares stay with the pool so you don’t have to reconfigure them.
Commercial Support
OmniOS offers Commercial Support if you want it.
iX Systems offers supported TrueNAS appliances.
Performance
On an All-in-one setup, I setup VMware ESXi 6.0, a virtual storage network and tested FreeNAS and OmniOS using iSCSI and NFS. On all tests MTU is set to 9000 on the storage network, and compression is set to LZ4. iSCSI volumes are sparse ZVOLs. I gave the ZFS server 2 cores and 8GB memory, and the guest VM 2 cores and 8GB memory. The guest VM is Windows 10 running Crystal Benchmark.
Environment:
- Supermicro X10SL7-F with LSI 2308 HBA flashed to IT firmware and passed to ZFS server via VT-d (I flashed the P19 firmware for OmniOS and then re-flashed to P16 for FreeNAS).
- Intel Xeon E3-1240v3 3.40Ghz.
- 16GB ECC Memory.
- 6 x 2TB Seagate 7200 drives in RAID-Z2
- 2 x 100GB DC S3700s striped for ZIL/SLOG. Over-provisioned to 8GB.
Latest stable updates on both operating systems:
FreeNAS 9.3 update 201503200528
OmniOS r151012 omnios-10b9c79
On Crystal Benchmark I ran 5 each of the 4000MB, 1000MB, and 50MB size tests, the results are the average of the results.
On all tests every write was going to the ZIL / SLOG devices. On NFS I left the default sync=standard (which results in every write being a sync with ESXi). On iSCSI I set sync=always, ESXi doesn’t honor sync requests from the guest with iSCSI so it’s not safe to run with sync=standard.
Update: 7/30/2015: FreeNAS has pushed out some updates that appear to improve NFS performance. See the newer results here: VMware vs bhyve Performance Comparison. Original results below.
Sequential Read MBps
OmniOS is faster at NFS while FreeNAS pulls ahead on iSCSI.
Sequential Write MBps
OmniOS is significantly faster with NFS sequential writes. No significant difference with iSCSI
Random Read 512K MBps
No significant difference on NFS, FreeNAS is faster on random iSCSi reads.
Random Write 512K MBps
OmniOS is faster at random NFS writes. No significant difference for iSCSI.
Random Read 4K IOPS
OmniOS is faster at NFS and iSCSI random reads.
Random Write 4K IOPS
OmniOS is faster wit NFS random writes, no significant difference for iSCSI.
Random Read 4K QD=32 IOPS
No significant difference except when using large blocks on NFS, then FreeNAS slows down.
Random Write 4K QD=32 IOPS
OmniOS tends to do a little better.
Performance Thoughts
So it appears, from these unscientific benchmarks that OmniOS on NFS is the fastest configuration, iSCSI performs pretty similarly on both FreeNAS and OmniOS depending on the test. One other thing I should mention, which doesn’t show up in the tests, is latency. With NFS I saw latency on the ESXi storage as high as 14ms during the tests, while latency never broke a millisecond with iSCSI.
One major drawback to my benchmarks is its only one guest hitting the storage. It would interest to repeat the test with several VMs accessing the storage simultaneously, I expect the results may be different under heavy concurrent load.
I chose 64k iSCSI block size because the larger blocks result in a higher LZ4 compression ratio; I did several quick benchmarks and found 16K and 64K performed pretty similarly, 16K performed better at random 4K write QD=1, but otherwise 64K was close to a 16K block size depending on the test. I saw a significant drop in random performance at 128k. Once again under different scenarios this may not be the optimal block-size for all types of workloads.
Hi BB,
About backup tools, you already tried to “bacula” and “bareos (fork Bacula).”
This seems to me pretty interesting tools.
Hello, Bigame. I actually setup and administered a bacula server at my last job, about 3-4 years ago, it worked pretty well. It’s probably a little more maintenance than I have time for on my home network.
Ben, do you have any experience with NexentaStor?
Hi, Frank. I have a little… I tried NexentaStor 3 years ago when I was first looking at building a ZFS server. The GUI was pretty nice, but I had to switch to Napp-It+OpenIndiana after about a week because the GUI became excruciatingly slow to the point that it would take 5 minutes to respond to each click. I’m not sure why and maybe it’s better now.. But now I don’t even consider NexentaStor for the same reason I don’t consider Oracle Solaris … the NexentaStor Community Edition license says: “Production use is not allowed with Community edition. It is designed and intended to help people become familiar with the product or for personal use.” I use my ZFS to store documents and servers related to consulting, ebay sales, and a rental property so I probably wouldn’t qualify under the “personal use” license… and the free software options are in my opinion better so I just stick with them and I don’t have to worry about whether or not what I’m doing on it is personal and I don’t have to worry about licensing or staying under 18TB RAW (not that I’m anywhere near that). I do think that NexentaStor is a fine product, but as far as licensing goes I prefer freedom.
Hi,
I have almost no experience with Solaris-based operating systems, but there are some features of OmniOS which appeal to me. Would you say OmniOS might be broadly equivalent to Solaris 10, or is it closer to 11? Would I get by with OmniOS by reading the Solaris {10 | 11} documentation, or are they too far apart at this stage?
Regards,
Gerard
Hi, Gerard. I’m not an expert on Solaris based systems, but OmniOS is based on Illumos which is a fork of Solaris 10, so my guess is the Solaris 10 documentation will be closer. Thanks, Ben
(Bit late to the party …)
Actually, not quite so much. OmniOS is much closer to Solaris 11.
All the community supported Solaris-ish OS’s were forked from the OpenSolaris codebase, and that is what became Oracle Solaris 11.
There has never been any community access to the Solaris 10 codebase.
Thanks for the info Renee.
FYI, you can run a phpVirtualbox jail so FreeNAS will become a type-2 hypervisor.
Omni is based on opensolaris which was the development train for solaris 11… so omni is halfway between 10 and 11
Any notes on performance? I have read a lot of stuff that says OI and Napp-IT perform much faster. Unfortunately the benchmarking tools in FreeNAS leave a LOT to be desired. I’d also like to know about fibre target performance, but have not had much opportunity to test that either.
I’m working on some OmniOS vs FreeNAS benchmarks from VMs using storage mounted in ESXi. I may be a month or two out from posting them–will probably just update this post. I don’t have fibre or 10Gb networking or anything so about all I can test is the all-in-one setup. If you do any testing please let me know the results or post a link to them.
There you go, I added some benchmarks results to the post. Your assessment about illumos performing faster appears to be accurate, especially on NFS.
I can’t speak for OmniOS, but the FreeNAS code is pretty ugly. It speaks volumes that they are now working on their third redesign of FreeNAS (FreeNAS 10).
Just wanted to thank you for writing this comparison. Really helpful. I’m a newb trying to set up a FreeNAS as a VM but wanted to do my due diligence before moving forward and throwing sensitive date on it.
Great article – thanks for writing it! I was especially impressed by the follow-up for 9.3. I hope you’ll keep your eye on our progress as we continue to improve 9.3 and get closer to shipping our first public demos of FreeNAS 10.
Thanks Jordan! I’m looking forward to the FreeNAS 10 release. I see that FreeNAS officially supports running under a hypervisor now! http://www.freenas.org/whats-new/2015/05/yes-you-can-virtualize-freenas.html
I was looking at the results and was wondering how many NFS threads you had configured in FreeNAS. The default is 4 and that is probably too low for your tests.
A couple other notes:
1) NFS on FreeNAS is kernel
2) Your results are all off cache. I see things like 80,000 IOPS from 6 7.2k spinning disks.
Hi, Josh.
1. Earlier in my post where I mentioned NFS runs in userspace on FreeNAS, I meant to refer to “SMB” not “NFS” (corrected, but please let me know if I’m incorrect about SMB running in userspace).
2. On my VMware / FreeNAS tests I was using NFS with the default number of threads, 4. I more or less left the sane defaults and didn’t attempt any tuning on either OS as there are a lot of variables (On OmniOS I could have done some sd.conf tuning to bypass cache flush on the DC S3700s, etc.) and I just didn’t want to get into all that. My test VM only had 2 vCPUs so I figured the default of 4 was enough. But I wouldn’t mind retesting with more if you think it will make a difference. You are correct that my results are from cache. I did nothing to try to make the tests large enough to exhaust the ARC. In fact, that might be part of the discrepancy, because I gave both OSes 8GB memory and FreeNAS has a larger memory footprint that means OmniOS probably is able to utilize more RAM for ARC. The difference may go away with more ram… but I can only test with what I have. On your 6 x 7.2k test are you running in RAID-Z2 like me (which I know is not ideal for VMware) or mirrored vdevs?
Thanks,
Ben
Which actually means these benchmarks mean nothing.
Perhaps. I can’t say. |:-)
1. SMB on FreeNAS runs in userspace. (Samba)
2. The random 32 thread tests will benefit from turning up NFS threads from 4. Each I/O has to wait for an nfs thread. If you have fewer threads than the I/O load you are doing you’ve essentially reduced the parallel test to be number of NFS threads. The default is very low because it’s so dependent on pool configuration. If you turn NFS threads up past what the pool will support you end up with a “molasses through a funnel” effect, which is to say that you’ll have a giant mess in the form of unusably high latency and reduced performance. A pool of 6 7.2K disks in a RAIDZ2 may not support turning NFS threads up to 32, however trying to run a test with a queue depth of 32 with NFS threads set less than 32 will bottleneck on NFS threads!
Another factor that can affect anything running ZFS is test blocksize versus ZFS recordsize and pool configuration.
In FreeNAS the default ashift of the pool is 4K. In a 6 drive RAIDZ2 that effectively means the lowest recordsize that is possible is 16K (4K records across 4 data drives) If you try to run a test with 4K blocksizes you’ll end up with read/modify/write causing your pool to do more work than the benchmark reports because your 4K read was really a 16K read from the pool’s perspective.
If your default recordsize is 128K, and your i/o testing tool creates a giant 4GB file for testing, ZFS will happily use the maximum size blocks to create your file (128K) Then if the tool tries to do 4K I/O ZFS is forced to deal with 128K blocks, causing the pool to do 32x the work that the test is requesting. The same thing can happen with iSCSI, however in that case FreeNAS tries to set the volblocksize intelligently based on your pool layout.
There’s another thing that can bite you. Many benchmarking tools use data that compresses well for their tests. ZFS has excellent compression, which oftentimes is enabled by default these days. (It is for FreeNAS) In addition ZFS has special handling for long strings of zeros, even if compression is off. Some testing tools try to use /dev/zero (or it’s equivalent) for their seed data. This is never going to work out well when using ZFS.
While 4K tests are the defacto standard for comparing things, it’s a very uncommon real world workload these days, and if you are comparing ZFS to ZFS probably shouldn’t even be considered, because it’s a pathological worst case for ZFS, especially in a RAIDZ configuration. Unfortunately if you are trying to get IOPs numbers increasing the test blocksize doesn’t help you too much because especially at GigE you start to bang into bandwidth constraints. GigE will only support ~8000 IOPs at 128K. Which is enough for your disks, but as you’ve seen on cache nearly an order of magnitude low. As a general rule of thumb a vdev of 7.2K disks will support 100 random IOPs
The end result is ZFS is really hard to benchmark well. It basically goes out of it’s way to foil benchmarks.
As far as your question about my results I wasn’t being clear. I meant I see in your results things like 80,000 IOPs from your RAIDZ2 of 6 spinning disks, which isn’t possible from the disks. :) If you suspect memory footprint of the OS is affecting cache hit rate (which is likely true) you can either zfs set primarycache=metadata pool/dataset to disable cache altogether, or report cache hit rates (visible in the FreeNAS reporting GUI, or via the cli when running arc_summary.py)
If you want to reach out to my privately feel free to do so. I have extensive experience benchmarking ZFS and my “brain dump” here is likely incomplete, and possibly misleading due to lack of adequate proof-reading. Someday I mean to write an exhaustive “Everything you need to know about benchmarking ZFS performance” but so far I’ve been unable to find the time for that.
Thanks Josh. I really appreciate your commenting here. The longer I’ve been watching iX Systems the more I’ve been impressed with the community support (for free software!) the business model and the employees there.
Good to know on the threads, I will increase the NFS threads and see what happens, may take me a few months since I’m in the middle of other things but worth the test (if anyone beats me to feel free to comment a link to your results).
I agree on 4K, I used it because it’s the “standard” and it takes 5 seconds to install CrystalDiskMark which only does 4K. Another benchmark tool I’ve used in the past is sysbench (e.g. https://b3n.org/ssd-zfs-zil-slog-benchmarks-intel-dc-s3700-intel-dc-s3500-seagate-600-pro-crucial-mx100-comparison/), do you think sysbench (especially things like the OLTP test) is a better indicator of real world performance?
The pool was set to LZ4 compression. On CrystalDiskMark I had data set to “Random”. The NFS / iSCSI tests are using 10GbE virtual VMware adapters (using the VMXNET3 driver).
ZFS is indeed hard to benchmark. It’s very time consuming to benchmark with enough data to exhaust the ARC and probably would have taken me several months to do in my free time and then if I made one mistake I’d have to throw out the results and start over so it may have been years. |:-) Fortunately for me my arc cache hit ratio is usually > 95% so benchmarking data that fits in ARC is actually a good performance indicator for me.
When comparing two systems, I make every effort to treat both equally, so at the very least I can say the tests are flawed in the same way for both. I triple checked that they were both running the same ZFS settings. E.g. LZ4, sync=standard for NFS, sync=always for iSCSI, atime disabled, both used ZVOL (even though Napp-It prefers file extents) for iSCSI, CrystalDiskMark with random data, etc. I didn’t try tuning the OS / service settings (like NFS threads) and instead opted to keep the defaults (I did think about this, but figured it would take me years to test every setting). But like you mentioned, these are not real world numbers, just benchmarks. All we can say from my results is how FreeNAS and OmniOS performed on this hardware with these settings in my environment.
When FreeNAS 10 is released I hope to benchmark both again if I have hardware available. I’ll be sure to drop you a line to get some advice ahead of time.
Replying from my phone this time.
Your points about testing with ZFS “on cache” are very valid. Most people do have high cache hit rates, so these numbers are of interest. (An interesting test would give 3D results as you’d add cache hit % to the mix. Sadly I don’t have time for that anymore than you do!)
Now my curiousity is piqued enough that I’m going to set up a repro case and unwind why there’s a performance difference in NFS.
Furthermore it’s time to re-evaluate the default settings. FreeNAS has an “autotuner” maybe it’s time to improve that to look at pool and network configuration. Your configuration is not at all “extreme” and if my position is the defaults aren’t good enough…
To be honest it wouldn’t surprise me at all if even with expert tuning to both systems OmniOS outperforms FreeNAS. Both ZFS and NFS came from Sun after all. However I’d like to see the difference be a few percentage points, not the large differential your tests showed.
At any rate, thank you for sharing your methodologies and taking the time to do this.
Hi, after CP announced end of life on the Solaris platform, I simply set up an ubuntu server VM on ESXi, shared the pool via NFS, and set up CP on Ubuntu. Works like a charm. Probably uses a little more resources, but not that much.
That’s good to know, Jon. I wasn’t if CrashPlan would allow that but it’s great that it does!
Yes, CP has no clue that it is not a native drive. Auto mounted via via fstab. Bonus is that CP client on Linux is 64 bit (versus 32 bit on Solaris), hence you can use more memory. I kept hitting the limit, and CP crashed (more data backed up = more memory consumption), but that is resolved in Linux due to 64 bit client.
My guess is that that is one reason to stop supporting Solaris is that they are soon finished with the native CP clients, and removed a platform to make it easier to support. The java client is a memory hog, so a native CP client will be welcomed with open arms.
Hmm.. I believe the FreeBSD linux emulation is only 32-bit. How much data are you backing up?
Don’t know about that. Max memory on 32bit is 3072 if I remember correctly.
According to the CP client I currently have (including moved/deleted files): 2,157,859 files, 150,500 directories, 10,3 TB (about 4TB is “active” data – i.e not deleted, moved, etc.). That requires more than 3 gig of memory.
In this post you use “2 x 100GB DC S3700s striped for ZIL/SLOG. Over-provisioned to 8GB” but in a prior post you use just “One DC S3700 100GB over-provisioned to 8GB used as the log device.”
I assume One (1) ZIL/SLOG (above) is the correct best practice – correct ?
IMHO mirroring ZIL/SLOG buys you nothing but expense, because SLOG is only used if something happens to the in-memory ZIL so in some non homogeneous sense your data is already “mirrored” – so the key is to focus on “power loss protection” for your SSD which comprises your SLOG.
Hi, Jon.
In some articles I’m only using 1 SLOG device because I didn’t have room in the chassis or I was currently using the other in another project so couldn’t spare 2. But my preferred setup with the drives I have is to run two striped for the SLOG–that’s because I have 100GB models which have lower write bandwidth. If I had a 200GB or 400GB model I’d probably just run a single drive.
Mirroring does have a slight stability enhancement. If you have a performance requirement and don’t want to lose performance when a ZIL fails then mirroring is a good way to mitigate that risk. Also, I have done a half a dozen SLOG drive pulls to simulate a SLOG failure during writes and a couple of the pulls produced a kernel panic in FreeNAS and I had to hard reset. Which means all writes in flight were lost. I’m not sure if pulling a drive out of a hotswap bay is similar to a drive dying, but assuming so it seems to me that it is possible that a SLOG failure can result in a kernel panic and in that scenario having a mirrored ZIL would be beneficial. Personally I think the risk is small, but depending on what you’re doing it’s something to consider. I would say having a second device to mirror your data to every 5 minutes would be your first priority over mirroring the SLOG device.
Striping has the advantage of increasing your write bandwidth, this is especially useful if all your writes are sync writes. The max theoretical write speed on a 100GB DC S3700 is 200MB/s, so if you need more you can get a larger model that supports more bandwidth (like a 200GB, or 400GB model), or in my case just stripe them. I think having a single faster drive is better than striping the ZIL.
Ben,
I based my opinions comments like “Well, the SLOG would have to die. But just the SLOG dying is not enough. (remember that the data is still in RAM, and written to the pool from there). In addition to the SLOG dying, you would also need to lose power (or otherwise freeze the system) in the second or so before the next write cycle from RAM completes. How likely is this anyway? On a stable system with a UPS, I would consider it VERY unlikely of this to happen.” https://forums.freenas.org/index.php?threads/time-to-reevaluate-the-mirror-your-slog-zil-drive-recommendation.23445/
Thanks for your explanation you and http://nex7.blogspot.com/ seem to share the same opinion and it seems pretty compelling.
“(optional but strongly preferred) Get a second dedicated log device (of the exact same type as the first), and when creating the log vdev, specify it as a mirror of the two. This will protect you from nasty edge cases.” http://nex7.blogspot.com/2013/04/zfs-intent-log.html
If there is comprehensive backup strategy (one that handles not only backing up the data, but making it quick and easy to restore as well) where a bit of down time is fine – I would still skip a mirror ZIL/SLOG. But abscens a good backup plan and given an extra drive bay (which sometimes isn’t available) and the budget I would mirror ZIL/SLOG in the future, but it’s a preference clients are sometimes very cheap.
Hi, Jon. Yeah, in a perfect world the failure of SLOG shouldn’t cause a kernel panic. But in my world it can. |:-)
Also, in my opinion (and I understand that cost is driven by clients), you should never put a ZFS system in production without a way to mirror the pool to a backup host. I have been able to product the infamous ZFS stuck log device bug in FreeNAS and OmniOS by pulling the SLOG during writes to simulate failure. I have been able to reproduce this on two completely different builds usually within 10 pull tests. When this stuck log device condition occurs the pool is still available but performance suffers because every write tries to go to the failed ZIL. You can’t remove it or replace it with a new one. The only way I’ve been able to recover is to zfs send the dataset over to a backup host, destroy the pool, rebuild it, and send the data back. And really the same goes to any SAN, not just ZFS. If the business can’t afford to be down a day because of a storage failure you should always have a hot backup ready to go.
First I would like to thank you for the great blog as I want to replace my current Windows Home Server running on HP MicroServer N36L Athlon2 1,3GHZ DualCore something by that hase better SMB / CIFS performance.
I was planning to use more or less exactly your tested HW setup.
Now coming to my question was it really only 114Mbps which means about 14 Mbyte per second you achieved by that powerful HW setup via SMB /CIFS?
Thanks!
/Oliver
Hi, Oliver. You’re welcome. It was 114MBps–Megabytes per second–where 125MBps is the theoretical maximum over gigabit. I had Intel NICS on both sides.
You may be interested in my https://b3n.org/supermicro-x10sdv-f-build-datacenter-in-a-box/ build as it’s pretty similar in form factor to the Microserver, but you can go up to 64GB or even 128GB on that motherboard. The main downside is only four hotswap bays like the Microserver.
Hi Ben,
I’m currently running OI and have been considering a move for a while considering the platform doesn’t get much love. The fact that CrashPlan are removing support for Solaris is pretty much the nail in the coffin. Have you had any experience with ZFS-on-Linux? I’m really torn between doing it myself on FreeBSD, just using FreeNAS, and running up Ubuntu with ZOL.
I will give you fair warning that CrashPlan on FreeNAS/FreeBSD tends to overwrite it’s run.conf file after every upgrade so you’ll need to fix it fairly frequently or write a script to fix it for you. https://forums.freenas.org/index.php?threads/crashplan-3-7-0-update-how-to-fix.26873/ If you want to stick with OI I’ve heard from several people you can run CrashPlan on a Linux VM somewhere and mount storage from OI using NFS.
For ZFS-on-Linux, they claim to be stable so it may be okay, but I think FreeBSD/FreeNAS and OI have been more battle tested in that area so personally I stick with OI, FreeBSD, and FreeNAS. If I was looking to switch away from OI just for CrashPlan I’d do the backup via NFS route. Of course if you want to switch just to try something different that may be a good reason. |:-)
Hi Ben,
I really enjoy your blogs posts. I’ve found them very useful. I recently decided to give OmniOS a go and I’m wondering where/how in OmniOS/Nappit does one setup the snapshot and previous version visibility you mentioned in this post? Is this something “built out of the box” or have you created scripts for this?
Thanks!
Hi, Dave. Thank you, the Previous versions feature in OmniOS / Napp-It works out of the box, there’s no setup needed. Hope you enjoy your system!
Ben
Thanks Ben! What tool did you use to generate the data for your benchmarks? I’m trying to run some myself to see if everything is setup properly.
If you want enclosure management and map your drives to enclosure slot, there is a great script for sas2ircu tool available here: https://github.com/eNiXHosting/DiskMap
It nicely maps drives to case/slot and it also shows in which pool the drive resides… I love it for managing my 200 drives in 6 jbods.
A new take on this would be interesting regarding FN11 and current state of OmniOS
I know the zfs commands more closely resemble those of Solaris 11 than 10. It gets confusing when looking up how to do things in ZFS as sometimes the Oracle docs are not clear about which version of Solaris they are for.
Just a note about pvscsi / vmware paravirtual driver. It’s available in standard pkg repo in OmniOS 151028. I just did a fresh install using LSI parallel, installed pvscsi and open-vm-tools, halted the VM and switched the bus type in VCSA to paravirtual and it booted up without a hitch.