SSD ZFS ZIL SLOG Benchmarks – Intel DC S3700, Intel DC S3500, Seagate 600 Pro, Crucial MX100 Comparison

I ran some performance tests comparing the Intel DC S3700, Intel DC S3500, Seagate 600 Pro, and Crucial MX100 when being used as a ZFS ZIL / SLOG Device.  All of these drives have a capacitor backed write cache so they can lose power in the middle of a write without losing data.

IMG_9271

Here are the results….

zfs_zil_ssd_comparison

The Intel DC S3700 takes the lead with the Intel DC S3500 being a great second performer.  I am surprised at how much better the Intel SSDs performed over the Crucial and Seagate considering Intel’s claims are not as great as the other two… Intel claims slower IOPS, and slower sequential performance yet it outperforms the other two SSDs.

SSDSeagate 600 ProCrucial MX 100Intel DC S3500Intel DC S3700
Size GB24025680100
ModelST240FP0021CT256MX100SSD1SSDSC2BB080G4SSDSC2BA100G3T
Sequencial (MB/s)500550Read 340 / Write 100Read 500 / Write 200
IOPS Random Read85,00085,00070,00075,000
IOPS Random Write11,00070,0007,00019,000
Endurance (Terabytes Written)13472451874
Warranty5 years3 Years5 years5 years
ZFS ZIL SLOG Results Below--------
oltp 2 thread66160732746
oltp 4 thread892349891008
random r/w20854314543641
random write8722618441661
seq write MB/s5149399

Drive Costs

Conclusion

  • For most workloads use the Intel DC S3500.
  • For heavy workloads use the Intel DC S3700.
  • The best performance for the dollar is the Intel DC S3500.

In my environment the best device for a ZFS ZIL is either the Intel DC S3500 or the Intel DC S3700.  The S3700 is designed to hold up to very heavy usage–you could overwrite the entire 100GB drive 10 times every day for 5 years!  With the DC S3500 you could write out 25GB/day every day for 5 years.  Probably for most environments the DC S3500 is enough.

I should note that both the Crucial and Seagate improved performance over not having a SLOG at all.

Unanswered Questions

I would like to know why the Seagate 600 Pro and Crucial MX100 performed so badly… my suspicion is it may be the way ESXi on NFS forces a cache sync on every write, the Seagate and Crucial may be obeying the sync command while the Intel drives are ignoring it because they know they can rely on their power loss protection mechanism.  I’m not entirely sure this is the difference but it’s my best guess.

Testing Environment

This is based on my Supermicro 2U ZFS Server Build:  Xeon E3-1240v3, The ZFS server is a FreeNAS 9.2.1.7 running under VMware ESXi 5.5.  HBA is the LSI 2308 built into the Supermicro X10SL7-F, flashed into IT mode.  The LSI 2308 is passed to FreeNAS using VT-d.  The FreeNAS VM is given 8GB memory.

Zpool is 3x7200RPM Seagate 2TB drives in RAID-Z, in all tests an Intel DC S3700 is used as the L2ARC.  Compression = LZ4, Deduplication off, sync = standard, encryption = off.  ZFS dataset is shared back to ESXi via NFS.  On that NFS share is a guest VM running Ubuntu 14.04 which is given 1GB memory and 2 cores.  The ZIL device is changed out between tests, I ran each test seven times and took the average discarding the first three test results (I disregarded the first three results to allow some data to get cached into ARC…I did not see any performance improvement after repeating a test three times so I believe that was sufficient).

Thoughts for Future Tests

I’d like to repeat these tests using OmniOS and Solaris sometime but who knows if I’ll ever get to it.  I imagine the results would be pretty close.  Also, of particular interest would be testing on  VMware ESXi 6 beta… I’d be curious to see if there are any changes in how NFS performs there… but if I tested it I wouldn’t be able to post the results because of the NDA.

Test Commands

Intel DC S3500 vs S3700 as a ZIL SSD Device Benchmarks

I ran some benchmarks comparing the Intel DC S3500 vs the Intel DC S3700 when being used as a SLOG/ZIL (ZFS Intent Log).  Both SSDs are what Intel considers datacenter class and they are both very reasonably priced compared to some of the other enterprise class offerings.

Update: 2014-09-14 — I’ve done newer benchmarks on faster hardware here: https://b3n.org/ssd-zfs-zil-slog-benchmarks-intel-dc-s3700-intel-dc-s3500-seagate-600-pro-crucial-mx100-comparison/

IMG_20140817_222031

SLOG Requirements

Since flushing the cache to spindles is slow, ZFS uses the ZIL to safely commit random writes.  The ZIL is never read from except in the case power is lost before “flushed” writes in memory have been written to the pool.  So to make a decent SLOG/ZIL the SSD must be able to sustain a power loss in the middle of a write without losing or corrupting data.  The ZIL translates random writes to sequential writes so it must be able to sustain fairly high throughput.  I don’t think random write IOPS is as important but I’m sure it helps some. Generally a larger SSD is better because they tend to offer more throughput.  I don’t have an unlimited budget so I got the 80GB S3500 and the 100GB S3700 but if you’re planning for some serious performance you may want to use a larger model, maybe around 200GB or even 400GB.

Specifications of SSDs Tested

Intel DC S3500 80GB

  • Seq Read/Write: 340MBps/100MBs
  • 70,000 / 7,000 4K Read/Write IOPS
  • Endurance Rating: 45TB written
  • Price: $113 at Amazon

Intel DC S3700 100GB

  • Seq Read/Write: 500MBs/200MBs
  • 75,000 / 19,000 4K Read/Write IOPS
  • Endurance Rating: 1.83PB written (that is a lot of endurance).
  • Price: $203 at Amazon

Build Specifications

Virtual NAS Configuration

  • FreeNAS 9.2.1.7 VM with 6GB memory and 2 cores.  Using VMXNET3 network driver.
  • RAID-Z is from VMDKs on 3×7200 Seagates.
  • SLOG/ZIL device is a 16GB vmdk on the tested SSD.
  • NFS dataset on pool is shared back to VMware.  For more information on this concept see Napp-in-one.
  • LZ4 Compression enabled on the pool.
  • Encryption On
  • Deduplication Off
  • Atime=off
  • Sync=Standard (VMware requests a cache flush after every write so this is a very safe configuration).

Don’t try this at home: I should note that FreeNAS is not supported running as a VM guest, and as a general rule running ZFS off of VMDKs is discouraged.  OmniOS would be better supported as a VM guest as long as the HBA is passed to the guest using VMDirectIO.  The Avoton processor doesn’t support VT-d which is why I didn’t try to benchmark in that configuration.

Benchmarked Guest VM Configuration

  • Benchmark vmdk is installed on the NFS datastore.  A VM is installed on that vmdk running Ubuntu 14.04 LTS, 2 cores, 1GB memory.  Para-virtual storage.
  • OLTP tests run against MariaDB Server 5.5.37 (fork from MySQL).

I wiped out the zpool after every configuration change, and ran each test three times for each configuration and took the average (in almost all cases subsequent tests ran faster because the ZFS ARC was caching reads into memory so I was very careful to run the tests in the same order on each configuration.  If I made a mistake I rebooted to clear the ARC).  I am mostly concerned with testing random write performance so these benchmarks are more concerned with write IOPS than with throughput.

Benchmark Commands

Random Read/Write:
# sysbench –test=fileio –file-total-size=6G –file-test-mode=rndrw –max-time=300 run

Random Write:
# sysbench –test=fileio –file-total-size=6G –file-test-mode=rndwr –max-time=300 run

OLTP 2 threads:
# sysbench –test=oltp –oltp-table-size=1000000 –mysql-db=sbtest –mysql-user=root –mysql-password=test –num-threads=2 –max-time=60 run

OLTP 4 threads:
# sysbench –test=oltp –oltp-table-size=1000000 –mysql-db=sbtest –mysql-user=root –mysql-password=test –num-threads=4 –max-time=60 run

Test Results

zil_random_read_write zil_random_write zil_oltp_4_theads zil_oltp_2_threads

 

SLOGTestTPS Avg
noneOLTP 2 Threads151
Intel DC 3500 80GBOLTP 2 Threads188
Intel DC 3700 100GBOLTP 2 Threads189
noneOLTP 4 Threads207
Intel DC 3500 80GBOLTP 4 Threads271
Intel DC 3700 100GBOLTP 4 Threads266
noneRNDRW613
Intel DC 3500 80GBRNDRW1120
Intel DC 3700 100GBRNDRW1166
noneRNDWR273
Intel DC 3500 80GBRNDWR588
Intel DC 3700 100GBRNDWR569

 

Surprisingly the Intel DC S3700 didn’t offer much of an advantage over the DC S3500.  Real life workload results may vary but the Intel DC S3500 is probably the best performance per dollar for a SLOG device unless you’re concerned about write endurance in which case you’ll want to use the DC S3700.

Other Observations

There are a few SSDs with power loss protection that would also work.  The Seagate 600 Pro SSD, and also for light workloads a consumer SSDs like the Crucial M500 and the Crucial MX100 would be decent candidates and still provide an advantage over running without a SLOG.

I ran a few tests comparing the VMXNET3 vs E1000 network adapter and there is a performance penalty for the E1000.  This test was against the DC 3500.

freenas_vmxnet_vs_e1000

NetworkTestTPS Avg
E1000gOLTP 2 Threads187
VMXNET3OLTP 2 Threads188
E1000gOLTP 4 Threads262
VMXNET3OLTP 4 Threads271
E1000gRNDRW1101
VMXNET3RNDRW1120
E1000gRNDWR564
VMXNET3RNDWR588

I ran a few tests with Encryption on and off and found a small performance penalty for encryption.  This test was against the DC S3700.

freenas_encryption_on_vs_off

EncryptionTestTPS Avg
OffOLTP 2 Threads195
OnOLTP 2 Threads189
OffOLTP 4 Threads270
OnOLTP 4 Threads266
OffRNDRW1209
OnRNDRW1166
OffRNDWR609
OnRNDWR569