Solaris 11, OpenIndiana, and NexentaStor benchmarks

Purpose: Determine fastest small NAS/SAN OS/Appliance software solution for home running under ESXi.
Constraints: Candidates must support ZFS, be relatively stable (I narrowed down my candidate list by only choosing solutions that other people have used successfully), and be able to run CrashPlan for data protection.

If you’re not familair with ZFS (Zettabyte File System), read ZFS – The Last Word on File Systems

OS/Appliance candidates

Oracle Solaris
  1. Solaris 11 + napp-it.  The latest OS release from Oracle.  For this test the text installer was used.  Solaris 11 is free for development and personal use but a support license is required for commercial use.
  2. OpenIndiana Server 1.51a + napp-it.  The fork off of Open Solaris when Oracle closed the project.    For this test the server installation ISO was used.  Open Indiana is free to use with no restrictions.
  3. NexentaStor 3.1 Community Edition.  Based on Open Solaris with a Debian based user-land.  Nexenta has built a great web GUI making it easy to setup a ZFS system.  I used Nexenta’s built-in gui instead of napp-it.  NexentaStor Update 2/19:  is free to use with up to 18TB of net storage (a limit that would be pretty hard to hit for a small home SAN/NAS), after that a license is required. does require a license for anything more than development and evaluation.
napp-it, is a web gui for running on top of OpenIndiana, Solaris 11, Nexenta, and a few others written by Günther Alka (aka Gea).  It’s one command to install and you have a web-interface to your OS for setting up your storage appliance.  I think Nexenta’s GUI is a little more polished and faster (you get stats, graphs, Ajax, etc.), but Gea’s Napp-it is solid and very light-weight.  When joining a WinServer 2008 R2 domain Napp-It had no problems, but I had to drop to a command shell to tweak some settings to get Nexenta to join.  The ACLs were also a bit easier to get going in Napp-It.  Overall I would say Nexenta is a little easier to use than Napp-It (I usually don’t have to dig too far into the menus to find something in Nexenta), but Napp-it won’t eat up as much memory.
Below is the testing environment:
HP Microserver

Hardware Configuration
  • HP ProLiant Microserver
  • CPU: AMD Turion II Neo 1.5GHz
  • Memory: 8GB (2x4GB) PC3 – 10600 ECC
  • HD: 250GB Seagate 7200 RPM SATA (I really wanted to perform the test with more drives but with the prices of drives after the Thailand floods it wasn’t affordable.)
  • Everything else is stock configuration.

Virtualized Environment

  • VMWare ESXi 5.0 installed on a 2GB SanDisk Cruzer
  • The SATA controller on the Microserver does not support pass through on ESXi so I wasn’t able to give the OS direct access to the drive.  VM files were used for the drives.
VM Configuration
Each OS was installed on the Seagate drive, it was given 4GB Memory, 30GB of storage for the OS and 40GB for the share/test drive, and both CPU cores.  During testing all other VMs were powered down.

Gigabit network all around, using Cat-5e cables (sorry, I don’t have any Cat-6 cables) through as Cisco/Linksys E3000 (gigabit router) running TomatoUSB.    I shutdown all programs that I thought would use a significant amount of network traffic.

NAS Share configuration
Shares were mounted via CIFS/SMB.  I did not test NFS since I don’t have a system outside the Virtual Server that can mount NFS shares.  CIFS shares were using standard sync, the default compromise between safety and performance.
SAN Share Configuration
The difference between a NAS and a SAN is a NAS shares a file-system, multiple people can access that system at the same time.  A SAN shares volumes at the block level so it’s more suitable for running a database on top of, but only one client can mount a SAN target (volume) at a time.  iSCSI targets were mounted from my Windows 7 desktop.  I setup the iSCSI pools to use up to 128K blocks, after thinking about it I should have used 512 byte blocks since that’s what the OS would expect, ZFS will automatically use the smallest block-size it can to fit the file-system blocks so the block-size would have been whatever NTFS defaults to when formatting a 10GB drive.
Testing Software
Crystal Mark 3.0.1 x64.  Tests set to repeat 3 times each.  Test size 100MB, test data set to random.

Tests Performed
  • Seq = Sequential Read/Write of 100MB.
  • 512K = Random Read/Write of 512K files (100MB worth).
  • 4K =  Random Read/Write of 4K files (100MB worth).
  • 4KQD32 =  Random Read/Write of 4K files (100MB worth) using a queue depth of 32.
ZFS Configurations
  • Standard – Typical ZFS share with nothing special.
  • Compression – LZJB Compression of each block.
  • Compress_Dedup – LZJB Compression of each block, and block level deduplication.
  • Gzip – Gzip-6 compression of each block.
  • Gzip9 – Gzip-9 compression of each block.
  • iCSCI, Compression – iSCSI target (block level shares) using LZJB compression.
Shortcomings with this test
  • My network should be on Cat-6, but I’m too cheap.
  • If my SATA-controller supported pass-through with ESXi, ZFS would use the disk cache resulting in faster speeds.
  • I only had a single 7200RPM HD available, we could have seen different results (likely faster across the board) with 3 or 4 drives in a RAID-Z setup, but that’s not going to happen until prices normalize.

Here are the test results.

Below are some specific OS Graphs (note that the scale changes with each one).
Nexenta Reads


Nexenta Writes


OpenIndiana Reads


OpenIndiana Writes



Solaris 11 Reads


Solaris 11 Writes
Pivot Data
Column Labels
Row LabelsNexentaStoreOpenIndianaSolaris11
Average of Seq31.8330.2829.49
Average of 512K32.3430.533.33
Average of 4K4.0374.0284.406
Average of 4KQD327.7137.3926.94
Average of Seq27.235.0775.65
Average of 512K37.4822.5967.54
Average of 4K3.2442.0823.992
Average of 4KQD321.4691.3224.261
Compress_Dedup Average of Seq29.51532.67552.57
Compress_Dedup Average of 512K34.9126.54550.435
Compress_Dedup Average of 4K3.64053.0554.199
Compress_Dedup Average of 4KQD324.5914.3575.6005
Average of Seq32.7728.6133.77
Average of 512K31.8531.2430.3
Average of 4K4.10723.884.576
Average of 4KQD328.0354.5827.892
Average of Seq48.973.2176.56
Average of 512K24.0523.8872.42
Average of 4K2.4992.7634.022
Average of 4KQD321.9972.8164.667
Compression Average of Seq40.83550.9155.165
Compression Average of 512K27.9527.5651.36
Compression Average of 4K3.30313.32154.299
Compression Average of 4KQD325.0163.6996.2795
Average of Seq32.330.4129.68
Average of 512K32.0730.4334.5
Average of 4K4.0573.7924.19
Average of 4KQD328.017.0537.187
Average of Seq13.1315.9257.24
Average of 512K12.8130.744.96
Average of 4K0.50.9324.487
Average of 4KQD320.5110.6444.404
Gzip Average of Seq22.71523.16543.46
Gzip Average of 512K22.4430.56539.73
Gzip Average of 4K2.27852.3624.3385
Gzip Average of 4KQD324.26053.84855.7955
Average of Seq30.1128.5130.58
Average of 512K32.2731.4631.42
Average of 4K4.2274.6524.571
Average of 4KQD328.0817.0517.432
Average of Seq12.9215.4355.88
Average of 512K14.8416.6537.9
Average of 4K0.520.7943.93
Average of 4KQD320.470.5544.014
Gzip-9 Average of Seq21.51521.9743.23
Gzip-9 Average of 512K23.55524.05534.66
Gzip-9 Average of 4K2.37352.7234.2505
Gzip-9 Average of 4KQD324.27553.80255.723
iSCSI, Compression
Average of Seq78.0481.9679.08
Average of 512K7173.968.94
Average of 4K3.8544.6193.59
Average of 4KQD3256.0760.1370.82
Average of Seq27.9437.8140.14
Average of 512K23.545.8546.35
Average of 4K3.4623.8324.274
Average of 4KQD3210.720.99429.99
iSCSI, Compression Average of Seq52.9959.88559.61
iSCSI, Compression Average of 512K47.2559.87557.645
iSCSI, Compression Average of 4K3.6584.22553.932
iSCSI, Compression Average of 4KQD3233.39530.56250.405
Average of Seq29.9331.5332.84
Average of 512K32.4931.2733.59
Average of 4K4.1144.6194.687
Average of 4KQD328.1687.4827.634
Average of Seq50.1711.8160.99
Average of 512K26.558.90266.44
Average of 4K4.054.284.501
Average of 4KQD323.1214.4654.431
Standard Average of Seq40.0521.6746.915
Standard Average of 512K29.5220.08650.015
Standard Average of 4K4.0824.44954.594
Standard Average of 4KQD325.64455.97356.0325


Nearly five months later…

Finally finished my 189.7GB backup to Crashplan’s servers.  Started uploading on July 25th and completed on December 17th.  Didn’t notice too much of an internet slowdown thanks to Tomato’s QoS routing to prioritize traffic during the process.

Tomato Screenshot of Bandwidth Usage
Tomato Outbound QoS rules to limit CrashPlan
Outbound QoS rules




This is a great example of someone recognizing something we all hate and making it better.  We all despise our thermostats.  And for good reason.  A simple task like changing the temperature shouldn’t be very confusing, but the thermostat designers have made it complicated.

My first apartment had this Energy Star programmable thermostat.  I think you had to flip the far right switch to heat or cool, the next switch to auto,  turn the knob to “run”, press up or down until the desired temperature displayed, then press “hold” which turns the air on and causes an “Override” message to be displayed.  “Override” as far as I can tell means this setting will be overridden in about an hour and will have to set again.

When we bought a house that came with an electromechanical thermostat I thought it was the best thing in the world.  Simple to program, it sure beat the digital thermostat.  But if you don’t have a consistent schedule (say you take a day off, or have different climate control needs on the weekends) it can be a little annoying to reprogram for exceptions.

Now, there’s a company called Nest Labs, started by a former Apple employee who hated his confusing thermostat, that’s come out with a better thermostat.  Really, all you need to do is turn it up and down and it learns from your patterns and starts adjusting the temperature itself (you can always manually override it).  Turn the heat down before you go to bed and it learns to turn itself down automatically.

It connects to your Wifi network so you can adjust the temperature and view reports from your computer or iPhone/Android app…

Ben’s Android App Picks

Updated 10/30/2011

Great improvements for Financial apps this year, most decent banks and brokerages have rolled out mobile deposit allowing you to deposit checks by merely taking pictures of them.

Schwab Deposit Check
Schwab Mobile – Mobile (camera) deposits, trading, transfers.

Fidelity Investments  – Mobile (camera) deposits, billpay, trading (supports trailing stop loss trades), market news, transfers.

(Both Schwab and Fidelity have a $10,000 daily mobile deposit limit, a lot better than J.P. Morgan Chase’s $1,000 limit which makes their mobile deposit app useless).

ING DIRECT – Billpay, transfers, P2P payments.

HP 12 C
Andro12C – This is the best HP 12c emulator for Android that I’ve come across.  The HP 12c is now 30 years old, and also happens to be my favorite financial calculator.  I use it to calculate interest on loans, mortgages, investments, depreciation schedules, etc.

QuickRegister – Record cash transactions then export QIF file into Quicken.

This has been a good year for Bible apps, YouVersion’s Bible app continues to improve and offer more free translations.  Also we also saw the introduction of Logos Bible Software, which links your commentaries and books with scripture references.

Logos Bible Study App
Logos Bible Software – The best Bible Study app.  It can do word studies, translation comparisons, but where it really shines is if you own one of the Logos Base Packages it will synchronize with your library so you have access to any commentary, hymnal, book that you own are licensed to access, and can run a Passage Guide which will pull up every commentary entry related to that passage.  Logos did a good job.

Bible (YouVersion) – Best free Bible reading app.  It can read to you, most popular Bible translations are free (including  NIV, ESV, NLT, NKJV, AMP, NASB, CEV, NET, WEB, NCV, TNIV, HCSB, as well as translations in foreign languages).  For publishers that allow it the Bibles can be downloaded for offline use.  I’m not interested in the social stuff so I haven’t tried it but it also does social interaction during a service (I guess you can comment and ask questions to other people with the app at the same event) with YouVersion Live, and also has hooks into Facebook, and Twitter.

Kindle (Amazon) – Can’t go wrong with Amazon’s Kindle App.  Free classics (G.K. Chesterton, Lewis Carroll, etc.)

Most weather apps could still use some improvements, typically what people want to know is what it’s going to do and when… and most require too many taps to get there.  One weather app did it right… but what I’d still like to see is what will today be compared to yesterday so I’d know to prepare for a warmer or colder day.

AIX Weather Widget

Aix Weather Widget – Quick weather view of the next 24 hours.

Business Calendar – ($4.99, I got it for free from the Amazon App Store) – Much better than the stock calendar, especially since the month view can show text entries instead of colored blocks.


Handy Calc
HandyCalc – The most powerful calculator.  Graphic calculater, currency converter, unit of measure conversions, fraction calculations, formula solving.

Google Voice – Transcribes your voicemail, intercepts dialer to make outgoing calls as your Google Voice number, free text messaging.

Gmail – Native Gmail App

Flickr – Finally, a flickr app!


Evernote -Note taking app (syncs with Evernote on your computer).

Amazon MP3 – Stream music from your Amazon library.

Pandora – Calm Eli down with music.

Amazon Store – Scan barcodes to see if Amazon beats the price (they usually do).

Amazon App Store – One fee paid app per day.  Some are even good.


Drudge Report
Drudge Report – Can’t do wrong with Matt Drudge..

The Wal Stree Journal Mobile – Read stuff that doesn’t make the socialist news.

Call Accounting Lite – Export phone calls logs to CSV (great for tracking how much time you spend on the phone with clients).

Other apps

Parcels – Track packages from just about any carrier.

Google Maps – Turn by turn GPS Navigation

My Tracks – Track your walks and bike rides using your GPS.  Shows you statistics like your average moving speed and elevation change.

Technology Geek Apps

Wifi Analyzer
Wifi Analyzer – Find out what W-Fi channels will get the least interference.

APG – PGP encryption for Android

KeePassDroid – Implementation of KeePass for the Android

Tasker ($6.49) – What is there that this app can’t do?  I use it mostly for location based actions.  Walk into church and my phone goes to silent mode, if I have a meeting scheduled at work phone goes to vibrate during that meeting.

CSipSimple – The best SIP client for the Android.  It’s sort of like Skype in that you can make phone calls over the internet but unlike Skype, you’re not locked into any particular provider (like email).  It’s pretty useless since I’m the only one who has it.

GrooVe IP – Make VoIP to POTS through Google Voice/Chat for free.


Quality of Service

Since moving to North Idaho our internet is slow, we’re regularly saturating our .5Mbps upload cap doing cloud backups and uploading pictures and videos which can make gaming or web browsing painfully slow and unreliable.  I tried to get QoS (Quality of Service) working on DD-WRT but it can only have QoS rules based on port, not remote IP or amount of data transferred per connection.  This works for giving voice and games higher priority since you can differentiate that by port but it had no way of throttling down Crashplan backups or Flickr uploads on  port 80 or 443 without also throttling internet browsing on those ports.

Tomato has better QoS support but unfortunately my Buffalo router isn’t supported by Tomato, but my trusty old Linksys WRT-54GL is, so I flashed it with TomatoUSB (well, first I flashed the wrong firmware and bricked my router but some smart guy made a revival guide where you can get it back by shorting a pin).  The default QoS rules actually work pretty well:

Tomato QoS

I ended up adding more rules to fine tune it a little more and now it’s running perfectly.

So, here’s my new network setup:

Home Network Diagram


If you can’t handle email just send me a fax

On Sat, May 28, 2011 at 10:01 AM, Aetna Notification>>New Communication on Aetna Navigator®
You have a new communication to view on your secure Aetna Navigator® member website.  Click here to display your online communication.
(snip useless info)


On Tue, Jun 14, 2011 at 10:43 PM, Benjamin Bryan>>Please send the contents of your message directly to my email, I don’t want to take the trouble to login to your site to view the message.



On Tue, Jun 14, 2011 at 10:43 PM, Aetna Communications>>Thank you for contacting Aetna, unfortunately you have replied to an unmonitored mailbox.  Please refer to the Contact Us link in the original email with any questions and/or concerns.
(snip useless info)


Guess it wasn’t important.

Compare that with the email I received from Amica, a company that has humans!

On Tue, Jun 14, 2011 at 9:57 AM, (removed name and email)> wrote:
Hi Mr. and Mrs. Bryan,
We spoke a few weeks about your move to Idaho… I just wanted to follow-up with you to see if you have had the chance to register your cars in Idaho? Once this has been done, we can get you started with an Idaho policy and cancel your California policy. I hope the move was a smooth one!

(removed name & contact info) 

Amica Mutual Insurance Company | Oregon Regional Office

Amica knows how to handle email.  We’ve sent quite a few emails back and forth over the years with Amica and we always get responses like this.  They’ve never sent anything computer generated.

Making Free VoIP calls from your Android Phone

Rainy day, time to finish up this blog post.

Raindrops coming down… 

So Eli and I did a little exploring of VoIP solutions and found four to enable free or low cost calling on phones with unlimited data plans (or over wifi).  I tested using the LG Optimus V from VM and the Droid Charge from VZW.

The first two methods use SIP and the third uses Google Voice and Google Chat.  The fourth uses Microsoft Skype.  Of the four I decided Google Chat was the best if you just want to skip to solution 3. 

Quick note on Android SIP clients

CSipSimple’s Dialer Intercept

I tested two Android SIP clients, Sipdroid and CSipSimple I found the latter more reliable on receiving calls while the phone was on standby so I used it for all testing.  I think SipDroid is designed to work against PBXes but I didn’t want to introduce extra network latency by adding in another layer.  I initially had trouble with CSipSimple draining the battery but I downloaded one of the nightly builds instead of using the stable release and that solved the problem –Update: CSipSimple just released the battery drain fix on the Android Market- and I also doubled the registration timing so it wasn’t polling the network as often.

1 – CSipSimple using SipGate & Google Voice & Google Voice Callback (Free incoming and outgoing calls)

Pros: Completely Free.
Cons: Free service so who knows about how reliable this will be, higher voice latency than other solutions I tested, only allows you to connect two devices.  Received a few spam calls on my SipGate number so they might be recycling numbers too quickly.

1. Setup an account on, Sipgate allows for unlimited free incoming calls and gives you a free number.
2. Setup Google Voice, forward Google Voice to your Sipgate number.
3. CSipSimple.  For incoming calls once logged into Sipgate you can call your Google Voice number and it rings CSipSimple,
4. To make outgoing calls download a free Android app: Google Voice Callback.

Google Voice Callback installs an intercept into the dialer, so when you dial a number it intercepts the call, initiates the call through Google voice which then calls your SipGate number (because incoming calls are free) then once you pick up with CSipSimple it calls the 3rd party number you were dialing.

This method is completely free for incoming and outgoing calls with regular phones.  I did notice high latency using this method.  But free is free.
2 – VoIP.MS  ($0.99/month + $0.01/minute incoming & $0.0125/minute outgoing)

Pros: Very Good call quality.  
Cons: More expensive, can’t call other SIP numbers easily (but you could signup for a callcentric account for sip to sip calls), I get about 2 to 3 spam calls a week on my number so I wonder if they recycle numbers too quickly. is my favorite SIP provider because of the PBX functionality.

1. Setup an account with
2. Setup subaccounts for all the SIP devices that will connect (Macbook, Ben’s Android, Kris’ Android, iPhone).
3. Setup call groups (e.g. group “Ben” calls all of my devices, “Kris” calls all of Kris’ devices, etc).
4. Purchase incoming phone number at $.99/month.
5. Setup IVR (digital receptionist), you can program it in many awesome ways, including nested menus.  Instead of explaining it here’s a screenshot:

This is a full blown IVR, you could run a business off of it.

Anyway, on the client side you just want to setup CSipSimple to integrate with the Android dialer, then when you punch a number you can select whether you want to use your cell provider or (or any other providers) to make the call.

Each device also gets it’s own internal extension, so Kris can dial 101 to ring my Android, 102 to ring the Mac, etc.

Optionally you could forward your Google voice number to your number.
SIP Appendix A: Setup your own domain on SIP
This was harder to find but easy to setup.  I had to look at RFC 2782, basically the SIP servers or clients are supposed to lookup an SRV record the same way mail servers do an MX record lookup to see where to direct mail.

The end result is you want service sip on port 5060 udp pointing to your sip provider (in my case

Here’s what it looks like on a GoDaddy control panel:

Once that’s setup, any calls to resolve to whatever resolves.

3 – Google Voice & Google Talk & GrooVe IP ($3.99).  *Ben’s Pick*

Pros: Completely free except for a one time app purchase, unlimited calling in the U.S.

Cons: Does not work consistently with Google Apps accounts (Google Gmail accounts are fine, I migrated my Google Voice account to a normal Gmail account and that solved the issues for me), not an official Google client so it could break in the future.

Steve Baehr pointed out to me that Google Voice and Google Chat do pretty much everything I mentioned above, except it’s much simpler and free… the only problem is Google doesn’t have an Android client.  But someone wrote one: GrooVe IP ($3.99 in the Android Market) lets you make and receive calls through Google Chat.  

1. Setup your Google Voice account to forward calls to Google Chat.
2. You must be logged out of Gmail, otherwise your incoming calls will forward to your computer (I would suggest setting up a new Gmail account just for this to avoid this problem).
3. Install and configure GrooVe IP

Groove IP’s native dialer intercept

Incoming calls to your Google Voice number will forward to Google Chat which will forward to GrooVe IP on the Android.  Outgoing calls go through Google Chat to Google Voice to their destination.  GrooVe IP can also be set to intercept calls from the native dialer.

4 – Microsoft Skype ($10.49/month)
Pros: Can make calls using a normal cell network (not over IP) if on Verizon.  Most popular VoIP service for free Skype to Skype calls.
Cons: Most expensive.  Only Skype to Skype calls are free.  Otherwise you’re paying Skype rates, which are not too bad: currently $4.49/month for unlimited calling in the U.S., plus $18/month/3 ($18 every 3 months) for a Skype number if you want to receive calls from land-lines.

Verizon Wireless has a custom Skype application that allows you to make and receive calls using the VZW voice network, which is very useful if voice quality is our primary concern.

If you don’t have Verizon phone you have to make calls over Wifi and in my test calls with Kris the latency was pretty high.  Also if you’re a VM customer it’s only an extra $15 for their 1200 minute plan so the cost savings are not high enough to make it worthwhile.

How robust is VoIP on Android?
While I like the GrooVe IP solution I won’t be relying on it exclusively.  I called Meredith today and he said the audio quality was worse than on Verizon and we did notice a little latency so after awhile we switched over to Verizon.  On my end I could hear Meredith better on GrooVe IP than I could on Verizon, I think this is because Verizon suppresses the audio or some of the audio frequencies.  While we were talking on Verizon the call dropped so we can’t exactly say cell phones are that robust yet either.

Apple’s Stock Hitachi Vs. Kingston SSDNow V+

This is why your next computer upgrade should be a SSD.

I ran the following tests on Apple’s stock Hitachi 5400RPM 320GB:

1. Cold boot
2. Logos passage guide on 1 Kings 8.
3. Open IE from Windows 7 in Parallels “cold” boot.
4. StarCraft II load 4v4 map with 7 AIs.

I then replaced the hard drive with the Kingston SSDNow V+ 128GB, used Apple’s migration assistant to copy the data over to the SSD and repeated the tests.

All tests were performed on the 2011 13″ Macbook Pro Core i5 2.3GHz 4GB memory.

Note that Mac OSX does not yet have TRIM support.  If you do buy a SSD for your Mac you should get one with an onboard garbage collection controller (such as the one I bought). If you have Windows 7 you should be fine with just about anything.