TracFone on Verizon MVNO Review

We’ve been using Ting for Kris’s phone the last 3 years, it’s been great (see my Ting Review)–but at our new house Sprint’s signal isn’t that good–it can’t really pull in a 3G signal consistently.  In rural North Idaho the best coverage is indisputably Verizon, so I limited my search to Verizon MVNOs which have a BYOD (Bring Your Own Device) program and also only considered plans with access to LTE data.

SmartPhone plan < $10/Month on Verizon LTE

After doing some research and asking questions on HowardForums I finally settled on TracFone, which offers a great plan for light to moderate Android users.  Looking at Kris’s past usage over the last 2 years TracFone (at today’s pricing) would have averaged $7.82/month!  This is for a full SmartPhone service that includes voice, texts, MMS, and data on Verizon’s LTE network. If we use it more we pay more, if we use it less we pay less.

TracFone is very confusing actually–the idea is you buy an airtime card or refill plan (25+ to choose from!) which gives you one or a combination of voice minutes, texts, megabytes, and service days.  There are six types of cards you can buy–for the most part they provide the same service just with a different way and frequency of paying for them.

TracFone Airtime Cards


TracFone 400 minute card

  • 1-year cards.   Provides 365 days of service.  For SmartPhones the face value of the minutes triples, and also provides an equal amount of tripled megabytes and texts.  So a 400 minute 1-year card provides 1200 minutes AND 1200 texts AND 1200MB (Could this be any more confusing?)
  • Pay As You Go Cards.  Provides 90 days of service.  For SmartPhones the face value triples and you get an equal amount of minutes, megabytes, and texts just like the 1-year cards.
  • Auto Refill.  Provides 30 or 90 days of service.  For SmartPhones the face value triples and you get an equal amount of minutes, texts, and megabytes like the 1-year and Pay As You Go cards.  These can be set to auto-renew every 30 or 90 days.  I should note that they won’t auto-renew when you run out of something.  They just renew every x days.
  • SmartPhone / BYOD cards.  The face value does not triple on these cards.  Provides 90 days of service along with the minutes, texts, and megabytes printed on the card.  These cards are cheaper in terms of minutes and texts, but contain a little less data and only 90 service-days.
  • Monthly Value.  Same as auto refill but refills monthly instead of every 90 days…except for the 30 day auto-refill card.
  • Data only, Text only, and Service only cards.  Then there are individual bucket options–you can purchase data, texts, and also pay to extend service.  The only bucket you can’t purchase by itself is voice minutes.


Warning: I should also note, that some cards say “Double” but they are actually a worse deal.  As far as I can tell the 1-year 800 double minute card doesn’t double or triple on a SmartPhone but it costs more than the 400 minute card!  Also, bonus codes don’t work with SmartPhones–probably because tripling the value is already a great deal.  Also, if you use the monthly-value or auto-refill I’d suggest buying a buffer of at least 30 days airtime in case the refill or renewal fails.

All of the buckets on the cards: minutes, texts, and megabytes never expire.  They carry over and stack with each other.  So if you buy two 90 day cards your service end date will get pushed out 180 days. One great thing about TracFone is if you don’t use up a the units in a bucket you get to keep it–essentially let it rollover forever as long as you don’t let the service lapse.  You can build up your minutes, texts and data and keep them forever as long as you keep extending the service–which you can do by extending the service online for $50 to push the service end date out a year or by purchasing a card with service days on it.

I listed out the TracFone cards below and roughly calculated their value efficiency (far right column).  You can download my TracFone Pricing Spreadsheet (LibreOffice / OpenOffice format but it should open in Excel as well).  I make no guarantee to the accuracy of the spreadsheet.  I used the prices from TracFone or Ebay (whichever was cheaper).  Screenshot below:


Surprisingly the larger cards aren’t always the most efficient.  However, they /might/ be depending on your usage (see below).

I calculated the value based on the cheapest way one is able to obtain units in that bucket.  The cheapest way to acquire data is 4GB for $50, texts are 1000 for $10, Service is 365 days for $50.    It’s kind of tricky to value minutes, but the cheapest way of obtaining them is the 1-year 400 plan for $85 (if the card is bought off eBay) so after subtracting the value of the other buckets that leaves the value of a minute at $0.007.  This gives us:

Cost per Month: $4.17
Cost per MB = $0.0125
Cost per text = $0.01
Cost per minute = $0.007


On the Value tab of the spreadsheet I also added a multiplier.  The reason is some people may not ever use very much of a particular bucket so you can set that multiplier to zero to rank cards that favor that bucket lower on the Value Efficiency column.  Likewise you may at times find yourself with an excess amount in a particular bucket.  For example if your service-end date is 3 or 4 years into the future you don’t get much value purchasing service days so you can set the Service multiplier to 0 (or maybe 0.5 if you still want to give it some value) to lower the value efficiency of that bucket (which would make the SmartPhone plans or Pay As You Go more attractive than the 1-year plans).

This is not an exact science, but close enough for me to help decide on which card to purchase.

(also, I found a similar spreadsheet that Hands Henderson maintains on Google Sheets.  He will likely keep his more up to date.)

Cell Phone Taxes, Fees, and Surcharges

Cell Phone Taxes and Fees

One thing I should note is if you buy the TracFone Cards through TracFone’s website you’ll also pay taxes, fees, and surcharges.  If you’re in a high tax state You can save some money purchasing the airtime cards from other sellers such as Ebay (just make sure you use a reputable seller and complete the transaction through a safe method like PayPal).  Some sellers on eBay will email a pin number to you within an hour of purchase.

Okay… how is this better than a regular cell phone plan?

Because, a regular cell phone plan is a “Use it or Lose it” model.  You pay for a bucket of minutes, texts, and megabytes (sometimes for an unlimited bucket).  In practice you have to choose a bucket based on your maximum usage in a month to avoid a service cutoff or overages.  But anything you don’t use is lost.  Generally for heavy users you’re better off with an expensive monthly plan, but for light users pay for what you use makes perfect sense.

Analyzing the last 2-years of usage

Phone Usage Last 2 Years

One thing to note is the peak data usage was just over 1GB, and data usage comes close to the 500MB threshold regularly enough that the minimum monthly use it or lost it plan we could comfortably get away with is one that provides 1GB of data.

What-If  — Running Cost Analysis Over 2 Years With Various Providers

I looked at several wireless providers (also included Ting for Comparison).

  • Ting (Sprint MVNO).  Pay for the bucket you fall into.
  • TracFone (Verizon/AT&T MVNO) – Prepaid with rollover.
  • PagePlus (Verizon MVNO) – $29.95/month for 1GB data, 1500 minutes, unlimited texts.
  • Verizon Prepaid – $45/month for 1GB data, unlimited minutes/texts.
  • Project Fi (Sprint/T-Mobile MVNO)  – $20 + $0.01/MB.  Unlimited minutes/texts.  Fi would likely still have the same coverage issue at our house but it can also work on WiFi.

If I took the last ~2 years usage and put it on either of these plans at today’s pricing this is what the total running cost would be:

Total Cost Analysis Over 2 Years - Ting, TracFone, PagePlus, Verizon, Project Fi

I excluded government cell phone fees and taxes, depending on the state you live in this can be a significant portion of the cost.  Currently it is possible to purchase TracFone and PagePlus cards without having to pay taxes.  With Ting, Project Fi, and Verizon Prepaid you won’t have an option to avoid the tax. (Interesting to note how closely Ting and Project Fi are matched in price for one device.  For multiple phones Ting would pull ahead).

Also, I didn’t include the time-value of money.  On a pre-paid service like TracFone you’re probably pre-paying anywhere from a few months to a few years in advance depending on how much buffer you want.  However, the ROI is so quick (usually within 3-4 months) that the value lost pre-paying is trivial next to the cost savings.  There is also additional risk with pre-paying–for example, if you for some reason need to cancel service with TracFone you’ll lose any airtime that’s been purchased in advance.  Once again the ROI is fast enough I don’t think it’s an issue.

Monthly Price Variance

Price consistency almost never pays…


A light user that uses around 100MB, 100 minutes, and 100 texts / month will pay $85/year or $7.08/month on TracFone.  A moderate user averaging 600 minutes, texts, and MB will pay $290/year, or $24/month.  When you start averaging above $30-40/month on a consistent bases is probably the point it would make more sense to switch to a Use it or lose it model like PagePlus, Red Pocket, or Selectel Wireless.   It’s hard for me to see how people would use their phone that much though… 600 minutes is 10 hours on the phone every month!  I’m not sure my ear would be able to handle that much talking.  Certainly I can see using that much one or two months out of the year.  And that’s a great usage scenario with TracFone–it doesn’t matter if you use up $80 worth of service in a month–what matters is your average usage over long periods of time.  That’s where you will save over Use it or lose it plans.

Hotspot with TracFone

This seems to work fine.  The Nexus 5X will Hotspot without checking for a subscription on TracFone (unlike Verizon UDP plans which charge an additional $20/month hotspot fee).


TracFone partners with several networks, including Verizon Wireless and AT&T.  I opted to use Verizon’s network because of the better coverage in rural North Idaho.

TracFone Verizon BYOP Coverage Map

TracFone has a few limitations–it does not roam outside of Verizon’s network, and also there is no international roaming.  If you’re a frequent traveler outside the U.S. or in an area with poor Verizon coverage this wouldn’t work well.

Voice / Texting / MMS / Data / Shortcode messaging.

Voice, Texting, MMS, and Data all seem to work great.  There is a feature called short-code messaging that TracFone does not support.  I guess if you watch ridiculous shows on TV you can text to a number to vote or something of that nature.  If you like to do that this isn’t for you.

Google Voice Integration  / Conditional Call Forwarding

I don’t like normal voicemail, I prefer to have VM transcribed  so I use Google Voice for this.  Verizon’s Call Forwarding codes worked fine.  *71yourgooglevoicenumber sets your phone up to forward to Google voice if you’re on the phone, reject a call, or don’t answer.  This works as expected.

Website.  The horror

The TracFone website is really not that great.  It’s slow, you can’t do basic things like port a number in.  And often when you try to do something it errors out.


I had to engage support to port Kris’s number to TracFone since their website couldn’t do it.

They have a chat support on their Facebook page (fortunately doesn’t require a Facebook login) or an 800 number.  800-367-7183.  The chat often was down but I was always able to get help on the phone.  One thing I do like about support is if they need to escalate to another person the original person stays on the line with you (at least that was the case for me).

I found support pretty helpful but not as informative about the processes as they could have been.  However, support was polite and always got things moving in the right direction.  One thing support didn’t inform me of is when porting a number in to replace an existing TracFone number the airtime gets wiped out (I was suspicious that this may happen so I bought a very cheap airtime card to test with).  After the port completed successfully the phone wasn’t working and support told me we needed to wait up to 48 hours—however it wasn’t working because of lack of airtime.  My assessment of TracFone support is that if you need hand-holding or will get upset over a glitch this isn’t the best service.  If you have patience you should be okay.

BYOD (Bring Your Own Device)

TracFone has a great BYOP program.  I activated with a Nexus 5X and it was simply a matter of activating with the instructions in the SIM card kit.  It was up and running in minutes.  The only reason I needed to involve support was I wanted to transfer Kris’s old number in.

ESN Number Issues with Carlos Slim

Carlos Slim owns a lot of pre-paid wireless MVNOs.  One issue is all of them share the same database–so if you activate an ESN number on TracFone it will be impossible to move it over to some of the other Carlos Slim owned MVNO provider (if you should ever want to move in the future) because it gets locked to that MVNO in their system.  However, because this is 4G LTE, the Sim Card will be used to activate the correct ESN on Verizon’s network, so if you have an old or broken Verizon Wireless phone you can provide TracFone that ESN instead of the ESN for your SmartPhone.

Getting Started

tracfone_sim_card_kitYou’ll want to buy or have a Verizon phone that will work.  I bought the Nexus 5X (read my review of it here).

Second, pick up a TracFone 4G LTE CDMA Activation kit (the 4G LTE without CDMA activation kit is for AT&T’s network).  You can buy it from TracFone or you may find it in a store like WalMart.  I followed the instructions that came with the activation kit.



Making sure you don’t run out

You don’t want to run out of a bucket or that particular service will stop working.  I really don’t want to babysit this, and I want plenty of units available in case they’re needed.  So to get started I purchased a 400 minute 1-year card off Ebay.   I setup a reminder to check the TracFone balance once a quarter (every 3 months) and if it falls below a certain threshold (say 800 minutes, 800 texts, and 1000MB) I’ll purchase the appropriate Airtime card to refill the balance and extend out the service-date.  I don’t mind pre-paying years in advance–compared to a $30/month plan (which is really the next best price on a Verizon MVNO).  Even paying a year in advance still has an ROI that beats the monthly use it or lose it plans at 3 months.

Final Thoughts

Beware of little expenses. A small leak will sink a great ship. – Benjamin Franklin

TracFone is a very affordable service for light and even moderate SmartPhone users who have the patience to figure things out.  Also, it’s easy to make mistakes on TracFone (like buying the 800 minute card that doesn’t triple)–if you’re the type to get angry over losing a few dollars over glitches and stupid policies this may not be the plan for you.  TracFone users will also need the  discipline to check on the  account every quarter or so to make sure they have plenty of airtime and service left before reaching the expiration.  If you can deal with TracFone the cost savings are well worth it.   Consider a $60/month plan, a $30/month plan, and TracFone at $10/month.  Look at it over a 2-year period.  The first plan will run $1,440, the second $720, but with TracFone only $240!  It’s hard to beat that pricing!

With T-Mobile shaking up the wireless industry, Google Fi introducing a service that spans across multiple networks, and Ting offering great auto-adjusting bucket plan the wireless industry is finally starting to get competitive.  For the consumers this is nothing but great news and I hope that we’ll see more improvement to infrastructure and pricing in 2016.


Nexus 5X Review

The stock ROMS on Verizon/Samsung phones are awful.  The phones come shipped with Samsung Touchwiz which makes the UI ugly and a half a dozen apps that you can’t uninstall–some eating up battery, some sending telemetry data to Samsung (using up your data), some of the changes Samsung does introduces vulnerabilities not to mention it takes Samsung and Verizon so long to push out updates after Android exploits are announced.  Then Verizon locks out the hotspot feature–which I rarely use but when you need that feature you need it.  But on the Samsung S3 and S4 I got around all of those problems by flashing Cyanogenmod (now owned by Microsoft).   When I upgraded to the Samsung S6 I thought I’d do the same thing but the bootloader is now locked out so that I can’t install a custom ROM.  That’s ridiculous.  I’m done with Verizon and Samsung phones.  It would be like purchasing a computer with a BIOS that locked out all operating systems except for Windows 8.  Oh, and it will only work with AOL.  If you move and want to use Comcast you’ll need to buy a new computer.

So I ordered the Nexus 5X which came yesterday.

Photo of Nexus 5X

It looks sort of like a communication device. Here it is in a minimal case.

The Nexus 5X is only available at the Google Store, and priced at $379.  This is actually not a bad price considering the phone is not subsidized and it will work on pretty much any carrier.


The most important feature of any Android phone is the flashlight.  I am happy to report that the Nexus 5X makes an excellent flashlight.


The USB cord is the new type-C connector.  Type-C is a huge improvement.  Both ends are the same, and it can be inserted upside down or right-side up.  This is a nice change, I can’t tell you how many times I try to plug in a USB cable at night in the dark and it’s always upside-down.  Every time.  Then I’ll rotate it and it’s somehow still upside down!  Not the case with Type-C.  Just plug it in.  The only way it could be better is if they made it circular.  Maybe that will come with USB-D.  The USB cable that came with the Nexus 5X is too short, looks to be about 3 feet.  It can’t reach my end table from the outlet.  If you plan on connecting your computer you’ll need a new computer with a USB-C connector or a USB-C to USB-A cable.

The Nexus 5X is missing is wireless charging capability–I never used that feature but it probably should have been in the phone.

Size Comparison

Side by side photo of S4, S6, and Nexus 5X

Left to right: Samsung S4, Samsung S6, and Nexus 5X all shown with minimal cases.

I prefer smaller single-handed phones, so the Nexus 5X is on the larger size of what I’ll tolerate, I’d probably prefer slightly smaller but when you’re main criteria is an unadulterated Android phone the only options are large (5X), and super-large (6, 6P).  For practical purposes the Nexus 5X doesn’t feel any bigger than the S6 even though it’s slightly larger.

Phone Cases, some more evil than others

I’m not a big fan of phone cases since they add to a phone’s bulk, but the Samsung S6 is extremely slick with the glass material so it easily slips off of any surface with the slightest bump and has tried to commit suicide a few times by vibrating itself off the end-table so I’ve started using thin cases.  When I searched Amazon for Nexus 5X cases, before the phone had even come out I noticed a 5-star rated Spigen case before the 5X phone had even be released.  How could reviewers possibly know how well the case is without the phone?  Apparently Spigen paid a bunch of people or gave them free cases in exchange for reviews.  Ironically, now that the Nexus 5X has started shipping and people are trying on their cases the Spigen case actually blocks the the Nexus 5X microphone!  Oops.

I, fortunately having at least an ounce of discernment saw through the Spigen case scam and ordered a black Ringke case.  The Ringke adds almost no bulk and fits the phone well and is a matte finish so it isn’t slippery.  I don’t use the screen protector that came with it, I’ve found they cut back on the screen responsiveness and I’m pretty careful and so far have never broken a screen.

Phone Material and Build Quality.

The build quality and material are fantastic.  I’ve read a few reviews saying the phone doesn’t feel as premium as an iPhone or Samsung S6 or Nexus 6P.  But does anyone really care to have an exotic glass or metal back on a phone?  I could care less whether my phone is made out of metal, glass, plastic or wood.  All I want in the build quality is the phone to be comfortable to hold and survive without breaking for 3 years.  I think the plastic material not only feels better but is a lot more likely than other materials to absorb damage from an accidental drop.  Also, the Nexus 5X is a lot lighter compared to the S6 because of the plastic build which to me is a nice plus.

Kris’s white Nexus 5X is also the same material, a nice non-slip plastic/rubber surface.

White Nexus 5X

Battery Life

This morning I woke up around 6am and used the phone pretty heavily all day.  The battery was at 18% by 9pm.  This is an improvement over the Samsung S6 which is usually down to 5% by then… although I’m not sure if it’s more of a factor of the Verizon/Samsung battery draining apps on the S6 or Lollipop vs. Marshmallow.


A big difference between the Samsung S6 and the Nexus 5X is the camera sensor–the S6 has a 16MP 16:9 aspect ratio sensor while the Nexus 5X uses the traditional 4:3 aspect ratio.  Both cameras can shoot in their non-native aspect ratios but will lose megapixels doing so (around 11MP for the S6 and 8MP for the 5X).

In low light conditions both did well, here is a shot of my messy bookshelf in full auto on both camera.  The 5X decided to HDR the shot and the S6 did not so I took one more with the 5X with HDR off.  In my opinion the images on the S6 are a little more detailed and less grainy than the 5X.  The Nexus 5X seems to have a slightly narrower depth of focus so for artistic shots (not that I have any photography skills) it’s slightly easier to get the depth of field that I want.

You’ll notice the 5X has a wider angle so it almost covers the same width as the S6 but crops less at the top and bottom–so the Nexus 5X will be better at taking a photo of everyone in a short room, the S6 will be zoomed in slightly more making it a little better for distant shots but it still manages to grab a little more data width-wise because of the 16:9 aspect ratio.  For practical purposes the S6 is probably better in this regard.

Here’s a few pictures I took with the 5X today…

One feature I don’t think the S6 has, is the 5X will take three shots for every photo you take, and automatically pick the best one…e.g. where eyes aren’t blinking.  I think this feature is more valuable than burst mode because now I don’t have to do the work of finding the best photo.  Let the computer choose the correct photo–that’s a great job to automate!  I find the colors tend to be a little better for most pictures on the Nexus 5X but sometimes the S6 does a better job depending on the shot–the 5X seems to have washed out background colors on some shots, usually the S6 gets it better but sometimes the 5X does a better job.

The camera on the Nexus 5X is a great, but the one on the Samsung S6 is better.  The images are slightly sharper and less grainy in low light, it focuses faster, it can do rapid continuous shooting (the 5X on the other hand can’t shoot rapidly at all), and more megapixels.  The default camera app for Samsung also allows more control of the camera if you want to adjust ISO, metering, etc.  I also found that the Nexus 5X took a little longer to focus than the Samsung S6.  Not to say the Nexus 5X camera is bad, it’s great.  It’s better than any point and shoot I’ve ever used and in some ways it beats my DSLR.  And you probably wouldn’t even notice except by comparing the photos side-by-side.  But if you’re buying a phone for the camera the S6 will be the better choice.

Pure Android OS — but what about the hotspot?!

The pure Android OS is fantastic.  One feature I like on Android is multi-user support–great for people that have kids.  I’m not sure why Samsung strips this stuff out in TouchWiz.


A pure Android OS is missing a few things that CyanogenMod adds–like the ability to automatically red-shift the screen when the sun sets so that using the phone in the evening doesn’t emit as much light in the blue spectrum which suppresses  melatonin delaying your ability to fall asleep.  There are Android apps that can redshift, but it’s nicer and more secure to have the OS do this.

The one thing I am disappointed in is that when enabling Mobile Hotspot (on Verizon at least) the phone checks to see if it has a hotspot subscription–this is very annoying because part of the FCC requirements on the 700MHz spectrum is that Verizon can’t deny, limit, or restrict the ability of customers to use devices or applications of their choice–including a hotspot.  This is one of the main reasons I wanted a pure Android OS.  Yes, I know there are work-a-rounds but I shouldn’t have to go to that trouble.

The way this is limited is probably a violation of Isaac Asimov’s 2nd law of robotics.  The owner of the phone should have control over this–if networks want to restrict something like this they need to restrict it on the network side (which they can’t because of the FCC requirements) but not on a device that’s not owned by them.

Hotspot crippling only on Verizon and certain carriers?

On Kris’s Nexus 5X which is activated on the Verizon network via TracFone does not check for a hotspot subscription–so the crippling may be specific to certain carriers.


The screen is great, I had no trouble seeing it in the sun outside, the pixel density isn’t as high as some phones but I can’t tell the difference.  I can’t see the individual pixels on the 5X–looks pretty smooth to me.

Carrier Portability

Probably one of the greatest features of this phone is it can be used on any carrier, Verizon, AT&T, T-Mobile, Sprint, and probably most carriers outside the United States as well.  Just pop in a SIM card and go.


NFC antenna is much better than the S6.  It’s located near the top instead of an arbitrary spot in the middle of the phone.  I have no trouble reading and writing data via NFC where the S6 struggled to even make a connection.

Fingerprint Scanner

I never used the thumbprint scanner on the S6 because I wasn’t sure if it would get uploaded to Samsung’s servers.  Google doesn’t upload your fingerprints to the cloud so it stays secure on the device.

The Nexus 5X has a scanner on the back of the phone, you can scan your finger in at a few different angles, it instantly unlocks the phone–you don’t even have to push the unlock button first, just pull the phone out of your pocket and get your finger on the scanner and it’s unlocked by the time you bring it up.

I do not believe that fingerprints are a secure means of securing a device since they can be easily forged, however it is extremely convenient so users will have to make their own assessment on whether they’ll use the feature or not.

Ben’s Rating

4 out of 5 stars.  Knocking a star for the crippled hotspot feature but otherwise a great phone and excellent value for the price.

The Side Affects of Smart Phones

I’m not much of a phone fanatic, but smart phones are very useful tools–I use them for just about everything other than making calls (actually I do make a call on occasion to order sushi).  In addition to being a fantastic flashlight Android phones are great tools to search and lookup information, the camera is better than any point and shoot I’ve used and it’s always with me unlike my DSLR, GPS navigation, hunting maps, star gazing, I’ve read a few books using the Kindle app, and on occasion I’ll be trying to figure out how to do something very simple that I’m completely incompetent at and I can find a YouTube video that tells me how to do it all without having to leave the task at hand.

I think one of the dangers from modern smart phones is they are way too distracting–I can’t tell you how many times I’ve seen people sit down for a meal and pull out their phones instead of having a conversation.  Unfortunately I can’t say I’m not guilty of this myself.  I’ve notice that people who do this tend to keep the conversation on topics that can be quickly discussed–you will never have a deep conversation with other people if their phones are out.  The mere sight of a phone sort of subconsciously tells everyone to keep to trivial topics.

With every new innovation society changes, we lose some things and gain some things–Socrates didn’t like books because they lost the dialogue which he believed to be essential to knowledge.  So while our culture may lose some things with cell phones, the benefits are also great.  While phones are incredibly useful, we need to make sure they don’t rule our lives.  I almost always leave my phone on because when there are emergencies and urgent issues I want to be available.  But I keep all the notifications off (especially email) except for calendar reminders, texting, and phone calls.  I tell people to send me an email if they can wait a day for a response,  Send me a text or call if an issue is so urgent that it’s worth interrupting me.  Some people have a hard time doing that, but it’s easy (in pure Android) to disable SMS notifications (and even silence the ringtone) when calls are from certain people.


pfSense Firewall HA Failover Cluster

Last night, I was attending a LAN party remotely and between games I noticed my pfSense router needed to be updated but of course an update brings down my internet for 30 seconds while it reboots which I didn’t want to do–and then I thought, I should really cluster this.  I found a straight-forward pfSense HA (Hardware/Device Failover) Configuration using CARP guide written by Michael Holloway and after following it ended up with something like this:

pfSense HA Diagram

I run pfSense under VMware (which I do not recommend unless you know what you are doing–if you do know what you are doing be sure to enable promiscuous mode on the VM switches  [which if you didn’t know perhaps you shouldn’t do this–you can end up getting into a circular dependency situation so just be sure you have an alternate way to get into VMware to troubleshoot pfSense in case it dies for some reason–there are several ways to do this: you can setup a backup VMkernel port with management enabled on a vSwitch connected to another physical adapter, or set VMware’s management interface to a static IP and set workstation to another static IP on that subnet.]) so networking hardware is free, so I deployed a second pfSense VM more or less identical to the master–I don’t run pfSense on my ZFS storage because I want my networking to come up before storage, but I did put each pfSense server on a separate hard drive.

Essentially I setup a sync interface ( and .2)  where the pfsense-master syncs everything in real-time to the pfsense-slave.  My “WAN Gateway” is a CradlePoint router.  The pfsense-master WAN IP is, and the pfsense-slave WAN IP is, and then I setup a WAN-CARP virtual IP of which is where all the WAN traffic goes out on, the master assumes the WAN-CARP.  If the master goes down then the slave will take it over and the CradlePoint router is none-the-wiser.  Pretty much the same thing for the LAN and DMZ, the CARP virtual IP is and respectively, if the master goes down the slave assumes the IPs.

pfSense actually syncs the connection states.  I established an ssh connection to a remote server, hard powered off the master and didn’t lose the connection!  I was also pinging a remote host and didn’t drop a single packet.  This is good, now I can upgrade each pfsense router independently with no downtime.  If the pfsense master goes down, or somehow gets disconnected from the WAN or LAN the pfsense slave will assume the virtual IPs.  I’ve tested powering off the master, disconnecting the WAN port, disconnecting the LAN port, etc.  As long as the> link stays up the salve will assume the role of the master during those scenarios, and as soon as the real master recovers it re-assumes the role of master.



Here’s my current battlestation…  since the office and guest bedroom are shared I recently migrated from a desktop to a Dell Latitude E5450 with a docking station so I could use the room as an office most of the time but undock when guests are staying the night.

Starting with the Lackracks

  • 2U server running VMware and FreeNAS
  • Samsung ML-2850ND Laserjet printer
  • On the top table networking stuff behind some videos
  • A Tivoli Model One table-top radio which I use for my computer speaker (stereo gives me a headache so it’s a mono-speaker)
  • Dell Docking station
  • Dell Latitude E5450 with Nvidia GeForce 840M.  I looked at a few options and came to this model because I wanted NBD support, a robust docking solution, and a GPU for the occasional game.  Getting upgrades from Dell is pricey so I bought from Amazon another 8GB memory module to bring it up to 16GB and a Samsung 850 EVO 1TB SSD.
  • Portable blue-ray drive so we can watch movies
  • A couple of ASUS monitors
  • Orochi Razer mouse.  This is the only mouse I’ve gotten to work well since it can reverse the buttons in the mouse firmware instead of relying on the OS which doesn’t always produce consistent results–you left handers know what I mean.
  • Microsoft Sidewinder X4 keyboard
  • On the far right a ScanSnap S1300–About 6 or 7 years ago I got an IRS audit with an accusation that I owed much more.  I am honest on my taxes but they don’t take a Quicken report as evidence.  So I had to prove it by digging through boxes of papers to find all the supporting documents on my return.  It was then that I decided to go paperless.  I now scan and OCR every document that comes across my desk per What Happens When You Send Me A Letter   We are now entirely paperless except for one small desk drawer where I file a few important papers.
  • 20150826_222938Speaking of important papers, in front of the laptop is a stamp I use to mark documents as scanned when I want to keep the physical copy–every once in awhile I run across a document I don’t want to shred such as my vehicle title.  The stamp prevents me from accidentally scanning the same papers multiple times if I’m not sure that I’ve scanned it in yet.


VMware vs bhyve Performance Comparison

Playing with bhyve

Here’s a look at Gea’s popular All-in-one design which allows VMware to run on top of ZFS on a single box using a virtual 10Gbe storage network.  The design requires an HBA, and a CPU that supports VT-d so that the storage can be passed directly to a guest VM running a ZFS server (such as OmniOS or FreeNAS).  Then a virtual storage network is used to share the storage back to VMware.


VMware and ZFS: All-In-One Design

bhyve, can simplify this design since it runs under FreeBSD it already has a ZFS server.  This not only simplifies the design, but it could potentially allow a hypervisor to run on simpler less expensive hardware.  The same design in bhyve eliminates the need to use a dedicated HBA and a CPU that supports VT-d.


Simpler bhyve design

I’ve never understood the advantage of type-1 hypervisors (such as VMware and Xen) over Type-2 hypervisors (like KVM and bhyve).  Type-1 proponents say the hypervisor runs on bare metal instead of an OS… I’m not sure how VMware isn’t considered an OS except that it is a purpose-built OS and probably smaller.  It seems you could take a Linux distribution running KVM and take away features until at some point it becomes a Type-1 hypervisor.  Which is all fine but it could actually be a disadvantage if you wanted some of those features (like ZFS).  A type-2 hypervisor that supports ZFS appears to have a clear advantage (at least theoretically) over a type-1 for this type of setup.

In fact, FreeBSD may be the best visualization / storage platform.  You get ZFS and bhyve, and also jails.  You really only need to run bhyve when virtualizing a different OS.

bhyve is still pretty young, but I thought I’d run some tests to see where it’s at…


This is running on my X10SDV-F Datacenter in a Box Build.

In all environments the following parameters were used:

  • Supermico X10SDV-F
  • Xeon D-1540
  • 32GB ECC DDR4 memory
  • IBM ServerRaid M1015 flashed to IT mode.
  • 4 x HGST Ultrastar 7K300 HGST 2TB enterprise drives in RAID-Z
  • One DC S3700 100GB over-provisioned to 8GB used as the log device.
  • No L2ARC.
  • Compression = LZ4
  • Sync = standard (unless specified).
  • Guest (where tests are run): Ubuntu 14.04 LTS, 16GB, 4 cores, 1GB memory.
  • OS defaults are left as is, I didn’t try to tweak number of NFS servers, sd.conf, etc.
  • My tests fit inside of ARC.  I ran each test 5 times on each platform to warm up the ARC.  The results are the average of the next 5 test runs.
  • I only tested an Ubuntu guest because it’s the only distribution I run in (in quantity anyway) addition to FreeBSD, I suppose a more thorough test should include other operating systems.

The environments were setup as follows:

1 – VM under ESXi 6 using NFS storage from FreeNAS 9.3 VM via VT-d

  • FreeNAS 9.3 installed under ESXi.
  • FreeNAS is given 24GB memory.
  • HBA is passed to it via VT-d.
  • Storage shared to VMware via NFSv3, virtual storage network on VMXNET3.
  • Ubuntu guest given VMware para-virtual drivers

2 – VM under ESXi 6 using NFS storage from OmniOS VM via VT-d

  • OmniOS r151014 LTS installed under ESXi.
  • OmniOS is given 24GB memory.
  • HBA is passed to it via VT-d.
  • Storage shared to VMware via NFSv3, virtual storage network on VMXNET3.
  • Ubuntu guest given VMware para-virtual drivers

3 – VM under FreeBSD bhyve

  • bhyve running on FreeBSD 10.1-Release
  • Guest storage is file image on ZFS dataset.

4 – VM under FreeBSD bhyve sync always

  • bhyve running on FreeBSD 10.1-Release
  • Guest storage is file image on ZFS dataset.
  • Sync=always

Benchmark Results

MariaDB OLTP Load

This test is a mix of CPU and storage I/O.  bhyve (yellow) pulls ahead in the 2 threaded test, probably because it doesn’t have to issue a sync after each write.  However, it falls behind on the 4 threaded test even with that advantage, probably because it isn’t as efficient at handling CPU processing as VMware (see next chart on finding primes).

Finding Primes

Finding prime numbers with a VM under VMware is significantly faster than under bhyve.


Random Read

byhve has an advantage, probably because it has direct access to ZFS.


Random Write

With sync=standard bhyve has a clear advantage.  I’m not sure why VMware can outperform bhyve sync=always.  I am merely speculating but I wonder if VMware over NFS is translating smaller writes into larger blocks (maybe 64k or 128k) before sending them to the NFS server.


Random Read/Write


Sequential Read

Sequential reads are faster with bhyve’s direct storage access.


Sequential Write

What not having to sync every write will gain you..


Sequential Rewrite




VMware is a very fine virtualization platform that’s been well tuned.  All that overhead of VT-d, virtual 10gbe switches for the storage network, VM storage over NFS, etc. are not hurting it’s performance except perhaps on sequential reads.

For as young as bhyve is I’m happy with the performance compared to VMware, it appears to be a slower on the CPU intensive tests.   I didn’t intend on comparing CPU performance so I haven’t done enough variety of tests to see what the difference is there but it appears VMware has an advantage.

One thing that is not clear to me is how safe running sync=standard is on bhyve.  The ideal scenario would be honoring fsync requests from the guest, however I’m not sure if bhyve has that kind of insight from the guest.  Probably the worst case under this scenario with sync=standard is losing the last 5 seconds of writes–but even that risk can be mitigated with battery backup. With standard sync there’s a lot of performance to be gained over VMware with NFS.  Even if you run bhyve with sync=always it does not perform badly, and even outperforms VMware All-in-one design on some tests.

The upcoming FreeNAS 10 may be an interesting hypervisor + storage platform, especially if it provides a GUI to manage bhyve.


Supermicro X10SDV-F Build; Datacenter in a Box

cs721_frontI don’t have room for a couple of rackmount servers anymore so I was thinking of ways to reduce the footprint and noise from my servers.  I’ve been very happy with Supermicro hardware so here’s my Supermicro Mini-ITX Datacenter in a box build:

Supermicro X10SDV-F Motherboard

This board looks like it’s designed for high density computing.  It is the most computing power I have ever seen crammed into a Mini-ITX board.  A Xeon D-1540 which will well outperform the Xeon E3-1230v3 in most tests, can handle up to 128GB memory, two nics (this also comes in a model that offers two more 10Gbe), IPMI, 6 SATA-3 ports, a PCI-E slot, and an M.2 slot.

Supermicro X10SDV-F Motherboard

Supermicro X10SDV-F

Cooling issue | heatsink not enough

The first boot was fine but it crashed after about 5 minutes while I was in the BIOS setup…. after a few resets I couldn’t even get it to post.  I finally realized the CPU was getting too hot.  Supermicro probably meant for this model to be in a 1U case with good air flow.  The X10SDV-TLN4F is a little extra but it comes with a CPU fan in addition to the 10Gbe network adapters so keep that in mind if you’re trying to decide between the two boards.

Noctua to the Rescue

I couldn’t find a CPU fan designed to fit this particular socket, so I bought a 60MM Noctua.

60MM Noctua FAN on X10SDV-F

60MM Noctua Fan

This is my first Noctua fan and I think it’s the nicest fan I’ve ever owned.  It came packaged with screws, rubber leg things, an extension cord, a molex power adapter, and two noise reducer cables that slow the fan down a bit.  I actually can’t even hear the fan running at normal speed.

Noctua Fan on Xeon D-1540 X10SDV-F

Notcua Fan on Xeon D-1540

There’s not really a good way to screw the fan and the heatsink into the motherboard together, but I took the four rubber things and sort of tucked them under the heatsink screws.  This is  surprisingly a secure fit, it’s not ideal but the fan is not going to go anywhere.

Supermicro CSE-721TQ-250B

This is what you would expect from Supermicro, a quality server-grade case.  It comes with a 250 watt 80 plus power supply.  Four 3.5″ hotswap bays, trays are the same as you would find on a 16 bay enterprise chassis.  Also has labels numbered from 0 to 4 so you could choose to label starting at 0 (the right way) or 1.  It is designed to fit two fixed 2.5″ drives, one on the side of the HDD cage, and the other can be used on top instead of an optical drive.

The case is roomy enough to work with, I had no trouble adding an IBM ServerRAID M1015 / LSI  9220-8i



I took this shot just to note that if you could figure out a way to secure an extra drive, there is room to fit three drives, or perhaps two drives even with an optical drive, you’d have to use a Y-splitter to power it.  I should also note that you could use the M.2. slot to add another SSD.


The case is pretty quiet, I cannot hear it at all with my other computers running in the same room so I’m not sure how much noise it makes.

This case reminds me of the HP Microserver Gen8 and is probably about the same size and quality but I think a little more roomier and with Supermicro IPMI is free.

Compared to the DS380 the Supermicro CS721 is a more compact.  The DS380 has the advantage of being able to hold more drives.  The DS380 can fit 8 3.5″ or 2.5″ in hotswap bays plus an additional four 2.5″ fixed in a cage.  Between the two cases I much prefer the Supermicro CS-721 even with less drive capacity.  The DS380 has vibration issues with all the drives populated, and it’s also not as easy to work with.  The CS-721 looks and feels much higher quality.


Storage Capacity

cs721_open_door6TB drives are about the largest you’d want to use, with four of those in RAID-Z (RAID-5) the case can provide up to 18TB of storage which is a good amount for any data hoarder wanting to get started.

Other Thoughts

What I would love to see from case manufacturers is a NAS case between the CS-721 (mini-ITX) and CS-743 (E-ATX).  The market could use a good mATX sized hotswap NAS case.

I’m pretty happy with the build and despite it’s small size it can house a lot of computational power.  8 cores (16 threads) and up to 128GB memory, and even up to 18TB of data.  With VMware and ZFS you could run a small datacenter from a box under your desk.

I should be posting some ZFS benchmarks on this build shortly.

FreeNAS 9.3 on VMware ESXi 6.0 Guide

This is a guide which will install FreeNAS 9.3 under VMware ESXi and then using ZFS share the storage back to VMware.  This is roughly based on Napp-It’s All-In-One design, except that it uses FreeNAS instead of OminOS.


Disclaimer:  I should note that FreeNAS does not officially support running virtualized in production environments.  If you run into any problems and ask for help on the FreeNAS forums, I have no doubt that Cyberjock will respond with “So, you want to lose all your data?”  So, with that disclaimer aside let’s get going:

Update: Josh Paetzel wrote a post on Virtualizing FreeNAS so this is somewhat “official” now.  I would still exercise caution.

1. Get proper hardware

SuperMicro X10SL7-F (which has a built in LSI2308).
Xeon E3-1240v3
ECC Memory

Hard drives.  The LSI2308 has 8 ports, I like do to two DC S3700s for a striped SLOG device and then do a RAID-Z2 of spinners on the other 6 slots.  Also get one (preferably two for a mirror) drives that you will plug into the SATA ports (not on the LSI controller) for the local ESXi data store.  I’m using DC S3700s because that’s what I have, but this doesn’t need to be fast storage, it’s just to put FreeNAS on.

2.  Flash the LSI 2308 to IT firmware.

Here’s instructions to flash the firmware:  FreeNAS 9.3 wants P16, which you can get from Supermicro:

UPDATE on Firmware: As of FreeNAS 9.3.1 the LSI drivers have been upgraded to v20 and now FreeNAS recommends the P20 firmware.

However, P20 is widely known to have bugs and performance issues, especially with fast SSDs  I am surprised by the P20 move.  I think it’s too aggressive to be moving to P20 firmware.  On storage systems I like to see firmware in use for more than a year before upgrading to it.  I’m surprised FreeNAS didn’t go with P19.  It has been widely used and very stable in the OmniOS/Napp-It community, and several people running FreeBSD 10 found that P19 was stable after first trying P20.

I did a little research and found a few references to P20 having an initially buggy release, but the issues may have been fixed in P20.00.04.00 which was released in May 2015.  I don’t know of anyone that has verified the issues were fixed.  As of today (2015-09-21) Supermicro has only released up to P20.00.02.00 for my LSI2308 so I presume that’s the buggy version that shouldn’t be used.

So the options are to run P20 which is too new to put in production, run P16 or P19 with a mismatched v20 driver, downgrade the driver to v19, or don’t upgrade to 9.3.1.  I really can’t recommend any of those options, they’re all bad.  So I suggest having good backups.

If you decide to run P16 or P19 on FreeNAS 9.3.1 which is what I’m doing (not that I am recommending this, I just think it’s the least bad option), this will result in a firmware driver mismatch amber warning light in FreeNAS, however you can un-check that alert to mute the warning.

Amber warning for firmware mismatch

3. Optional: Over-provision ZIL / SLOG SSDs.

If you’re going to use an SSD for SLOG you can over-provision them.  You can boot into an Ubuntu LiveCD and use hdparm, instructions are here:  You can also do this after after VMware is installed by passing the LSI controller to an Ubuntu VM (FreeNAS doesn’t have hdparm).  I usually over-provision down to 8GB.

4. Install VMware ESXi 6

ImageThe free version of the hypervisor is here. I usually install it to a USB drive plugged into the motherboard’s internal header.

Under configuration, storage, click add storage.  Choose one (or two) of the local storage disks plugged into your SATA ports (do not add a disk on your LSI controller).

5. Create a Virtual Storage Network.

For this example my VMware management IP is, the VMware Storage Network ip is, and the FreeNAS Storage Network IP is

Create a virtual storage network with jumbo frames enabled.

VMware, Configuration, Add Networking. Virtual Machine…

Create a standard switch (uncheck any physical adapters).

Image [8]

Image [11]


Add Networking again, VMKernel, VMKernel…  Select vSwitch1 (which you just created in the previous step), give it a network different than your main network.  I use for my storage so you’d put for the IP and for the netmask.

Image [12]

Some people are having trouble with an MTU of 9000.  I suggest leaving the MTU at 1500 and make sure everything works there before testing an MTU of 9000.  Also, if you run into networking issues look at disabling TSO offloading (see comments).

Under vSwitch1 go to Properties, select vSwitch, Edit, change the MTU to 9000.  Answer yes to the no active NICs warning.

Image [14]

Image [15]

Then select the Storage Kernel port, edit, and set the MTU to 9000.

Image [17]

Image [18]

6. Configure the LSI 2308 for Passthrough (VT-d).

Configuration, Advanced Settings, Configure Passthrough.

Image [19]

Mark the LSI2308 controller for passthrough.

Image [20]

You must have VT-d enabled in the BIOS for this to work so if it won’t let you for some reason check your BIOS settings.

Reboot VMware.

7. Create the FreeNAS VM.

Download the FreeNAS ISO from

Create a new VM, choose custom, put it on one of the drives on the SATA ports, Virtual Machine version 11, Guest OS type is FreeBSD 64-bit, 1 socket and 2 cores.  Try to give it at least 8GB of memory.  On Networking give it three adapters, the 1st NIC should be assigned to the VM Network, 2nd NIC to the Storage network.  Set both to VMXNET3.  Then add a 3rd NIC and set it to E1000 and on the VM Network which we’ll use temporarily until the VMXNET3 drivers are up and running.


SCSI controller should be the default, LSI Logic Parallel.

Choose Edit the Virtual Machine before completion.

Here you can add a second boot drive for a mirror if you have two local storage drives.

Before finishing the creation of the VM click Add, select PCI Devices, and choose the LSI 2308.

Image [32]

And be sure to go into the CD/DVD drive settings and set it to boot off the FreeNAS iso.  Then finish creation of the VM.

8. Install FreeNAS.

Boot of the VM, install it to your SATA drive (or two of them to mirror boot).


After it’s finished installing reboot.

9. Install VMware Tools.

The FreeBSD compat6x and perl packages are no longer available on their FTP site.  I’ve updated the instructions to install the binary version from VMware Tools Installer. 


In VMware right-click the FreeNAS VM,  Choose Guest, then Install/Upgrade VMware Tools.  You’ll then choose interactive mode.

Mount the CD-ROM and copy the VMware install files to FreeNAS:

Once installed Navigate to the WebGUI, it starts out presenting a wizard, I usually set my language and timezone then exit the rest of the wizard.

Under System, Tunables
Add three Tunables.  Variables for each should be: vmxnet3_load, vmxnet_load, and vmmemctl_load,  The type should be Loader and the Value YES on all three.

(I think all that’s needed is the vmxnet3_load)

Reboot FreeNAS.  On reboot you should notice that the VMXNET3 NICS now work (except the NIC on the storage network can’t find a DHCP server, but we’ll set it to static later), also you should notice that VMware is now reporting that VMware tools are installed.


If all looks well shutdown FreeNAS (you can now choose Shutdown Guest from VMware to safely power it off), remove the E1000 NIC and boot it back up (note that the IP address on the web gui will be different).

10.  Update FreeNAS

Before doing anything let’s upgrade FreeNAS to the latest stable under System Update.

This is a great time to make some tea.

Once that’s done it should reboot.  Then I always go back again and check for updates again to make sure there’s nothing left.

11. SSL Certificate on the Management Interface (optional)

On my DHCP server I’ll give FreeNAS a static IP, and setup an entry for it on my local DNS server.  So for this example I’ll have a DNS entry on my internal network for

If you don’t have your own internal Certificate Authority you can create one right in FreeNAS:

System, CAs, Create internal CA.  Increase the key length to 4096 and make sure the Digest Algorithm is set to SHA256.


Click on the CA you just created, hit the Export Certificate button, click on it to install the Root certificate you just created on your computer.  You can either install it just for your profile or for the local machine, I usually do local machine, and you’ll want to make sure to store it is in the Trusted Root Certificate Authorities store.



Just a warning, that you must keep this Root CA guarded, if a hacker were to access this he could generate certificates to impersonate anyone (including your bank) to initiate a MITM attack.

Also Export the Private Key of the CA and store it some place safe.

Now create the certificate…

System, Certificates, Create Internal Certificate.  Once again bump the key length to 4096.  The important part here is the Common Name must match your DNS entry.  If you are going to access FreeNAS via IP then you should put the IP address in the Common Name field.


System, Information.  Set the hostname to your dns name.

System, General.  Change the protocol to HTTPS and select the certificate you created.  Now you should be able to go to use https to access the FreeNAS WebGUI.

12. Setup Email Notifications

Account, Users, Root, Change Email, set to the email address you want to receive alerts (like if a drive fails or there’s an update available).

System, Advanced

Show console messages in the footer.  Enable (I find it useful)

System Email…

Fill in your SMTP server info… and send a test email to make sure it works.

13.  Setup a Proper Swap

FreeNAS by default creates a swap partition on each drive, and then stripes the swap across them so that if any one drive fails there’s a chance your system will crash.  We don’t want this.

System, Advanced…

Swap size on each drive in GiB, affects new disks only. Setting this to 0 disables swap creation completely (STRONGLY DISCOURAGED).   Set this to 0.

Open the shell.  This will create a 4GB swap file (based on


System, Tunables, Add Tunable.

Variable=swapfile, Value=/usr/swap0, Type=rc.conf


Next time you reboot on the left Navigation pane click Display System Processes and make sure the swap shows up.  If so it’s working.


14. Configure FreeNAS Networking

Setup the Management Network (which you are currently using to connect to the WebGUI).

Network, Interfaces, Add Interface, choose the Management NIC, vmx3f0, and set to DHCP.


Setup the Storage Network

Add Interface, choose the Storage NIC, vmx3f1, and set to (I setup my VMware hosts on 10.55.0.x and ZFS servers on 10.55.1.x), be sure to select /16 for the netmask.  And set the mtu to 9000.


Open a shell and make sure you can ping the ESXi host at

Reboot.  Let’s make sure the networking and swap stick.

15. Hard Drive Identification Setup

Label Drives.   FreeNAS is great at detecting bad drives, but it’s not so great at telling you which physical drive is having an issue.  It will tell you the serial number and that’s about it.  But how confident are you in knowing which drive fails?  If FreeNAS tells you that disk da3 (by the way, all these da numbers can change randomly) is having an issue how do you know which drive to pull?  Under Storage, View Disks, you can see the serial number, this still isn’t entirely helpful because chances are you can’t see the serial number without pulling a drive.  So we need to map them to slot numbers or labels of some sort.


There are two ways you can deal with this.  The first, and my preference, is sas2ircu.  Assuming you connected the cables between the LSI 2308 and the backplane in proper sequence sas2ircu will tell you the slot number the drives are plugged into on the LSI controller.  Also if you’re using a backplane with an expander that supports SES2 it should also tell you which slots the drives are in.  Try running this command:


You can see that it tells you the slot number and maps it to the serial number.  If you are comfortable that you know which physical drive each slot number is in then you should be okay.

If not, the second method, is remove all the drives from the LSI controller, and put in just the first drive and label it Slot 0 in the GUI by clicking on the drive, Edit, and enter a Description.



Put in the next drive in Slot 1 and label it, then insert the next drive and label it Slot 2 and so on…

The Description will show up in FreeNAS and it will survive reboots.  it will also follow the drive even if you move it to a different slot.  So it may be more appropriate to make your description match a label on the removable trays rather than the bay number.

It doesn’t matter if you label the drives or use sas2ircu, just make sure you’re confident that you can map a serial number to a physical drive before going forward.

16.  Create the Pool.

Storage, Volumes, Volume Manager.

Click the + next to your HDDs and add them to the pool as RAID-Z2.

Click the + next to the SSDs and add them to the pool.  By default the SSDs will be on one row and two columns.  This will create a mirror.  If you want a stripe just add one Log device now and add the second one later.  Make certain that you change the dropdown on the SSD to “Log (ZIL)”  …it seems to lose this setting anytime you make any other changes so change that setting last.  If you do not do this you will stripe the SSD with the HDDs and possibly create a situation where any one drive failure can result in data loss.


Back to Volume manager and add the second Log device…


I have on numerous occasions had the Log get changed to Stripe after I set it to Log, so just double-check by clicking on the top level tank, then the volume status icon and make sure it looks like this:


17.  Create an NFS Share for VMware

You can create either an NFS share, or iSCSI share (or both) for VMware.  First here’s how to setup an NFS share:

Storage, Volumes, Select the nested Tank, Create Data Set

Be sure to disable atime.


Sharing, NFS, Add Unix (NFS) Share.   Add the vmware_nfs dataset, and grant access to the storage network, and map the root user to root.


Answer yes to enable the NFS service.

In VMware, Configuration, Add Storage, Network File System and add the storage:


And there’s your storage!


18.  Create an iSCSI share for VMware

Note that at this time, based on some of the comments below with people having connection drop issues on iSCSI I suggest testing with heavy concurrent loads to make sure it’s stable.  Watch dmesg and /var/log/messages on FreeNAS for iSCSI timeouts.  Personally I use NFS.  But here’s how to enable iSCSI:

Storage, select the nested tank, Create Zvol.  Be sure compression is set to lz4.  Check Sparse Volume.  Choose advanced mode and optionally change the default block size.  I use 64K block-size based on some benchmarks I’ve done comparing 16K (the default), 64K, and 128K.  64K blocks didn’t really hurt random I/O but helped some on sequential performance, and also gives a better compression ratio.  128K blocks had the best better compression ratio but random I/O started to suffer so I think 64K is a good middle-ground.  Various workloads will probably benefit from different block sizes.


Sharing, Block (iSCSI), Target Global Configuration.

Set the base name to something sensible like:  Set Pool Available Space Threshold to 60%

iscsi _target_global

Portals tab… add a portal on the storage network.


Initiator.  Add Initiator.


Targets.  Add Target.


Extents.  Add Extent.


Associated Targets.  Add Target / Extent.


Under Services enable iSCSI.

In VMware Configutration, Storage Adapters, Add Adapter, iSCSI.

Select the iSCSI Software Adapter in the adapters list and choose properties.  Dynamic discovery tab.  Add…


Close and re-scan the HBA / Adapter.

You should see your iSCSI block device appear…


Configuration, Storage, Add Storage, Disk/LUN, select the FreeBSD iSCSi Disk,


19.  Setup ZFS VMware-Snapshot coordination.

Storage.  Vmware-Snapshot.  Add VMware-Snapshot.  Map your ZFS dataset to the VMware data store.



20. Periodic Snapshots

Add periodic snapshot jobs for your VMware storage under Storage, Periodic Snapshot Tasks.  You can setup different snapshot jobs with different retention policies.


21. ZFS Replication

If you have a second FreeNAS Server (say you can replicate the snapshots over to it.  On, Replication tasks, view public key. copy the key to the clipboard.

On the server you’re replicating to,, go to Account, View Users, root, Modify User, and paste the public key into the SSH public Key field.  Also create a dataset called “replicated”.

Back on

Add Replication.  Do an SSH keyscan.


And repeat for any other datasets.  Optionally you could also just replicate the entire pool with the recursive option.

21.  Automatic Shutdown on UPS Battery Failure (Work in Progress).

The goal is on power loss, before the battery fails to shutdown all the VMware guests including FreeNAS.  So far all I have gotten is the APC working with VMware.  Edit the VM settings and add a USB controller, then add a USB device and select the UPS, in my case a APC Back-UPS ES 550G.  Power FreeNAS back on.

On the shell type:

dmesg|grep APC

This will tell you where the APC device is.  IN my case it’s showing up on ugen0.4.  I ended up having to grant world access to the UPS…

For some reason I could not get the GUI to connect to the UPS, I can selected ugen0.4, but under the drivers dropdown I just have hyphens —— … I set it manually in /usr/local/etc/nut/ups.conf

However, this file gets overwritten on reboot, and also the rc.conf setting doesn’t seem to stick.  I added this tunable to get the rc.conf setting…


And I created my ups.conf file in /mnt/tank/ups.conf.  Then I created a script to stop the nut service, copy my config file and restart the nut service in /mnt/tank/

Then under tasks, Init/Shutdown Scripts I added a task to run the script post init.


Next step is to configure automatic shutdown of the VMware server and all guests on it…  I have not done this yet.

There’s a couple of approaches to take here.  One is to install a NUT client on the ESXi, and the other is to have FreeNAS ssh into VMware and tell it to shutdown.  I may update this section later if I ever get around to implementing it.

22. Backups.

Before going live make sure you have adequate backups!  You can use ZFS mirroring with a fast link.  For slow network connections Rsync will work better (Took under Tasks -> Rsync tasks) or use a cloud service like CrashPlan.

 Setup Complete… mostly.

Well, that’s all for now.