FreeNAS Corral on VMware 6.5

This guide will install FreeNAS 10 (Corral) under VMware 6.5 ESXi, then via NFS share ZFS backed storage back to VMware.  This is an update of my FreeNAS 9.10 on VMware 6.0 Guide.

“Hyperconverged” Design Overview

FreeNAS Vmware

FreeNAS is installed as a Virtual Machine on the VMware Hypervisor.  An LSI HBA in IT Mode is passed to FreeNAS via VT-d Passthrough.  A ZFS pool is created on the disks attacked to the HBA.  ZFS provides RAID-Z redundancy and an NFS dataset is then shared from FreeNAS and mounted from VMware which is used to provide storage for the remaining guests.  Optionally containers and VM guests can run directly on FreeNAS itself using bhyve.

FreeNAS Corral

FreeNAS 10 (now called FreeNAS Corral) is a major rewrite over FreeNAS 9.10, the GUI has been overhauled, it has a CLI interface, and an API.  I think the best feature is the bhyve hypervisor and docker support.  To some degree for a single all-in-one hypervisor+NAS server you may not even need VMware and be able to get away with bhyve and docker.

FreeNAS Corral Dashboard

Like anything new I advise caution against running it in a production environment.  I do see quite a few rough edges and a few missing features that are available in FreeNAS 9.10.  I imagine we’ll see frequent updates with polishing and features added.  A good rule of thumb is to wait until TrueNAS hardware is shipping with the “Corral” version.   I think this is the best release of FreeNAS yet, and it is going to be a great platform moving forward!

1. Get Hardware

This is based on my Supermicro X10SDV Build.  For drives I used 4 x White Label NAS class HDDs (see ZFS Hard Drive Guide) and two Intel DC S3700s (similar models between S3500 and S3720 should be fine), which often show up for a decent price on Ebay.  One SSD will be used to boot VMware and provide the initial data storage and the other used as a ZIL.

You will need an HBA to pass storage to the FreeNAS guest.  I suggest the ServerRAID IBM M1015 flashed to IT mode, or you can usually find the LSI 9210-8i already flashed to IT mode for a decent price on eBay.  You will also need a Mini-SAS to 4x SAS SATA Forward Breakout Cable.

2. IPMI Setup

Go ahead and plug in the network cables to the IPMI management port, as well as at least one of the normal ethernet ports.

This should work with just about any server class Supermicro board…. first download the Supermicro IPMIView tool (I just enter “Private” for the company).  Once installed run “IPMIView20” from the Start Menu (you may need to run it as Administrator).

Scan for IPMI Devices… once it finds your Supermicro server select it and Save.

Login to IPMI using ADMIN / ADMIN (you’ll want to change that obviously).

IPMI Login

KVM Console Tab…

KVM Console Tab

Load the VMware ISO file to the Virtual DVD-ROM drive…

Download the VMware ESXi Free Hypervisor.

Select ISO file, Open Image, select the VMware ISO file which you can download here, and then hit “Plug In”

KVM Virtual Storage

Power on

KVM Power On

Hit Delete repeatedly…

KVM Boot

Change the boot order, I made the ATEN Virtual CD/DVD the primary boot devices, and my Intel SSD DC S3700 that I’ll install VMware to secondary, and disabled everything else.

BIOS Boot Order

Save and Exit, and it should boot the VMware installer ISO.

3. Install VMware ESXi 6.5.0

Install ESXi

VMware Installer

Install to the Intel SSD Drive.

VMware Install Select Drive

Once installation is complete “Plug Out” the Virtual ISO file before rebooting.

Unplug ISO file

Once it comes up get the IP address (or set it if you want it to have a static IP which I highly recommend).

VMware screen

4. PCI Passthrough HBA

Go to that address in your browser (I suggest Chrome).  Manage, Hardware, PCI Devices, select the LSI HBA card and Enable Passthrough.

Passthrough LSI HBA

Reboot

5. Setup VMware Storage Network

In the examples below my LAN / VM Network is on 10.2.0.0/16 Final Portgroups(255.255.0.0) and my Storage network is on 10.55.0.0/16.  You may need to adjust for your network.  My storage network is on VLAN 55.

I like to keep my Storage Network separate from my LAN / VM Network.  So we’ll create a VM Storage Network portgroup with a VLAN ID of 55.

Networking, Port groups, Add Port Group

Add Port Group

Add VM Storage Network with VLAN ID of 55.

(you can choose a different VLAN ID, my storage network is 10.55.0.0/16 so I use “55” to match the network so that I don’t have to remember what VLAN goes to what network, but it doesn’t have to match).

Add a second port group just like it called Storage Network with the same VLAN ID (55).

Storage Network

Add VMKernel NIC

VMKernel NIC

Attach it to the Storage Network and give it an address of 10.55.0.4 with a netmask of 255.255.0.0

VMKernel Storage

You should end up with this…

6. Create a FreeNAS Corral VM

Create VM

FreeBSD (64-bit)

Create VM

Install it to the DC S3700 Datastore that VMware is installed on.

Add PCI Device and Select your LSI Card.

Select HBA PCI

Add a second NIC for the VM Storage Network.  You should have two NICS for FreeNAS, a VM Network and a VM Storage Network and you should set the Adapter Type to VMXNET 3 on both.

VMXNET3

Add NIC

I usually give my FreeNAS VM 2 cores, if doing anything heavy (especially if you’ll be running docker images or bhyve under it you may want to increase that count).  One rule with VMware is do not give VMs more cores than they need.  I usually give each VM one core and only consider more if that particular VM needs more resources.  This will reduce the risk of CPU co-stops from occurring.  Gabrie van zanten’s How too many vCPUs can negatively affect performance is a good read.

2 Cores

ZFS needs memory.  FreeNAS 10 needs 8GB memory minimum.  Lock it.

Made the Hard Disk VMDK 16GB.  There’s an issue with the VMware 6.5 SCSI controller on FreeBSD/FreeNAS.  You’ll know it if you see an error like:

UNMAP failed. disabling BIO_DELETE
UNMAP CDB: 42 00 00 00 00 00 00 00 18 000.
CAM status: SCSI Status Error.
SCSI status: Check Condition.
SCSI sense: ILLEGAL REQUEST asc:26,0 (Invalid field in parameter list).
Command byte 0 is invalid.
Error 22, Unretryable error.

To prevent this, change the Virtual Device Node on the hard drive to SATA controller 0, and SCSI Controller 0 should be LSI Logic SAS

SATA Controller

Add CD/DVD Drive, under CD/DVD Media hit Browse to upload and select the FreeNAS Corral ISO file which you can download from FreeNAS.

Add CD-ROM

7. Install FreeNAS VM

Power on the VM…

Select the VMware disk to install to.  I should note that if you create two VMDKs you can select them both at this screen and it will create a ZFS boot mirror, if you have an extra hard drive you can create another VMware data store there and put the 2nd vmdk there.  This would provide some extra redundancy for the FreeNAS boot pool.  In my case I know the DC S3700s are extremely reliable, and if I lost the FreeNAS OS I could just re-import the pool or failover to my secondary FreeNAS server.

Install FreeNAS to VMDK

Boot via BIOS.

Once FreeNAS is installed reboot and you should get the IP from DHCP on the console (once again I suggest setting this to a static IP).

If you hit that IP with a browser you should have a login screen!

8. Update and Reboot

Before doing anything…. System, Updates, Update and Reboot.

Update

(Note: to get better insight into a task progress head over to the Console and type: task show).

9. Setup SSL Certificate

First, set your hostname, and also create a DNS entry pointing at the FreeNAS IP.

Create Internal CA

Export Certificate

Untar the file and click the HobbitonCA.crt to install it, install it to the trusted Root Certificate Authorities.  I should note that if someone were to compromise your CA or gain the key they could do a MITM attack on you forging SSL certificates for other sites.

Create a Certificate for FreeNAS

Create Certificate

Listen on HTTP+HTTPS and select the Certificate.  I also increase the token Lifetime since I religiously lock my workstation when I’m away.

Listen on HTTPS

And now SSL is Secured

SSL Secured

 

10. Create Pool

Do you want Performance, Capacity, or Redundancy?  Drag the white circle thing where you want on the triangle and FreeNAS will suggest a zpool layout.  With 4 disks I chose “Optimal” and it suggested RAID-Z which is what I wanted.  Be sure to add the other SSD as a SLOG / ZIL / LOG.

Pool Creation

11. Create Users

It’s probably best not to be logging in as root all the time.  Create some named users with Administrator access.

12. Create Top Level Dataset

I like to create a top level dataset with a unique name for each FreeNAS server, that way it’s easier to replicate datasets to my other FreeNAS servers and perform recursive tasks (such as snapshots, or replication) on that top level dataset without having to micromanage them.  I know you can sometimes do recursive tasks on the entire pool, but oftentimes I want to exclude certain datasets from those tasks (such as if those datasets are being replicated from another server).

If you’d like to see more on my reasoning for using a top level dataset see my ZFS Dataset Hierarcy

Storage, tank3, Datasets, New…

Top Level DataSet

13. Setup Samba

Services, Sharing, SMB, set the NetBIOS name and Workgroup and Enable.

Storage, SMB3, Share, to create a new dataset with a Samba Share.  Be sure to set the ownership to a user.

SMB Share

14. Setup NFS Share for VMware

I believe at this time VMware and FreeNAS don’t work together on NFSv4, so best to stick to NFSv3 for now.

NFS Share for VMware

Mount NFS Store in VMware by going to Storage, Datastores, new datastore, Mount NFS datastore.

NFS Mount

15. Snapshots

I setup automatic recursive snapshots on the top level dataset.  I like to do pruning snapshots like this:

every 5 minutes -> keep for 2 hours
every hour -> keep for keep for 2 days
every day -> keep for 1 week
every week -> keep for 4 weeks
every 4 weeks -> keep for 12 weeks

And SAMBA has Previous Versions integration with ZFS Snapshots, this is great for letting users restore their own files.

SMB ZFS Integration

16. ZFS Replication to Backup Server

Before putting anything into production setup automatic backups.  Preferably one onsite and one offsite.

Peering, New FreeNAS, and enter the details for your secondary FreeNAS server.

FreeNAS Peering

 

Now you’ll see why I created a top level dataset under the pool….

Storage, Tank3, Replications, New, select the stor2.b3n.org Peer, source dataset is your top level dataset, tank3/ds4, and target dataset is tank4/ds4 on the backup FreeNAS server.

Compression should be FAST over a LAN or BEST over a low WAN.

FreeNAS Replication

Go to another menu option and then back to Storage, tank3, Replications, replication_ds4, and Start the replication and check back in a couple hours to make sure it’s working.  My first replication attempt hung, so I canceled the task and started it again.  I also found that adjusting the peer interval from 1 minute to 5 seconds under Peering may have helped.

FreeNAS Notifications

16.1 Offsite Backups

It’s also a good idea to have Offsite backups, you could use S3, or a CrashPlan Docker Container, etc.

17. Setup Notifications

You want to be notified when something fails.  FreeNAS can be configured to send an email or sent out Pushbullet notifications.  Here’s how to setup Pushbullet.

Create or Login to your Pushbullet account.  Settings, Account, Create an Access Token

PushBullet Access Token

Services, Alerts & Reporting, Add the access key (bottom right) and configure the alerts to send out via Pushbullet.

PushBullet Setup

You can use the Pushbullet Chrome extension or Android/iOS apps to receive alerts.

18. bhyve VMs and Docker Containers under FreeNAS under VMware

Add another Port Group on your VM Network which allows Promiscuous mode, MAC address changes, and Forged transmits.  You can connect FreeNAS and any VMs you really trust to this port group.

Trusted Portgroup

Power down and edit the FreeNAS VM.  Change the VM Network to VM Network Promiscuous

Network Change

Enable Nested Virtualization, under CPU, Hardware virtualization, [x] Expose hardware assisted virtualization to the guest OS.

Enabled Nested Virtualization

After booting back up you should be able to create VMs and Docker Containers in FreeNAS under VMware.

And more….

Use at your own risk.

More topics may come later if I ever get around to it.

Simulate Sunrise With WiFi Bulb and Home-Assistant

Living up North in the Winter we have long hours of darkness because of the earth’s tilt away from the sun.  This means getting up before the sun rises, and it’s a bit annoying to be jolted awake at 6:00am by an alarm when it’s pitch black outside.   Or if I wake up before the alarm goes off, it’s dark, and I can’t tell if I should be going back to sleep or getting ready to get up without consulting a clock.

Earth Title in Winter (for Northern hemisphere)

Wake-up Naturally with Your Own IoT Bulb

WiFi BulbFor Christmas I got a couple of these MagicLight WiFi Smart LED Bulbs.

(The same bulb is also sold under a different brand name: Flux Wifi Smart LED Bulb.)

There are quite a few IoT (Internet of Things) WiFi Light bulbs on the market, the reason I like these is they don’t rely on the vendors software to control them, and they don’t need to connect out to some cloud service on the internet which increases ones surface area to hackers.

Connect to WiFi

When the bulbs are initially turned on they power on and create their own WiFi hotspot, a phone app connects to it and programs it to connect to your WiFi network.  As with all IoT devices I suggest having a dedicated IoT WiFi SSID and VLAN to keep them off the main network.  They should get an IP address from DHCP, I then give it a static IP assignment with DHCP in pfSense.

Automate with Home Assistant

Home Assistant Controlling LightBulbNext, install Home Assistant (which is a free open source home automation platform) on a server.  I spun up an Ubuntu 16.04 VM.

The MagicLight / Flux Bulbs aren’t smart enough to gradually turn on or off, but I used multiple automation tasks to simulate a gradual fade-on over 30 minutes.  The example below will gradually make the light brighter.  It starts very dim, at 5:15am and stays dim for awhile.   This won’t wake me up if I’m still asleep.  Then around 5:40am it starts to get brighter at a faster rate until it reaches full brightness at 6:10am.

This wakes me up “naturally” every winter morning.  I’m usually awake well before 6 and feel much better than if I had used an alarm.

The light stays on until 8am then it turns off and waits for the next day.

Home Assistant Main Screen ScreenShot

The nice thing about waking up to a gradual light is if I’m already waking up I’ll get up sometime after 5:15am, but if I’m in a deep sleep it won’t wake me up suddenly so I can get a few extra minutes of sleep until around 6am.

Here’s the part I added to configuration.yaml

There are also some other things one could do, Home Assistant can also monitor the weather and sunrise times.  I could probably spend a little more effort and make the script only activate the bulb if the sun hasn’t risen yet, or I could have the bulb wake me up earlier if there’s a lot of snow so I have more time to shovel.  Maybe it could be blue when it’s raining so I know to grab my hat.

My home automation script could definitely use some improvements, but even in it’s present state it’s a big improvement over waking up to an alarm.

Wireless APs and Switches | UniFi

When all of your network devices lose access to the internet all at the same time regularly throughout the day, there is not much to blame other than a bad network cable to your Wireless Access Point (AP), or the Access Point itself.  It wasn’t the cable.  My old Cisco-Linksys E3000’s days were numbered.  Skype calls were dropping, Emby videos streams were getting interrupted, websites weren’t loading.  As with most technical things, the burden to set things right fell on my shoulders.

Ubiquiti UniFi

It was past time to upgrade to 802.11ac anyway.  I use pfSense for my router so all I want is a Wireless AP, I don’t need a combo, so I started my search.  I don’t really like researching APs because consumer devices are pretty awful at security, and enterprise devices involve support contracts and enterprise software and sometimes the security is just as bad.  But WiFi router recommendations are one of the most frequently asked questions from friends and family, and I’ve never had a good answer …until now.   I came across UniFi made by Ubiquiti.  These are the wireless AP’s that Linus Torvalds uses. The products appear to be marketed towards Businesses and Enterprises, but the software to run it is free, and pretty much all I need for my home/soho environment can be configured through the web-interface.

UniFi Access Points (AP)

AP on Wall
AP Mounted on Wall, LEDs turned off

I purchased the UAP-AC-PRO which is their high end model as well as the budget model, the UAP-AC-LITE.  There’s also a “Long Range” model which sits in between them, the UAP-AC-LR which I did not get.

The UniFi AP (Wireless Access Point) looks more like a smoke detector than a wireless access point.  A typical install is mounting them on the ceiling.  Here’s mine mounted on a wall (the circular ring LED is normally blue which is too bright at night, but fortunately it can be turned off).

Power over Ethernet (PoE)

The AP is powered by PoE.  This means you don’t need an AC-DC adapter, instead it gets it’s power from the Ethernet cable.  This works on standard Cat 5e, Cat6, or Cat6a cable.  Normally PoE devices require an expensive PoE capable switch, and I was a bit hesitant of getting into the PoE world, but as long as you buy a single unit and not their bulk pack the UniFi APs usually comes bundled with a PoE injector to get you started.

I had no idea what a PoE injector was, but it turns out to be really simple.  It’s a little box with a power cable, and two Ethernet ports, LAN and PoE.  Just plug the LAN port into your switch and your AP into the PoE port.  Couldn’t be any simpler.  Now, if you’re running a fleet of WiFi access points it probably makes sense to get a PoE switch.  But for one or two in a house the PoE injector is fine.

PoE Injector

Switch -> Ethernet -> PoE Injector -> Powered Ethernet -> AP

Two PoE Options:  Passive or IEEE 802.3af/at

Now, there are a couple of different kinds of PoE.

Here’s the difference: Passive PoE is as dumb as an electrical outlet.  It just sends power through the Ethernet cable whether you need it or not… and this can damage devices not designed for Passive PoE if you accidentally plug a powered Ethernet cable into them.  The much better standard is 802.3af and 802.3at PoE.  With this power isn’t provided until the device requests it, which means it’s very safe and you can plug non-PoE devices into PoE ports without blowing them up.

The UAP-AC-PRO uses 802.3af.

The UAP-AC-LITE and UAP-AC-LR products require passive PoE.  However, I have seen possible signs that Ubiquiti is switching all their products to the IEEE 802.3af/at standards, so it may be worthwhile waiting for the newer models if you don’t want to spend the extra for the Pro model and can afford to wait.

The UniFi Controller

UniFi Controller Overview
UniFi Controller Overview Page

So, these Access Points don’t run a web-server with a management interface.  This is a business/enterprise class solution so it’s meant to be centrally controlled from a single controller.  You will need to download the UniFi Controller (which is free).   Once it’s running you can access it via web browser or the UniFi App for Android or iOS.  The controller can be installed on Windows, Linux, or MacOSX.  If you don’t care about collecting stats it doesn’t need to be up and running all the time so it can be run on a workstation, but if you have a server I recommend running it there.  I created an Ununtu 16.04 VM called “unifi.b3n.org”  I gave it 1GB RAM, 30GB HDD, and 1 core which seems to be plenty.

The install process is straight forward…

Create a file, /etc/apt/sources.list.d.100-ubnt-list

Then add the GPG Keys and install the software.

Go to https://unifi.example.com:8443 (See the bottom of this post for info on setting up a real certificate).

The first time you access it you get Wizard to set it up, after creating an account and such it will have you adopt the UniFi APs on your network.  If they’re plugged in it will find them automatically.  It not only manages APs but also manages UniFi branded routers, switches, cameras, VoIP devices, etc.

Wireless Mapping Tool
Wireless Mapping Tool

I can see how it would help manage a fleet of wireless equipment across multiple sites.  You can see all the devices connected, the AP they’re connected to, signal strength, connection speed, data they’ve used, how they’re authorized to be on the network, VLANs, etc.  I’ve hidden a lot of columns in the screenshot below but it gives you an idea of the data you can get on wireless clients.

UniFi Wireless Clients
Wireless Clients

Neighboring APs

Wasted SpectrumThe UniFi also keeps track of every wireless AP that it has seen.  My neighbors seem to have a lot of HP Smart Printers and TVs that need to waste RF spectrum running their own APs for some reason.  Cars Have APs?  It looks like a lot of cars have their own APs now days?  At least I’m guessing these MitsumiE APs are automobiles that have driven by my house.

Probably Car Hotspots?
Probably Car Hotspots?

UniFi Android / iOS App

The Android app is is just as capable (and I presume it is on iPhone as well), I didn’t do a thorough comb through but at a quick glance it appears every screen and configuration setting in the web interface is available in the Android App.

AP Models Comparison

The APs perform well.  Since I installed the UniFi we have not had a single wireless connection drop, even if I put the AP power settings at their lowest it has better range than my previous AP.  I also setup both APs and my devices had no trouble roaming between the two APs as needed while maintaining connections.

The three main models are:

  • UAP-AC-LITE – 2×2 MIMO on both bands (budget)
  • UAP-AC-LR – 3×3 MIMO on 2.4, 2×2 MIMO on 5GHz (middle)
  • UAP-AC-PRO – 3×3 MIMO on both bands (fastest)

Does 3×3 MIMO make a difference for 2×2 clients?  You might get better reception, but probably not a noticeable difference.  However, if you do have 3×3 capable clients you should see a benefit going to a 3×3 AP.

UAC-AP-PRO vs UAC-AP-LITE Performance and Coverage with 2×2 Clients

Most wireless clients are only 2×2 MIMO these days, and even though I tend to run the latest hardware I only have 2×2 devices which can connect at a maximum speed of 866.7Mbps.  A 3×3 MIMO AP can improve performance of 2×2 MIMO clients because the extra antenna might provide a better signal.  That’s the theory anyway.

I can’t really tell a difference between the two routers in my house, in the Android App Wireless Test I get better uploads speeds on the Pro than I do the Lite, which might be due to it’s extra antenna but I don’t see that performance benefit on our laptops when transferring files back and forth between them and my FreeNAS unit.

I do think I get slightly better upload speeds on the Pro model when I’m far away from the AP.  This may be due to the extra antenna or it could just be subjective.

As far as real life performance on 5GHz setting the channel width to 80Mhz I get about 50-60Mb/s down and 30-40Mb/s up pretty consistently throughout the house, and that’s with multiple wireless clients connected and a pretty saturated RF spectrum.  Here’s an RF Scan at my house… there’s really not a single empty channel even on 5Ghz.

RF Scan

 

UniFi Managed Switches

US-8-60W
US-8-60W

Ubiquiti also sells managed switches, ranging from 8 to 48 ports with a variety of PoE options.  I’ve been wanting try out managed switches so I picked up their small 8-port.  Since I’m running these at home low noise is extremely important.   The two switches that fit the bill are the 8-port US-8-60W (with 4 PoE ports) and the 24-port US-24 (without PoE), both of these models are fanless and silent.

UniFi US-24
US-24

The US-24 doesn’t have PoE on it.  The US-8-60W has four 802.3af PoE ports.  I should note that this switch cannot do passive PoE so it won’t be able to power UniFi’s passive PoE equipment (such as the UAP-AC-LITE).

Ubiquiti US-8-60W UniFi

There are two banks of LEDs, top row is only for the four PoE ports on the right and light up orange if PoE is activated.  The bottom row lights up green on gigabit links and orange for 100Mbps links.  There’s also a blue/white LED on the far left front of the router that’s off.  I do not like blue or white LEDs.  Fortunately as soon as I provisioned it the UniFi Controller automatically turned it off based on my site preferences.

VLAN Configuration

After getting a quick primer from a Network Engineer on how VLAN tagging works I decided to start VLAN tagging my network.

Under the UniFi Controller you can setup your VLANs, I programmed all of mine in above.  Something that is a bit confusing is there are two Network Purpose types that support VLAN tagging, “Corporate” and “VLAN-Only”.  There is no difference between the two, unless you are using the USG (UniFi Security Gateway), which can run a DHCP server for each “Corporate” network type.  Since I’m using pfSense instead of the USG I setup mine as vlan-only.

Then it’s fairly trivial to manage the ports, setting up trunking and access ports for certain VLANs.  In my case port 2 is my trunk port and goes to my pfSense router.  I also ran my Northland Cable connection through the switch so I could get some bandwidth insights.

As always, the UniFi Controller provides some pretty neat insights, it picked up devices not only connected to it but also found devices connected to other switches (notice most of the devices below were found on port 2 which connects up to my VMware vSwitch).

Switch Insights

And UniFi provides great statistics and insights into traffic flow on the switch.

Appendix A: Setting an SSL Certificate on the UniFi Controller.

By default the Unifi controller runs on port 8443 with a self-signed SSL certificate.  It is ridiculously difficult to set a custom cert… I know how to work with Java keystores but I just couldn’t get the ace.jar Java cert importer to accept my intranet cert.  Then I read the CA cert had to be in DER format which also didn’t work…. arrgh.  Suddenly it hit me that setting up certs on nginx is easy, it would be much simpler to set up an SSL certificate on an nginx reverse proxy on port 443.  I want the UniFi Controller listening on 443 anyway, and even better, I don’t have to touch any UniFi configuration files or certs.

If you’re running an internal CA like I am you can just generate an internal Cert, or if you need a public cert Let’s Encrypt should work just as well.  Here’s an example of generating one from FreeNAS.

FreeNAS Unifi Cert Generation

Export the certificate and key and save them to /etc/nginx/cert.crt and /etc/nginx/cert.key.  The configuration is a pretty standard nginx reverse proxy, the only issue I initially ran into was the UniFi controller reported a  “WebSocket connection error” warning, so I enabled nginx’s proxy support for WebSockets (which the configuration below takes care of).  Other than that it’s a straight forward reverse proxy.

Edit /etc/nginx/sites-available/default :

And restart nginx and then I was able to browse to https://unifi.b3n.org with a green SSL cert.

DNS Hijacked? Slow? Setup Unbound on pfSense

Why Is It Slow?

When you request a website, say, b3n.org, your computer needs the IP address.  So it sends out packets through your router/firewall, your modem, and out to your ISPs DNS Servers.  Your ISP’s DNS server will probably have it cached, if not it queries the authoritative (starting with the Root Name Servers) recursively to find out what the authoritative DNS servers are and then queries those DNS servers.  It gets the IP address, and sends it back to your computer.  Your computer can then query the server IP for b3n.org.  Any latency along this process will result in delays.  If you ever type in a url in the address bar and nothing happens for a few hundred milliseconds and then suddenly the website starts to load this is likely the problem.

Typical DNS Lookup Path

Is Your DNS Hijacked by Your ISP?

It’s pretty easy for ISPs to hijack DNS queries.  A small number of ISPs (Comcast, CenturyLink, Time Warner, Cox, Rogers, Charter, Verizon, Sprint, T-Mobile, Frontier, etc.) have been caught doing exactly that.  Want to know why?  Advertising revenue.  When you misspell a domain some ISPs, instead of returning an NXDOMAIN (does not exist) like any RFC compliant DNS server it will resolve the domain anyway, point it at a page they control, and advertise to you!  This is a really bad idea.  But there is a way to prevent your ISP from doing this…

Using Google’s Nameservers

If you’re not tech savvy using 8.8.8.8 and 8.8.4.4 is probably better than your ISPs nameservers.  It won’t hurt, and will probably help, but it may not help… it’s very trivial for an ISP to route those IPs to their own servers and some do.

Even if your ISP is pure goodness and would never do that, someone could setup a rogue DNS server posing as theirs and intercept all your DNS traffic.

The only solution is to query the Root name servers for authoritative DNS servers and use DNSSEC.  Cut out any 3rd party DNS provider and run your own DNS server locally.

Setup an Unbound Server on pfSense

pfSense Unbound Lookups

Unbound is a high performance caching DNS server.  Unbound queries recursively authoritative DNS servers directly, completely bypassing your ISP.  It uses DNSSEC to make sure your queries haven’t been tampered with.  And best of all, it caches DNS results locally (like your ISP would) but since it’s on your own network, the cached DNS queries are local!

You can setup a local FreeBSD server and run Unbound on it, but if you’re already using a router like pfSense or OPNsense you can setup an Unbound server in a few clicks.

Open up pfSense, first make sure the forwarder under Services, DNS Forwarder, is disabled.  Slowness warning: if you are running a low query lookup network such as on your home network having the forwarder disabled may cause lookups to be slower because you’re having to traverse the DNS servers regularly to get results… this can sometimes take a second or two and result in DNS timeouts while it’s trying to traverse the DNS nameservers.  If you find that unbound performance is slow I’d suggest turning on forwarding mode which will use the DNS servers specified in pfSense under system, general setup.  In this case I’d recommend pointing them at 8.8.8.8 and 8.8.4.4.  If you run with forwarding enabled you should verify that your ISP is not hijacking your DNS results, if they are you should switch ISPs.

  • Go to Services, DNS Resolver.
  • Enable the DNS Resolver
  • Select the Network interfaces that you want Unbound to listen on (do not select ALL, you’ll definitly want to select LAN).
  • System Domain Local Zone Type: Transparent
  • Enable DNSSEC Support
  • Do NOT enable Forwarding Mode
  • You can also choose to register DHCP addresses in the DNS Resolver which is very handy if you’re using pfSense to manage DHCP.

pfSense Unbound

  • Under System, General Setup
  • Make sure all DNS Server fields are empty.  DNS Server Override and
  • Disable DNS Forwarder should be unchecked.

pfSense General Setup

Finally, Under Services, DHCP Server, set your DNS Server to your pfSense’s LAN IP.  As your DHCP clients renew their lease they’ll start using pfSense for DNS.

As far as performance if you have low latency to your ISPs DNS you probably won’t notice anything.  But if you’re on a high latency connection  with 70ms pings like I am, this makes a big difference.

 

$5 DigitalOcean, Vultr & Lightsail Benchmarks

Amazon Lightsail has entered the VPS market, competing directly with DigitalOcean and Vultr.  I for one welcome more competition in the $5 cloud server space.  I wanted to see how they perform so I spun up 24 cloud servers, 8 for each provider and ran some benchmarks.

$5 Cloud Server Providers Compared

DigitalOcean, Vultr, and Amazon Lightsail offer more expensive plans, but this post is dealing with the low-end $5 plans.  Here’s how they compare:

DigitalOcean

  • DigitalOcean Logo1 CPU Core
  • 512MB Memory
  • 20GB HDD (extra block storage @ $0.10/GB/month)
  • 1TB Bandwidth ($0.02/GB overage fee in U.S.).
  • Free DNS
  • Best team management – DigitalOcean lets you create multiple-teams and you can add and remove users from those teams.
  • 99.99% SLA
  • Floating IPs
  • Ubuntu, FreeBSD, Fedora, Debian, CoreOS, CentOS

Vultr

  • Vultr Logo1 CPU Core
  • 768MB Memory
  • 15GB HDD (extra block storage @ $0.10/GB/month)
  • 1TB Bandwidth ($0.02/GB overage fee in U.S.)
  • Free DNS
  • Account sharing – allows you to setup multi-user access.
  • 100% SLA
  • Floating IPs (currently can’t setup automatically, requires support setup)
  • Ubuntu, FreeBSD, Fedora, Debian, CoreOS, CentOS, Windows, or any OS with your Custom ISO.

Amazon Lightsail

  • Amaon LightSail Logo1 CPU Core
  • 512MB Memory
  • 20GB HDD (block storage not available)
  • 1TB Bandwidth ($0.09/GB overage fee in U.S.)
  • 3 Free DNS zones (redundancy across TLDs as well).
  • 99.95% SLA
  • Amazon Linux or Ubuntu

Geographic Locations

All three providers have multiple geographic locations worldwide.  Vultr has the most locations in the United States, while Amazon has more geographic locations in the world (although only Virginia is available to LightSail at this point in time).

DigitalOcean Locations

DigitalOcean World Map showing DC locations in Toronto, San Francisco, New York City, London, Amsterdam, Frankfurt, Bangalore, and Singapore

Vultr Global Locations

Vultr map showing DC locations in Seattle, Silicon Valley, Los Angeles, Dallas, Miami, Atlanta, New Jersey, Chicago, London, Frankfurt, Amsterdam, Paris, Singapore, Tokyo, Sydney

Amazon Lightsail Global Infrastructure

AWS world map showing DC regions in Virginia, Oregon, California, Ireland, Frankfurt, Singapore, Tokyo, Seoul, Sydney, Sao Paulo, and China

API Automation

All providers offer an API.  In practice DigitalOcean has been around the longest and thus is more likely to be supported in automation tools (such as Ansible).  I expect support for the other APIs to catch up soon.

And finally…

Benchmarks

CPU Test – Calculating Primes

Number of seconds needed to compute prime numbers.  On the CPU test Amazon Lightsail consistently outperformed, with Vultr coming in second and DigitalOcean last.  CPU1 and CPU2 are 1 and 2 threads respectively calculating primes up to 10,000.  CPU4 is a 4-threaded test calculating primes up to 100,000.

Lower is better.

Memory

Lower is better.

(I accidentally omitted the memory test from my parser script and didn’t realize it until the last test ran, so this is the average of 4 results per provider)

OLTP (Online transaction processing)

Higher is better.

The OLTP load test simulates a transactional database, in general it measures latency on random inserts, updates, and reads against a MariaDB database.  CPU, memory, and storage latency all can effect performance so it’s a good all around indicator.  This test measures the number of transactions per second.  In this area Vultr outperformed DigitalOcean and Amazon Lightsail in 2 and 4 thread tests, while Lightsail took the lead in the 8-thread test.  I don’t know why Lightsail started to perform better under multi-threaded tests, however, my guess is that while Lightsail doesn’t offer the fastest single-threaded storage IOPS it may have better multi-threaded IOPS–but I can’t say for sure without doing some different kinds of tests.  DigitalOcean performed the worst in all tests–probably due to it’s slower CPU and memory speed.

Random IOPS

Higher is better.

Transactions per second.  In random IOPS Vultr provided the best consistent performance, DigitalOcean comes in second place with wide variance, and Lightsail comes in last, but it was by far the most predictable.

Sequential Reads / Writes / Re-writes

Higher is better.

This simply measures sequential read/write speeds on the hard drive.   Vultr offers the most consistent high performance, DigitalOcean is all over the place but generally better than Lightsail which comes in last.

Latency (Ping ms) U.S. Locations

Lower is better.

The U.S. latency is all close enough that it doesn’t matter.

Latency (Ping ms) Worldwide Locations

Lower is better.

International Latency, again the results are pretty close.

Download Speed Tests from U.S. Locations

Higher is better.

Downloading data from various locations.  It’s really hard to conclude any meaningful analysis from this… the faster peering in New York probably has to do with DigitalOcean and Vultr being located in New York vs Lightsail’s location in Virginia.

 

Upload Speed Tests to U.S. Locations

Higher is better.

Due to the similarities in the test results I think the bandwidth constraints are on the other side, or at peering.

Download Speed Tests from Worldwide Locations

Higher is better.

Who knows what one could conclude from this, it seems like various providers have different quality peering to different worldwide locations, but there are so many variables it’s hard to say.

 

Upload Speed Tests to Worldwide Locations

Higher is better.

Similar groupings for the most part.

Testing Methodology

I spun up 24 x $5 servers, 8 for each VPS provider.  I spun up 12 servers yesterday and ran tests, destroyed the VMs, then created 12 new servers today and repeated the tests.  All tests were run in the Eastern United States.   I chose that region because the only location available currently in Amazon Lightsail is Virginia, so to get as close as I could I deployed Vultr and DigitalOcean servers out of their New York (and New Jersey) data centers.  New York is a great place to put a server if you’re trying to provide low latency to the major populations in the United States and Europe without using a CDN.

If the provider had multiple data centers in a region I tried to spread them out.

  • DigitalOcean – I deployed 4 servers in NYC3, 2 in NYC2, and 2 in NYC1.
  • Vultr – All 8 servers deployed in their New Jersey data center.
  • Amazon Lightsail.  Deployed in their Virginia location, 2 in each of their four AWS high availability zones.

All the tests I ran are relatively short duration, I did not benchmark sustained loads which may produce different results.  My general use case is a web-server or small build server with intermittent workloads.  I often spin up servers for a few hours or days and then destroy them once they’re done with their tasks.

Testing Scripts

The testing scripts I used are available in my GitHub meta-vps-bench repository.  The testing scripts are very rudimentary and could be improved.  It runs sysbench and speedtest benchmarks.  The following commands were run on each server as root:

apt update
apt upgrade
reboot
git clone https://github.com/ahnooie/meta-vps-bench.git
cd meta-vps-bench
./setup.sh
./bench.sh
./parse.sh
cat speed*.result

I tried to stagger starting the tests so that multiple speedtests against the same location had a low risk of occurring at the same time… but it may not always work out that way.  I ran all tests twice per server which gives 48 total results (16 for each provider).

This script is for testing.  I do NOT recommend running this on production servers.

Data

I have published my results to Tableau’s Public Cloud.

Profit Center

This was a lot of work, give Ben some money!

Sign up for DigitalOcean using this link and I’ll get $25 service credit!

Sign up for Vultr using this link and I’ll get $30 credit!

Sign up for Amazon Lightsail and I get nothing.  Oh well.

Well, that’s it.

 

Best Gifts for Computer Hackers 2016

Looking for a Christmas gift idea for your computer geek?  Here’s a short gift guide with a few ideas I think would make great gifts.  Unlike a lot of other top gift idea lists written by non-tech people just to make a sale, I’m actually a developer and these are the sort of things that I would enjoy (in fact most of them I own or at the very least had a chance to play with).

Here’s some gifts your geek, hacker, developer, programmer, tech enthusiast, etc. may enjoy:

WiFi RGB LED Light

Wifi LightbulbMagicLight WiFi Smart LED Light Bulb ($).  This looks like a normal light-bulb, but it can connect to your WiFi network and be controlled by your SmartPhone, or through home automation software, or Python scripts.  This Bulb can change to any color.  You can send it HTML Hex Color Codes!  If you live up in North Idaho like I do you can program your light to gradually get brighter in the morning to wake you up naturally in the months where the Sun doesn’t rise until late in the day.  Or program it to redshift in the evenings before bedtime so the blue light isn’t messing with your circadian rhythm.  Or have it turn red as a warning when you’ve left the garage door open after dark!  Put a few outside on your house and set them to be certain colors during the Holidays (Red & Green at Christmas, Orange during Halloween, Red, White, and Blue for Independence Day).

Raspberry Pi 3

Raspberry Pi 3 KitRaspberry Pi 3 Starter Kit ($$).  Every technology enthusiast would enjoy a Raspberry Pi.  There are so many projects you could do… build your own weather station, automatic sprinkler system, home automation server, arcade, even a small computer, tiny server, thermometer, etc.

Python Book

Python Programming Book CoverPython Programming for Beginners ($) by Jason Cannon.  Yes, the name comes from Monty Python.  Python is becoming a well loved language and is growing fast, and is fun to learn and practical.  I have been seeing a lot of increase of this language lately.  This is one of the best programming languages to learn, even if you’re not a programmer.  This book is perfect for someone new to Python or even for someone starting out learning to code for the first time.

Mechanical Keyboard

Mechanical Keyboard
MasterKeys Pro L Mechanical Keyboard.  ($$$).  (This is the latest model, I use an older version of this keyboard at work).  If your hacker is on the young side there’s a good chance he has never experienced the joy of typing on a mechanical keyboard and may not even know they exist!  Does your keyboard let you press every single key on the keyboard simultaneously and they ALL register?  This keyboard does.  Cherry MX KeysThis keyboard has 3 switch options.  Cherry MX Red, Brown, or Blue.  I linked to the Cherry MX Brown version but there are several different switch types:  Cherry MX Reds have no tactile bump, they are linear so great for FPS or RTS gaming where speed matters.  Cherry MX Blues provide an audible click and a tactile bump and are great for typing (unless noise is a concern), Cherry MX Browns provide subtle tactile feedback with no audible click making it a great all-purpose keyboard.  The MX browns are my favorite Cherry switch and it’s what I recommend starting with for most people if you don’t know what you want.

I should mention, that by “no audible click” I mean no added click noise.   Kris tells me the “silent” Browns and Reds are loud compared to a typical keyboard.  The Blues are even louder. 

Civilization

Civ 6 Screenshot
Civilization VI ($$).  This game is one of the longest running series, and in my opinion one of the best turn-based strategy games on the market.  Your gamer geek can play single-player, or online with friends.  Starting out with a single Settler and building cities… what I like about Civilization is the unique ways to win.  Most games are about World Domination through force.  But in Civilization that is just one of many ways to win.  In addition to Domination you can obtain Victory through Culture, Religion, or Science.

Chicory Coffee & Beignet Donuts

Cafe Du Monde
Cafe Du Monde – New Orleans

cafe_du_monde_mix_setChicory Coffee & Beignet Donuts ($).  If you are ever visiting New Orleans you should stop by the Cafe Du Monde (open 24/7) for some Beignet Donuts and Café au lait.  But the next best thing is giving the gift of coffee and donuts for those early mornings or late night programming sessions.  This is one of my favorite coffee flavors, it has a unique taste and everyone I’ve brewed it for loves it.

YubiKey

yubikey-neo-1000-2016-444x444YubiKey Neo ($).  If your hacker is concerned about security you might consider getting him the YubiKey Neo.  It’s a 2nd Factor Authentication device which works with Android (using NFC), Linux, Mac, and Windows.  Everyone should be locking down their accounts (Email, Github, etc.) with a Yubikey.  Yubico is one of the more reputable companies.  Last year a security bug was discovered in the OpenPGP applet and they offered free replacement (including free shipping) for all the affected devices.  Their software to work with they key is open source on GitHub.  YubiKey supports such a large variety of MFA authentication methods including FIDO U2F, HOTP, TOPT, Yubico OTP, PIV-Compliant SmartCard, HMAC-SHA1 challenge response, etc.  It’s really the only authentication device you need.  I can authenticate with just about any service and protocol using a single YubiKey.

ESV Bible

ESV MacArthur Study Bible PersonalESV MacArthur Study Bible Personal Size ($).  Of course, it would be remiss of me not to include a gift that has to do with the very reason we celebrate Christmas.  From the Creation and Fall of man, the Son of God coming to earth to die on the cross to take the penalty for our sins, and raising from the dead so that anyone who believes in Him will have eternal life.  I received this as a gift a few years ago and it’s to date my favorite Bible.  I don’t think you’ll find a higher quality Bible at this price point, it’s even Smyth Sewn which surprised me!  MacArthur has some of the most scholarly and practical (easy enough for me to understand) Study Bible notes on the market today.  His notes are extensive enough to be helpful, yet the personal edition is still small enough to be portable.

Well, that’s my guide for this year.  Wishing everyone a Happy Thanksgiving and a Merry Christmas.

 

Is Your WiFi Unstable?

The most Frequently Asked Question from my Family, Friends, and FOAFs…

Laptop Buyer: What Kind of Laptop Should I buy?
Ben: Get one with an Intel Wireless card

WiFi Cards Matter

wifi_cardThe first piece of advice I have is make sure your wireless card is made by Intel.  Do not get anything else.  You might see other tempting wireless cards for so much less by Dell, Broadcom, Ralink, Killer, Realtek, etc.  These WiFi cards might work with most WiFi hotspots, they might work most of the time, but don’t get them.  The problem is they aren’t robust.  I’ve seen them drop connections randomly, not be able to connect to certain wireless APs, drop out the signal when the Microwave is running, etc.  At best case it works fine but later on a driver update might make it worse.  It is not worth saving a few bucks to deal with these issues.  Pay extra for an Intel branded WiFi Card.  It might cost you $20 more and save you months of frustration.  You’ll thank me later when your card isn’t disconnecting randomly.

This brings me to the 2nd most Frequently Asked Question….

My Wireless Keeps Disconnecting.  Help!

Laptop buyer: So, my wireless signal keeps dropping out.
Ben: Did you get an Intel Card like I told you?
Laptop buyer: No….
Ben: Were you trying to save money and went too cheap?
Laptop buyer: Yes…..

And the 3rd most Frequently Asked Question….

Can You Fix My WiFi Stability?

If Eli can fix it, you can fix it.

replacing_wireless

You will need to swap out your WiFi card.

If you’re in the situation where you bought a laptop with a flaky WiFi card, it’s easy to fix!  Grab an inexpensive Intel 7260 WiFi Card from Amazon.  On most laptops the WiFi card is easily accessible from behind the back cover, usually it’s not more work than a memory upgrade.  Unplug the antenna connectors from your unstable wireless card, pop it out, and put the new card in and hook it up.  Your WiFi connections will now be robust.

Back Story

I don’t say this because I’m an Intel fan.  I just want things to work.  Every couple of years I give another brand a try just to make sure my “only Intel” advice is relevant.  I’ve had the same experience with non-Intel brands the last 15 years!

Last year I decided to buy a cheap laptop to watch movies on (we don’t have a TV) and it came with a Dell DW 1704 / Broadcom 4314 Wireless Card.  I bought it just to see if things had gotten better.  They haven’t.  This wireless NIC can’t stream a full length movie from my media server without losing the wireless signal several times.

And it’s not just me, earlier this year several of my colleagues bought Dell XPS laptops with Killer Branded WiFi cards.  They just don’t work reliably in scenarios that Intel chips do.  In their case they couldn’t connect to several APs.  In my case the connection would drop several times a day.  This was both in Windows 10 and in Linux.  And yes, I tried disabling power saving mode on the WiFi adapter.

I’ve had friends and family not be able to even connect to certain APs at all until they swapped out their Broadcom, Killer, or Ralinks for an Intel card.  Now, you might get lucky and find another brand that works.  To me it’s not worth the hassle.

The next time you buy a computer, get one with an Intel WiFi card.