Take a look at your keyboard. Push a few of the keys. It probably feels mushy, and it’s most likely a rubber-dome keyboard. These weren’t made for typing, they were made to be cheap. You have this dome of rubber that the key sits on, as you press the key feels mushy until the rubber dome sort of collapses and closes the circuit registering the keypress.
A mechanical keyboard by contrast has a real switch and most give audible and tactile feedback when the switch engages. These are much more expensive to produce, but they were very common back in the 80s, and are far superior to most keyboards made today. When people spent $5,000 dollars on a computer it made sense to put $500 dollars into the keyboard! On the left is an image of IBM’s infamous buckling spring key switch, which is by many considered the best keyboard switch ever designed. It simply includes a spring, that when pressed far enough, buckles, causing the spring to hit the wall making an audible click as well as moving the plate to engage the circuit.
MX Cherry Switches
There are a variety of switch types, the most popular and easiest to find keyboard switch on the market today is the Cherry MX mechanical switch. Cherry style switches can be identified by the plus on the stem which the keycap fits over. It comes in a variety of switch types for different styles of typing, designated by colors… here are some of the most popular models and how they behave.
Three Most Popular Cherry MX Switches
Cherry MX Blue – An audible click and tactile feedback on engagement. This switch is popular for typing activities and generally preferred among authors and programmers. It does make a loud click noise so I don’t own this style but I have typed on blues and they are a fantastic typing experience. The one thing I don’t care for in this switch is the reset point is a bit higher than the activation point which requires a little more return travel to re-engage. It’s not really a big deal, just my preference. Most typists actually prefer this as it prevents an accidental double-strike and most gamers do not like this.
Cherry MX Red – No audible click, and no-tactile feedback. This is a linear switch with consistent force all the way through. Because there’s no bump on engagement to “slow you down” and the reset and activation point are the same this switch is very popular with gamers who need to rapidly press a key as quickly as possible. There is no audible click, but they are still noisy if you bottom out the keys.
Cherry MX Brown – No audible click, and light tactile feedback. This is my favorite switch, you get tactile feedback, the activation point is close to the reset point and there’s no audible click (which is a bonus if you don’t want to wake people up, however I should mention despite the keys being “silent” they’re still much louder than a rubberdome). If you’re going to get one all purpose switch (and most of us probably should–switching between different key types all the time probably isn’t that helpful) I think the MX Browns are great. My only complaint with browns is I feel the tactility could be a little sharper.
Less Common Cherry Switches
Cherry also makes a few other switch types with a stiffer spring that are less common but can be found as an option on high end keyboards.
Cherry MX Green – Similar to the blue but a stiffer spring, this isn’t the same as a buckling spring but probably about the closest you can get in the Cherry switch.
Cherry MX Black – Just like the red but a stiffer spring
Cherry MX Clear – Similar to the brown but a stiffer spring. If this was more common I would consider using this over the browns since I prefer a heavier switch.
Cherry MX isn’t the only game in town but they’re manufacturing the most keys today. There are also Topre, ALPS, Kailh, and of course the traditional Buckling Spring Switch. This switch was found in the IBM Model F and IBM Model M. It includes a spring that sits inside each key, as the key is depressed passed a certain point the spring buckles, causing it to whack the side of the key causing tactile feedback and an audible click as the flipper hits the plate. While this is a fantastic keyboard it’s very loud. I had to stop using it when I got roommates. Unicomp bought the rights and manufacturing equipment from IBM so it is still possible to buy a brand new Model M Keyboard with a modern layout today.
Before purchasing a mechanical keyboard. It’s best to test out the switches. 4-key Cherry MX samplers usually have the most common Red, Black, Blue, and Brown switch.
Generally speaking if you deviate from the most common Red, Blue, or Brown switches your options are going to be more limited and more expensive, but you can purchase 12-key samplers to try even more switch types.
Things To Consider In a Keyboard
One of the major issues with modern keyboards is ghosting, especially if you’re left handed and play games. Keyboard manufacturers build keyboards to handle simultaneous keystrokes in the WASD region, which is great if you’re right-handed, but awful if you’re left-handed and using something like OKL; or PL:’ and you find you can’t press O, K, and space at the same time without random keystrokes being sent to the computer. Nicer keyboards tend to have 6KRO (6 Key Rollover) which means you can press any 6 keys simultaneously and have them all register correctly. Some keyboards also support NKRO (N-Key Rollover) which means you can literally press every single key on the keyboard at the same time without losing a keystroke.
PS/2 vs USB
PS/2 is superior. Most modern keyboards have a USB connector, but that doesn’t mean it’s superior to PS/2. The “legacy” PS/2 port has several advantages. First it supports NKRO (some higher end keyboards can do NKRO with USB but it’s not as common), and 2nd on USB your computer polls the keyboard periodically thousands of times a second. The conversation between the keyboard and computer goes like this:
CPU: Hey, any keys pressed? USB Keyboard: Nope. (wait a few milliseconds) CPU: Hey, any keys pressed? USB Keyboard: Nope. (wait a few milliseconds) CPU: Hey, any keys pressed? USB Keyboard: Nope. You: Press J (wait milliseconds) CPU: Hey, any keys pressed? USB Keyboard: Yes, “J” is pressed.
And this can result in a few milliseconds delay between the time you press a key and your CPU realizes it. By contrast on a PS/2 Connection there is no polling, instead PS/2 sends an interrupt to the CPU:
CPU: if a is not null then … !!!interrupted!!! USB Keyboard: Hey CPU! “A” is pressed NOW!
For me and most people this won’t matter. Pro-gamers may prefer PS/2.
I prefer full size keyboards. One thing the bothers me is the wasted space above the numpad. So I always look for keyboards that utilize the space… it’s a great spot to have volume controls like this Ducky keyboard provides.
You can get smaller keyboards without the numpad/tenkey, or even smaller without the arrows, some like the Happy Hacking Keyboard don’t even come with function keys. Instead you have to hit modifier keys to get to the keys you want. I think this is a trick to sell you less keys. I use all the keys fairly often so I prefer to have a full keyboard without having to use modifier keys.
Keycaps are easily swapped out later so not as important. But the two most common plastic materials are PBT and ABS. PBT is higher quality, wears slower and is more expensive. ABS is cheaper and if there is any texture on the keys it wears faster. Caps often use subliminal dye or are doubleshot (two colors of plastic molded together) for the lettering which means the key markings will practically never wear off.
The two most popular layouts are US ANSI and International ISO. Chances are you’ll pick what you’re used to depending on the country you’re in. The two have a different layout especially when it comes to the position of special characters. Since I learned to type in the United States I always get the US ANSI layout (I would probably prefer a larger enter key… but I’m not willing to sacrifice the size of a left-shift).
My Favorite Keyboards (so far)…
After testing a number of keyboard brands Cooler Master and Ducky are my favorites among keyboards you can still buy today for a reasonable price. The keyboards are well built, nice and heavy, and are priced well considering their quality and durability. They both have models with an extra 4 keys above the numpad so as not to waste that area.
Here are the two that I use every day…
My current keyboard is an old Ducky Premier with Cherry MX brown switches. It uses quality PBT keys, and has a nice blue-grey color scheme. Nothing fancy about it. This model has been retired but there are plenty of newer Ducky Keyboards.
Cooler Master Masterkeys Pro
At work I use the Cooler Master Masterkeys Pro, the LED backlighting is annoying so I turn that off, but I really like the smooth ABS keycaps that come with it.
Sidewinder X4 (legacy)
Lastly, there is one non-mechanical keyboard that is fantastic. And that is the Microsoft Sidewinder X4. I must say this is the best rubberdome keyboard ever made. I like the feeling of it better than the more expensive Topre keyboards. Unfortunately I wore mine out and they don’t sell them anymore.
Why Mechanical Keyboards?
I like them for the same reason I like buttons, knobs, and switches over touchscreen interfaces. There is no substitute for physical feedback. On a rubber-dome you can’t tell when a key engages so you press it all the way and bottom out every time. With a mechanical keyboard you quickly learn bottoming out your keys isn’t necessary. Instead you press the key and at some point you feel and hear it pass the activation point so you stop pressing and release it.
Another advantage is the durability, Cherry MX switches are MTBF rated for 50 million keypresses. Older keyboards like the Model M and Model F still work today despite being over 30 years old.
What about Laptop Keyboards?
Unfortunately most laptops are built with cheap keyboards and short key-travel. While there are a few exotic models that come with mechanical keyboards, aside from those the best laptop keyboards in general are going to come with Gaming or Business Class Laptops which I mention on my laptop buying guide. Of those, the Lenovo ThinkPad brand is well known for having the best laptop keyboards in the market. They’re still rubberdome, but less bad than most.
Can you recommend a laptop? It’s one of the questions I’m asked several times a month… and I realized I should just write a guide. So here are some options I think are great and things I think you need to consider before buying a laptop:
Business Vs. Consumer Laptops
Most brands have at least two laptop lines. Consumer class and business class. Consumer class laptops are generally junk. Support is usually bad. Safety isn’t a priority (some consumer laptops have been known to catch fire), generally manufacturers experiment with new features on their consumer lines, consumer laptops sometimes ship with malware, or lots of junk or trial software. They’re not as rugged, the left hinge will break after a year or two. Parts are hard to come by so you can’t fix them. The Wifi cards aren’t Intel so can’t connect to every Wireless network. Don’t buy them. Stick with the business class laptops. It is usually better to buy an old used or refurbished business class laptop than a newer model consumer class.
Marketing is notorious for making things confusing. It’s not obvious what’s business class and what’s consumer quality. Here’s the translation for you (I’ve bolded what I believe are the better quality more rugged products):
Dell Business Product Lines
Latitude = Business / Enterprise
3xxx = budget business laptops, not that great a quality
5xxx = Workhorse
6xxx = I call this the bulky line, but high quality (discontinued)
7xxx = premium ultrabooks
Precision = Business Powerful Workstations, High Performance CPUs and GPUs
XPS = Premium Consumer line. They sort of sit between the consumer and business lines. Great quality, price, and specs but not as rugged as the Latitudes or Precision
Dell Consumer Product Lines
Inspiron = Consumer Line
Alienware = Consumer Gaming Laptops
Chromebook = More like netbooks that run ChromeOS instead of Linux or Windows… unless all you need is a browser stay away from these.
Lenovo Business Product Lines
Thinkpad = Business / Enterprise
X = Thin & Light Ultrabooks
T = Flagship, thinner than P but more powerful than X. Best keyboards are found on the T series.
P = Powerful Workstations, High Performance CPUs and GPUs (formerly W).
E = Small Business laptops –budget, not very good
L = Affordable, not as good as T but a step up from E.
Yoga = Tablet / laptop convertibles (not as rugged)
13 = 13 inch chromebook or netbook (not as rugged)
11e = 11 inch educational notebooks (not really business class)
A = Same as T series but with an AMD processor
Lenovo Consumer Product Lines
Yoga (not to be confused with “ThinkPad Yoga)
IdeaPad = Consumer stuff
Lenovo = Consumer
Legion = Consumer Gaming laptops
Chromebooks = Chromebook or netbooks
Yoga Books = tablet type things
I have included Dell XPS and Apple Macbooks for comparison, they tend to be well built machines but I wouldn’t consider them business class. They’re more in the “prosumer” class. I generally don’t recommend them but they may be good options if you you’re not moving them around a lot. If you want OSX Macbooks are obviously going to be the best bet even though you’re not going to get the ruggedness you’d get with a Latitude or ThinkPad. If you’re going to run Windows or Linux I’d recommend a Latitude or ThinkPad.
Deciphering Model Numbers:
2nd digit after the first number indicates screen size. The “4” in “7480” indicates a 14 inch screen.
3rd digit indicates the generation, almost matching up with the year. The 8 in 7480 = 2018 model year (Latitude is off by a year).
T470, the first 4 indicates the screen size, 7 is roughly the model year. Not sure what the last digit means. Sometimes a model number has a suffix, e.g. T470s or T470p which may differentiate it further (P = powerful, S = slim)
Recommended New Laptops
Latitude 7280 – Quality Ultra Portal laptop, thin and light. Rugged and likely to survive a drop from a few feet. 2.8 LBS.
ThinkPad X270 – Ultra Portable Laptop. Great little notebook, runs a little on the thick side (easier to grip) but the advantage is memory isn’t soldered on and has room for a 2.5 inch drive bay. Has two batteries (internal and external so you can swap the external without powering down) which can go up to 25 hours. This is by far the most modular 12.5 inch laptop.
(in the 12.5 inch category these screens are small, so 1366 x 768 is okay if you need little larger text, otherwise upgrade to 1920 x 1080)
Used / refurbished options include anything in the Latitude 72xx, Thinkpad x2xx series. X220 and earlier have classic keyboards which many consider superior.
13.3 Inch ultrabook
Latitude 7380 – Almost no bezel so it’s the same size as most 12 inch laptops, business version of the very popular Dell XPS, the Latitude version adds a little more durability so I would opt for the Latitude 7380 over the XPS 13.
XPX 13 – This Dell is the “prosumer” version of the above, it’s high quality but not as durable. I don’t think it would hold up to being dropped as well as the 7380, but it’s still a good laptop. WARNING: Some XPS machines don’t ship with Intel branded wireless cards. Make sure it’s Intel.
Macbook Pro 13 – Great laptop, newer ones have an annoying touchbar instead of function keys so watch out for that (unless you want it for some reason).
Used / refurbished options include older gen Dell XPS or a Latitude 7370. 13.3 is a fairly new category so you probably won’t find too many used laptops with this scrern size.
14 Inch Ultrabook (thin and light)
ThinkPad X1 Carbon (5th Gen). A 14 inch screen in the size of a 12 or 13 inch laptop. Very sleek, thin and narrow bezel and quite sturdy. Lightweight, thin, it’s one of the best ultrabooks on the market. Memory cannot be upgraded or replaced so order it with what you need.
Latitude 7480 – Great high quality business laptop. Memory is upgradable. With this latest model E-port snap in docking support has been dropped so if you want docking you’ll need a USB-C dock.
ThinkPad T470s. Thicker than the X1 carbon but thinner than a T470. Still supports snap-in docking and memory can be upgraded. The chassis is slightly less rigid than on the T470 or X1. Also one ram slot is soldered on so won’t be upgradable (2nd ram slot is normal)
Used/refurbished options: Older generation Thinkpad X1, Latitude 74xx, and ThinkPad T4xxs.
14 Inch Workhorse, All purpose laptop
ThinkPad T470 – This is one of the best all around laptops. It’s not too thin that it’s hard to grip, but thin enough to not be bulky. Fantastic keyboard, probably the best on the market. Two batteries, one internal and one external so the external can be swapped out without losing power. With a 6-cell battery (which will cause a bulge) it can get 20 hours battery life, or opt for a 3-cell that’s flush with the laptop. For longevity this laptop is the most modular model in the ThinkPad lineup as far as swapping parts so you should be able to make it last longer if anything breaks. No GPU options for buyers in the U.S.
ThinkPad T470p – Quad Core for heavy CPU and an NVIDIA 940MX GPU making it one of the most powerful notebooks in the 14 inch category. Oddly it does not have a USB-C port.
Latitude 5480 – A little thicker than the 7480, Can be configured with Nvidia 930MX GPU. Latest generation drops the E-docking port. I use an older version of this laptop at work, the E5470, and at home I use a E5450 with NVIDIA. Both have been great computers, and the E-Dock (which is now discontinued) is very robust. Can be configured with quad core processors.
ThinkPad T25 Retro – 25th Anniversary Limited edition. Essentially a high end T470 with an NVIDIA 940MX GPU… and a classic 7-row keyboard. This is the best keyboard available on any laptop made today. I believe this is the only ThinkPad on a T470 chassis to have both a GPU and USB-C port. Unfortunately it’s on the pricey side
Used/refurbished options are the ThinkPad T4xx and ThinkPad T4xxp, Dell E64xx, Dell E54xx, Dell 54xx.
15.6 Inch “ultrabook”
ThinkPad P51s – thin “ultrabook” equipped with Quad Core processor (can be equipped with a Xeon) and NVIDIA Quadro GPU
Precision 5520 – This is one of the few precisions I would consider more prosumer than business class. It’s a re-branded Dell XPS 15, the screen has almost no bezel and the laptop is the same size as many 14 inch laptops. I don’t think it would hold up to make abuse because of how thin it is. However, for a mostly stationary laptop it’s fantastic. Can be equipped with Xeon E3. Note that some of these models don’t ship with Intel Wireless cards which may cause problems. Make sure it’s Intel.
XPS 15 – Same thing as the above. Note that some XPS models are not shipping with Intel Wireless cards which may cause connection problems.
Macbook Pro 15 – Great laptop, newer ones may have a touchbar which I find annoying but can be configured with a normal function key row.
Used / Refurbished options include the ThinkPad P5x series, older gen Dell XPS 15, Precision 5510. This is a newer category so there won’t be as many older models here.
One thing to look out for is the keyboard layout, some 15 inch models have the keyboard offset to the left to make room for a numpad. Some people would rather have the numpad and some would rather have a centered keyboard.
Used / Reburbised Models are Precision 75xx, Precision 35xx, and Dell E65xx, ThinkPad P5x, ThinkPad T5xx, ThinkPad W5xx.
Used / Refurbished – Precision 77xx, P7x, ThinkPad W7xx, Macbook Pro 17 inch.
Buying Used / Refurbished
There are some risks buying used. USB firmware hacks, malware, etc. However, it’s a great way to save money and some sellers provide a 1-year warranty. Most businesses keep ThinkPads and Latitudes for 3-5 years then sell them so you can save a significant amount of money just staying 3 to 5 years behind. Generally you want to buy the laptop from the guy that always kept it docked so it’s still in great condition. Keep in mind that the reason businesses cycle through laptops is the productivity lost due to running slower and fixing failing components is greater than the cost of just buying a new laptop proactively. Just something to keep in mind if you value your time.
It’s probably better to get a used / refurbished ThinkPad or Latitude than it is to buy a new consumer laptop. For newer refurbished items the Dell Outlet, Lenovo Outlet, and the Refurbished Mac store are good places to look.
One of the best places to pick up old refurbished ThinkPads may be WalMart’s website. Also there are plenty of refurbished and used laptops on eBay and sometimes they can be found on Amazon as well. If you are not comfortable installing an OS make sure it comes with a fresh install of Windows and the seller is highly rated and offer returns. Many sellers also offer a warranty.
For used laptops the ThinkPad T, X, P, and W series will be a higher quality than the L and E. Latitude 5000, 6000, and 7000 will be higher quality than the 3000 series.
To roughly find the age of a computer consider the current models for ThinkPad are T470, the middle 7 roughly means it’s a 2017 notebook. Same for the Latitude 5480, the 8 means it’s roughly a 2017 year notebook (guess Latitude is +1 on the year). So if you’re looking around on eBay know that a T440 or Latitude E5440 is roughly a 2013-2014 notebook. The years don’t quite line up perfectly but gives you a general idea. Another indicator to look at for age is the generation of Intel processor used (see CPU section below).
ThinkPad Computrace warning for used ThinkPads. Some ThinkPads have a Computrace feature which allows the owner to track down or remotely disable a laptop if lost or stolen. If enabled only the owner (or one of the previous owners who enabled it) can turn it off. You’ll want to make sure that is turned off before buying a used laptop or if you get one with it enabled ask the owner to turn it off and if they’re not able return it for a refund. If you can’t track the previous owner you can call Computrace and they can attempt to contact the owner for you.
Things You Should Consider
Brand. Dell vs Lenovo. Dell Latitude has better support, service, and screens. Lenovo laptops have better keyboards, build quality, and durability. Both are pretty similar and both brands offer a comparable product in almost every size/model.
Docking Support. Many laptops have the ability to dock into a “docking station”. Dell and Lenovo have proprietary docking connectors and docks. These are great solutions if you’re often working in an office or home. At my house and office I have a docking station hooked up to dual monitors, ethernet, keyboard and mouse. It’s convenient to dock in and have a full desktop experience (having multiple screens increases productivity) then undock when I’m on the go. Not all laptops support docking, but if it’s something you’re interested in be sure to check for that capability.
Customer Support. When issues occur I’ve found Dell to have the best support, usually after a 30 minute phone call they’ll have a technician scheduled to come out the next day. Lenovo is 2nd, you’ll get the same result but usually a longer phone call. In my experience when a Macbook breaks you’re going to be out of commission for a week or two while you send it off for repair.
Warranty. Basic vs NBD (Next Business Day) Onsite. Basic warranty usually means a part will get mailed to you, or you’ll ship your laptop and wait a few weeks for it to come back. When buying new, you have the option to get a more advanced warranty. If you are in situations where a broken computer can be costly then pay extra to get NBD onsite support. A technician will meet you wherever you are, at your house, conference, etc. the next business day with a spare part if something needs to be replaced. For road warriors who can’t have downtime this is a must. On the other hand, if you aren’t traveling consider the cost of NBD vs just having an extra laptop on hand (perhaps your old laptop) you could use while your main one is under repair.
I generally purchase the cheapest warranty (1 year basic) because I have a spare and if my computer breaks early I’ll just buy a new one. Over the long run I think this is cheaper.. but if I was a frequent traveler I’d probably opt for a 3 or 4 year NBD warranty.
Ultrabook vs Mobile Workstation. Ultrabooks are designed to be as thin and light as possible, often because of the smaller size heat can’t be dissipated as quickly so the CPU can’t run at a sustained load for long periods of time without throttling, or a weaker CPU is used. Most people won’t notice throttling and this is becoming less of an issue as CPUs become more efficient. The other sacrifice ultrabooks make is shorter key travel so they don’t have a great typing experience, and fewer ports, slots, and extras like GPUs. Sometimes components like RAM are soldered on and batteries may not be replaceable.
Mobile Workstations can usually be outfitted with more battery, more processing power, more key travel giving them a fantastic typing experience, and are generally easy to service They tend to be heavier, but generally more durable and more likely to be found with more ports, not throttle under heavy load, can get them with a GPU, and often have trappable batteries.
Ways to save money. So, in most cases there are several base configurations which can be customized. I have found in general that Memory and Hard drives are more expensive upgraded through Dell or ThinkPad’s store. Often it’s cheaper to buy a base configuration unit with the CPU you want and then buy your own memory and hard drive. For most people swapping out the hard drive will be difficult because the OS will have to be reloaded so may not be worth it. Sometimes memory is not replaceable so check the laptop your buying to see if it is. Generally this is possible on the workstations and a hit and miss on the ultrabooks. If buying a ThinkPad read the ThinkPad Introduction page which has links to Perks discounts.
Wireless Card. Always get Intel. If it’s not Intel branded, don’t buy the laptop.
Touchscreen. I don’t like touchscren but some people do. Usually both options are available.
Glossy or Matte Screen. I much prefer Matte, I don’t want to see my own reflection in the screen. Usually both options are available.
Screen Size and Quality.
Ono of the most popular screen sizes (and my favorite) is 14.4″, it allows for a full-size keyboard (without tenkey) and seems to me to be the right balance between portability and using it like a workstation (faster CPUs, optional GPUs, more key travel on the keys. The ThinkPad T470 and Latitude 5480 are great workstations in this class, and the Latitude 7480 and ThinkPad X1 Carbon (which is lighter than a lot of 13.3 and 12.5 inch laptops) are great ultraboooks.
For frequent travelers going to 13.3″ or 12.5″ may be better. If you need a bigger screen or a ten-key then a 15.6″ or 17″ is the way to go.
Dell is going to have better quality screens for brightness and color than Thinkpads in general. I think 1920×1080 (FHD) screen resolution is pretty decent. You may want to avoid higher resolutions than that like that because many applications can’t scale properly and become difficult to read.
Apple Laptops have a 16:10 aspect ratio instead of a 16:9. 16:9 is the aspect ratio that movies are in, but in most cases the 16:10 (extra vertical space) would be preferable.
Some newer laptops are coming out with aspect ratios with more vertical space such as 3:2 which is a good compromise between 4:3 and 16:9 but they haven’t made it to mainstream yet.
Keyboard. The best keyboards will be on the ThinkPads, and the best of those will be on the Thinkpad T series, and the best one on the market today is the ThinkPad 25 but at a high cost premium. If you use a computer to consume media this won’t matter. If you’re going to be docked in most of the time it’s not a big deal since you’ll use an external keyboard. If you type a lot on your laptop the ThinkPads will be better than Dells or Apples.
Keyboard Lighting: Most laptops have a backlit option, if you want it make sure it’s there. Some older ThinkPads have a “ThinkLight” which is a light on the top of the screen that shines down on the keyboard.
CPU: Stick to the Intel Core i5 or i7 CPUs, whichever is cheapest. For the most part there is very little difference between an i5 and i7, in smaller computers the i5 will perform as well or better than an i7 because it puts out less heat so doesn’t have to throttle as much. AMD processors have been behind Intel in Laptops, would consider them 1 or 2 generations behind Intel although they have started to close the gap with the Rzyen processors they’re still a year behind Intel. I would consider a newer AMD CPU if the price was right but for anything older than 7th gen AMD stick to Intel.
In general, since the i series most CPU generational changes are not that substantial, maybe adding 10-20% boost in performance between each generation so the need to buy a new computer often to get a faster CPU is not particularly great these days. Most of the gains are around power consumption and battery life. However, the 8th gen CPUs which should be widely available next year (2018) offer about a 30-40% improvement over 7th gen because of an increase in core count. You can tell which generation you’re buying by looking at the first number after the “i5” or “i7” E.g. a Core i5-7600 is a 7th Gen. The Core i5-8600 is 8th Gen.
Memory: 8GB should be your absolute minimum. I always get 16GB memory, but I try to buy a laptop with the smallest amount of memory possible and buy extra memory from Amazon.
Hard drive: The single best thing you can do for computer performance is to get an SSD. You do need to watch out for size. NVMe SSDs tend to be faster. Both will well outperform a normal hard drive. SSDs are smaller so make sure you get an adequate size. Minimum of 256GB for most people. If you are my mother in law maybe a terabyte minimum.
Graphics Card / GPU: Most laptops are not great for gaming. If you are buying a dedicated gaming laptop most of my recommendations are not ideal and you many want to look at other options. But if you do play video games you should consider getting a laptop with an AMD or NVIDIA card in it, you’ll be better off than without it. You’re not going to get the performance from a laptop that you would out of a desktop gaming computer, but you can get pretty far. Having a GPU usually cuts into battery life but it’s not as bad as it used to be… most laptops can shutdown the discrete video card when not in use and use the built in Intel HD graphics on the CPU which is more battery friendly. Another option is to get a laptop without a GPU, but use an eGPU enclosure and buy a desktop GPU to put in it… it will connect to your computer via Thunderbolt port.
Batteries. There are usually a few options for batteries. Many laptops don’t have removable batteries. For laptops with removable batteries smaller ones tend to sit flush with the laptop. Some laptops also offer larger battery packs (and even slices) that make the laptop bulkier but can provide more than 20 hours of battery life.
Some laptops can be adjusted to make the battery last longer by reducing the charge cycles. E.g. set your laptop to not start charging the battery until it drops below 80% instead of 95%, and having it charge to only 90% capacity may improve the longevity of the battery quite a bit at the cost of perhaps an hour of battery life of run-time.
DVD Drives. It’s hard to find newer laptops with DVD drives, but some are available, especially if buying older used models. Generally you can buy a blu-ray laptop drive and swap it out if you want to watch blu-way video.
Ports. Consider what ports you will want on your laptop. Is Ethernet important? How many USB ports do you need? What about USB-C? What about a docking port? If you present frequently maybe you want a laptop with a VGA port and an HDMI port? What about SD Card readers? Headphone jack? Do you have to use a Smart Card to access certain systems? In most cases I’ve found I use ports less frequently than I think I would–for me an SD Card reader, Ethernet and a couple USB ports is all I need.
Webcam, Microphone, and Speakers. If you care about these things google the laptop model you’re looking at plus the word “review” and read a few reviews to see if you can get a sense of the microphone, webcam, and speaker quality. Some laptops have the webcams placed at the bottom of the screen instead of the top which results in a weird angle when on using video calls. Also, some laptops don’t have very good speakers so check reviews to see if they’re good, my Dell Latitude E5450’s speaker is so weak I can’t really hear the audio in movies 3 feet away unless there’s absolutely no other noise.
When do new laptop models get released? It depends, I usually see new Latitudes and ThinkPads announced and released between January and April. Often new models are announced at electronic shows. But it depends on whether Intel and all the other suppliers are on schedule so things often get shifted around quite a bit.
Are there other good laptops than the ones you mentioned? Yes there are. There are other decent brands, some of the consumer laptops are fantastic. I don’t know every possible laptop out there at every given moment. This guide is meant to be more of a generic guide looking at good laptop lines over time, with the availability of NBD support if needed, and docking solutions across a wide range of options from workstations with GPUs to ultrabooks. For the most part those come from ThinkPad and Dell, but that doesn’t mean a gem isn’t produced under other brands from time to time.
Ben’s Law: within a 4 hour block of time, for each unit of uninterrupted time in hours (t), the value of productivity and creativity is roughly t^2.
An interruption resets t to zero.
p = t^2 c = t^2
if t = 1 (1 hour of uninterrupted time) then p (productivity) = 1 and c (creativity) = 1
if t = 2 then p = 4 (4 times more productive then at t = 1) and c = 4
and so on…
Now, I say roughly, because around the 4th hour–as it gets closer to lunch productivity starts to go down, the curve probably looks more like the below but p or c=t^2 is close enough.
Uninterrupted Development – Ideal 4 hour block of time
The below is very difficult to achieve. This only happens to me once or twice a month, but when I get a 4 hour block of uninterrupted time I get more done during the last two hours of that block than I do in an entire week!
Writing programs is not at all like rote work, or any job where you’re following a procedure and can just pick up where you left off. Development is more of a creative task, it requires time to ramp up, load what you’re trying to accomplish in your head. You can’t always switch into creativity mode on demand and just start coding, you just find yourself one second staring at the code, and the next moment you’re unaware of your surroundings, you’re in the zone and the longer you can stay there the more you can accomplish. I would say programming is more creative than most people think. It’s more like painting, or writing a book, or composing music than it is engineering. Interrupting a programmer is like interrupting a musician in the middle of a song.
Interrupted Development – Real World 4 hour block of time sliced to bits
This is more like the real world, and probably is a better indicator of most programmer’s 4 hour blocks of times. You can get some work done this way, but it takes about a week to do what could be a day’s worth of work. A quick interruption sometimes won’t cause enough damage to reset back to zero, but anything over a few minutes will do so.
4 x 2TB HGST RAID-Z, 100GB Intel DC S3700s for ZIL (over-provisioned at 8GB) on an M1015. In Environments 1 and 2 this was passed to FreeNAS via VT-d.
2 x Samsung FIT USBs for booting OS (either ESXi or FreeNAS)
1 x extra DC S3700 used as ESXi storage for the FreeNAS VM to be installed on in environments 1 and 2 (not used in environment 3).
E1. ESXi + FreeNAS 11 All-in-one.
Setup per my FreeNAS on VMware Guide. Ubuntu VM with Paravirtual is installed as an ESXi guest, on NFS storage backed by ZFS on FreeNAS which has raw access to disks running under the same ESXi hypervisor using virtual networking. FreeNAS given 2 cores and 10GB memory. Guest gets 1GB memory. Guest tested with 1C and 2C.
E2. Nested bhyve + ESXi + FreeNAS 11 All-in-one.
Nested virtualization test. Ubuntu VM with VirtIO is installed as a bhyve guest on FreeNAS which has raw access to disks running under the ESXi Hypervisor. FreeNAS given 4 cores and 12GB memory. Guest gets 1GB memory. Guest tested with 1C and 2C. What is neat about this environment is it could be used as a stepping stone if migrating from environment 1 to environment 3 or vice-versa (I actually tested migrating with success).
E3. bhyve + FreeNAS 11
Ubuntu VM with VirtIO is installed as a bhyve guest on FreeNAS on bare metal. Guest gets 1GB memory. Guest was backed with a ZVOL since that was the only option. Tested wih 1C and 2C.
All environments used FreeNAS 11, E1 and E2 used VMware ESXi 6.5
A reboot of the guest and FreeNAS was performed between each test so as to clear ZFS’s ARC (in memory read cache). The sysbench test files were recreated at the start of each test. The script I used for testing is https://github.com/ahnooie/meta-vps-bench with networking tests removed.
No attempts on tuning were made in any environment. Just used the sensible defaults.
Disclaimer on comparing Apples to Oranges
This is not a business or enterprise level comparison. This test is meant to show how an Ubuntu guest performs in various configurations on the same hardware with constraints of a typical budget home server running a free “hyperconverged” solution–a hypervisor and FreeNAS storage on the same physical box. Not all environments are meant to perform identically…my goal is just to see if the environments perform “good enough” for home use. An obvious example of this is environments using NFS backed storage are going to perform slower than environments with local storage… but it should still at the very least max out a 1Gbps ethernet. This set of tests is designed to benchmark how I would setup each environment given the constraint of one physical box running both the hypervisor and FreeNAS + ZFS as the storage backend. The test is limited to a single guest VM. In the real world dozens, if not hundreds or even thousands of VMs are running simultaneously so advanced hypervisor features like memory deduplication are going to make a big difference. This test made no attempt to benchmark such. This is not an apples to apples test, so be careful what conclusions you derive from it.
CPU 1 and 2 threaded test
I’d say these are equivalent, which probably shows how little overhead there is from the hypervisor these days, though nested virtualization is a bit slower.
CPU 4 threaded test
Good to see that 2 cores actually performs faster than 1 core on a 4 threaded test. Nothing to see here…
Memory Operations Per Second
Horrible performance with nested, but with the hypervisor on bare metal ESXi and bhyve performed identically.
Once again nested virtualization was slow.. other than that neck and neck performance.
OLTP Transactions Per Second
The ESXi environment clearly takes the lead over bhyve, especially as the number of cores / threads started increasing. This is interesting because ESXi outperforms despite an I/O penalty from using NFS so ESXi is more than making up for that somewhere else.
Disk I/O Requests per Second
Clearly there’s an advantage to using local ZFS storage vs NFS. I’m a bit disappointing in the nested virtualization performance since from a storage standpoint it should be equivalent to bare metal FreeNAS, but may be due to the slow memory performance in that environment.
Disk Sequential Read/Write MBps
No surprises, ZFS local storage is going to outperform NFS
Well there you have it. I think it’s safe to say that bhyve is a viable solution for home (although I would like to see more people using it in the wild before considering it robust–I imagine we’ll see more of that now that FreeNAS has a UI for it). For low resource VMs E2 (nested virtualization) is a way to migrate between E1 and E3–but it’s not going to work for high performance VMs because of the memory performance hit.
This guide will install FreeNAS 10 (Corral) under VMware 6.5 ESXi, then via NFS share ZFS backed storage back to VMware. This is an update of my FreeNAS 9.10 on VMware 6.0 Guide.
“Hyperconverged” Design Overview
FreeNAS is installed as a Virtual Machine on the VMware Hypervisor. An LSI HBA in IT Mode is passed to FreeNAS via VT-d Passthrough. A ZFS pool is created on the disks attacked to the HBA. ZFS provides RAID-Z redundancy and an NFS dataset is then shared from FreeNAS and mounted from VMware which is used to provide storage for the remaining guests. Optionally containers and VM guests can run directly on FreeNAS itself using bhyve.
FreeNAS 10 (now called FreeNAS Corral) is a major rewrite over FreeNAS 9.10, the GUI has been overhauled, it has a CLI interface, and an API. I think the best feature is the bhyve hypervisor and docker support. To some degree for a single all-in-one hypervisor+NAS server you may not even need VMware and be able to get away with bhyve and docker.
Like anything new I advise caution against running it in a production environment. I do see quite a few rough edges and a few missing features that are available in FreeNAS 9.10. I imagine we’ll see frequent updates with polishing and features added. A good rule of thumb is to wait until TrueNAS hardware is shipping with the “Corral” version. I think this is the best release of FreeNAS yet, and it is going to be a great platform moving forward!
1. Get Hardware
This is based on my Supermicro X10SDV Build. For drives I used 4 x White Label NAS class HDDs (see ZFS Hard Drive Guide) and two Intel DC S3700s (similar models between S3500 and S3720 should be fine), which often show up for a decent price on Ebay. One SSD will be used to boot VMware and provide the initial data storage and the other used as a ZIL.
Go ahead and plug in the network cables to the IPMI management port, as well as at least one of the normal ethernet ports.
This should work with just about any server class Supermicro board…. first download the Supermicro IPMIView tool (I just enter “Private” for the company). Once installed run “IPMIView20” from the Start Menu (you may need to run it as Administrator).
Scan for IPMI Devices… once it finds your Supermicro server select it and Save.
Login to IPMI using ADMIN / ADMIN (you’ll want to change that obviously).
KVM Console Tab…
Load the VMware ISO file to the Virtual DVD-ROM drive…
Select ISO file, Open Image, select the VMware ISO file which you can download here, and then hit “Plug In”
Hit Delete repeatedly…
Change the boot order, I made the ATEN Virtual CD/DVD the primary boot devices, and my Intel SSD DC S3700 that I’ll install VMware to secondary, and disabled everything else.
Save and Exit, and it should boot the VMware installer ISO.
3. Install VMware ESXi 6.5.0
Install to the Intel SSD Drive.
Once installation is complete “Plug Out” the Virtual ISO file before rebooting.
Once it comes up get the IP address (or set it if you want it to have a static IP which I highly recommend).
4. PCI Passthrough HBA
Go to that address in your browser (I suggest Chrome). Manage, Hardware, PCI Devices, select the LSI HBA card and Enable Passthrough.
5. Setup VMware Storage Network
In the examples below my LAN / VM Network is on 10.2.0.0/16 (255.255.0.0) and my Storage network is on 10.55.0.0/16. You may need to adjust for your network. My storage network is on VLAN 55.
I like to keep my Storage Network separate from my LAN / VM Network. So we’ll create a VM Storage Network portgroup with a VLAN ID of 55.
Networking, Port groups, Add Port Group
Add VM Storage Network with VLAN ID of 55.
(you can choose a different VLAN ID, my storage network is 10.55.0.0/16 so I use “55” to match the network so that I don’t have to remember what VLAN goes to what network, but it doesn’t have to match).
Add a second port group just like it called Storage Network with the same VLAN ID (55).
Add VMKernel NIC
Attach it to the Storage Network and give it an address of 10.55.0.4 with a netmask of 255.255.0.0
You should end up with this…
6. Create a FreeNAS Corral VM
Install it to the DC S3700 Datastore that VMware is installed on.
Add PCI Device and Select your LSI Card.
Add a second NIC for the VM Storage Network. You should have two NICS for FreeNAS, a VM Network and a VM Storage Network and you should set the Adapter Type to VMXNET 3 on both.
I usually give my FreeNAS VM 2 cores, if doing anything heavy (especially if you’ll be running docker images or bhyve under it you may want to increase that count). One rule with VMware is do not give VMs more cores than they need. I usually give each VM one core and only consider more if that particular VM needs more resources. This will reduce the risk of CPU co-stops from occurring. Gabrie van zanten’s How too many vCPUs can negatively affect performance is a good read.
To prevent this, change the Virtual Device Node on the hard drive to SATA controller 0, and SCSI Controller 0 should be LSI Logic SAS
Add CD/DVD Drive, under CD/DVD Media hit Browse to upload and select the FreeNAS Corral ISO file which you can download from FreeNAS.
7. Install FreeNAS VM
Power on the VM…
Select the VMware disk to install to. I should note that if you create two VMDKs you can select them both at this screen and it will create a ZFS boot mirror, if you have an extra hard drive you can create another VMware data store there and put the 2nd vmdk there. This would provide some extra redundancy for the FreeNAS boot pool. In my case I know the DC S3700s are extremely reliable, and if I lost the FreeNAS OS I could just re-import the pool or failover to my secondary FreeNAS server.
Boot via BIOS.
Once FreeNAS is installed reboot and you should get the IP from DHCP on the console (once again I suggest setting this to a static IP).
If you hit that IP with a browser you should have a login screen!
8. Update and Reboot
Before doing anything…. System, Updates, Update and Reboot.
(Note: to get better insight into a task progress head over to the Console and type: task show).
9. Setup SSL Certificate
First, set your hostname, and also create a DNS entry pointing at the FreeNAS IP.
Create Internal CA
Untar the file and click the HobbitonCA.crt to install it, install it to the trusted Root Certificate Authorities. I should note that if someone were to compromise your CA or gain the key they could do a MITM attack on you forging SSL certificates for other sites.
Create a Certificate for FreeNAS
Listen on HTTP+HTTPS and select the Certificate. I also increase the token Lifetime since I religiously lock my workstation when I’m away.
And now SSL is Secured
10. Create Pool
Do you want Performance, Capacity, or Redundancy? Drag the white circle thing where you want on the triangle and FreeNAS will suggest a zpool layout. With 4 disks I chose “Optimal” and it suggested RAID-Z which is what I wanted. Be sure to add the other SSD as a SLOG / ZIL / LOG.
11. Create Users
It’s probably best not to be logging in as root all the time. Create some named users with Administrator access.
12. Create Top Level Dataset
I like to create a top level dataset with a unique name for each FreeNAS server, that way it’s easier to replicate datasets to my other FreeNAS servers and perform recursive tasks (such as snapshots, or replication) on that top level dataset without having to micromanage them. I know you can sometimes do recursive tasks on the entire pool, but oftentimes I want to exclude certain datasets from those tasks (such as if those datasets are being replicated from another server).
Services, Sharing, SMB, set the NetBIOS name and Workgroup and Enable.
Storage, SMB3, Share, to create a new dataset with a Samba Share. Be sure to set the ownership to a user.
14. Setup NFS Share for VMware
I believe at this time VMware and FreeNAS don’t work together on NFSv4, so best to stick to NFSv3 for now.
Mount NFS Store in VMware by going to Storage, Datastores, new datastore, Mount NFS datastore.
I setup automatic recursive snapshots on the top level dataset. I like to do pruning snapshots like this:
every 5 minutes -> keep for 2 hours every hour -> keep for keep for 2 days every day -> keep for 1 week every week -> keep for 4 weeks every 4 weeks -> keep for 12 weeks
And SAMBA has Previous Versions integration with ZFS Snapshots, this is great for letting users restore their own files.
16. ZFS Replication to Backup Server
Before putting anything into production setup automatic backups. Preferably one onsite and one offsite.
Peering, New FreeNAS, and enter the details for your secondary FreeNAS server.
Now you’ll see why I created a top level dataset under the pool….
Storage, Tank3, Replications, New, select the stor2.b3n.org Peer, source dataset is your top level dataset, tank3/ds4, and target dataset is tank4/ds4 on the backup FreeNAS server.
Compression should be FAST over a LAN or BEST over a low WAN.
Go to another menu option and then back to Storage, tank3, Replications, replication_ds4, and Start the replication and check back in a couple hours to make sure it’s working. My first replication attempt hung, so I canceled the task and started it again. I also found that adjusting the peer interval from 1 minute to 5 seconds under Peering may have helped.
16.1 Offsite Backups
It’s also a good idea to have Offsite backups, you could use S3, or a CrashPlan Docker Container, etc.
17. Setup Notifications
You want to be notified when something fails. FreeNAS can be configured to send an email or sent out Pushbullet notifications. Here’s how to setup Pushbullet.
Create or Login to your Pushbullet account. Settings, Account, Create an Access Token
Services, Alerts & Reporting, Add the access key (bottom right) and configure the alerts to send out via Pushbullet.
You can use the Pushbullet Chrome extension or Android/iOS apps to receive alerts.
18. bhyve VMs and Docker Containers under FreeNAS under VMware
Add another Port Group on your VM Network which allows Promiscuous mode, MAC address changes, and Forged transmits. You can connect FreeNAS and any VMs you really trust to this port group.
Power down and edit the FreeNAS VM. Change the VM Network to VM Network Promiscuous
Enable Nested Virtualization, under CPU, Hardware virtualization, [x] Expose hardware assisted virtualization to the guest OS.
After booting back up you should be able to create VMs and Docker Containers in FreeNAS under VMware.
Use at your own risk.
More topics may come later if I ever get around to it.
Living up North in the Winter we have long hours of darkness because of the earth’s tilt away from the sun. This means getting up before the sun rises, and it’s a bit annoying to be jolted awake at 6:00am by an alarm when it’s pitch black outside. Or if I wake up before the alarm goes off, it’s dark, and I can’t tell if I should be going back to sleep or getting ready to get up without consulting a clock.
There are quite a few IoT (Internet of Things) WiFi Light bulbs on the market, the reason I like these is they don’t rely on the vendors software to control them, and they don’t need to connect out to some cloud service on the internet which increases ones surface area to hackers.
Connect to WiFi
When the bulbs are initially turned on they power on and create their own WiFi hotspot, a phone app connects to it and programs it to connect to your WiFi network. As with all IoT devices I suggest having a dedicated IoT WiFi SSID and VLAN to keep them off the main network. They should get an IP address from DHCP, I then give it a static IP assignment with DHCP in pfSense.
Automate with Home Assistant
Next, install Home Assistant (which is a free open source home automation platform) on a server. I spun up an Ubuntu 16.04 VM.
The MagicLight / Flux Bulbs aren’t smart enough to gradually turn on or off, but I used multiple automation tasks to simulate a gradual fade-on over 30 minutes. The example below will gradually make the light brighter. It starts very dim, at 5:15am and stays dim for awhile. This won’t wake me up if I’m still asleep. Then around 5:40am it starts to get brighter at a faster rate until it reaches full brightness at 6:10am.
This wakes me up “naturally” every winter morning. I’m usually awake well before 6 and feel much better than if I had used an alarm.
The light stays on until 8am then it turns off and waits for the next day.
The nice thing about waking up to a gradual light is if I’m already waking up I’ll get up sometime after 5:15am, but if I’m in a deep sleep it won’t wake me up suddenly so I can get a few extra minutes of sleep until around 6am.
Here’s the part I added to configuration.yaml
- platform: flux_led
alias: Office Light 15 at 5:30am
alias: Office Light 65 at 5:40am
alias: Office Light 120 at 5:50am
alias: Office Light 145 at 6:00am
alias: Office Light 165 at 6:05am
alias: Office Light 200 at 6:08am
alias: Office Light 255 at 6:10am
alias: Turn off at 8am
There are also some other things one could do, Home Assistant can also monitor the weather and sunrise times. I could probably spend a little more effort and make the script only activate the bulb if the sun hasn’t risen yet, or I could have the bulb wake me up earlier if there’s a lot of snow so I have more time to shovel. Maybe it could be blue when it’s raining so I know to grab my hat.
My home automation script could definitely use some improvements, but even in it’s present state it’s a big improvement over waking up to an alarm.
When all of your network devices lose access to the internet all at the same time regularly throughout the day, there is not much to blame other than a bad network cable to your Wireless Access Point (AP), or the Access Point itself. It wasn’t the cable. My old Cisco-Linksys E3000’s days were numbered. Skype calls were dropping, Emby videos streams were getting interrupted, websites weren’t loading. As with most technical things, the burden to set things right fell on my shoulders.
It was past time to upgrade to 802.11ac anyway. I use pfSense for my router so all I want is a Wireless AP, I don’t need a combo, so I started my search. I don’t really like researching APs because consumer devices are pretty awful at security, and enterprise devices involve support contracts and enterprise software and sometimes the security is just as bad. But WiFi router recommendations are one of the most frequently asked questions from friends and family, and I’ve never had a good answer …until now. I came across UniFi made by Ubiquiti. These are the wireless AP’s that Linus Torvalds uses. The products appear to be marketed towards Businesses and Enterprises, but the software to run it is free, and pretty much all I need for my home/soho environment can be configured through the web-interface.
The UniFi AP (Wireless Access Point) looks more like a smoke detector than a wireless access point. A typical install is mounting them on the ceiling. Here’s mine mounted on a wall (the circular ring LED is normally blue which is too bright at night, but fortunately it can be turned off).
Power over Ethernet (PoE)
The AP is powered by PoE. This means you don’t need an AC-DC adapter, instead it gets it’s power from the Ethernet cable. This works on standard Cat 5e, Cat6, or Cat6a cable. Normally PoE devices require an expensive PoE capable switch, and I was a bit hesitant of getting into the PoE world, but as long as you buy a single unit and not their bulk pack the UniFi APs usually comes bundled with a PoE injector to get you started.
I had no idea what a PoE injector was, but it turns out to be really simple. It’s a little box with a power cable, and two Ethernet ports, LAN and PoE. Just plug the LAN port into your switch and your AP into the PoE port. Couldn’t be any simpler. Now, if you’re running a fleet of WiFi access points it probably makes sense to get a PoE switch. But for one or two in a house the PoE injector is fine.
Now, there are a couple of different kinds of PoE.
Here’s the difference: Passive PoE is as dumb as an electrical outlet. It just sends power through the Ethernet cable whether you need it or not… and this can damage devices not designed for Passive PoE if you accidentally plug a powered Ethernet cable into them. The much better standard is 802.3af and 802.3at PoE. With this power isn’t provided until the device requests it, which means it’s very safe and you can plug non-PoE devices into PoE ports without blowing them up.
The UAP-AC-PRO uses 802.3af.
The UAP-AC-LITE and UAP-AC-LR products require passive PoE. However, I have seen possible signs that Ubiquiti is switching all their products to the IEEE 802.3af/at standards, so it may be worthwhile waiting for the newer models if you don’t want to spend the extra for the Pro model and can afford to wait.
The UniFi Controller
So, these Access Points don’t run a web-server with a management interface. This is a business/enterprise class solution so it’s meant to be centrally controlled from a single controller. You will need to download the UniFi Controller (which is free). Once it’s running you can access it via web browser or the UniFi App for Android or iOS. The controller can be installed on Windows, Linux, or MacOSX. If you don’t care about collecting stats it doesn’t need to be up and running all the time so it can be run on a workstation, but if you have a server I recommend running it there. I created an Ununtu 16.04 VM called “unifi.b3n.org” I gave it 1GB RAM, 30GB HDD, and 1 core which seems to be plenty.
The install process is straight forward…
Create a file, /etc/apt/sources.list.d.100-ubnt-list
deb http://www.ubnt.com/downloads/unifi/debian unifi5 ubiquiti
Go to https://unifi.example.com:8443 (See the bottom of this post for info on setting up a real certificate).
The first time you access it you get Wizard to set it up, after creating an account and such it will have you adopt the UniFi APs on your network. If they’re plugged in it will find them automatically. It not only manages APs but also manages UniFi branded routers, switches, cameras, VoIP devices, etc.
I can see how it would help manage a fleet of wireless equipment across multiple sites. You can see all the devices connected, the AP they’re connected to, signal strength, connection speed, data they’ve used, how they’re authorized to be on the network, VLANs, etc. I’ve hidden a lot of columns in the screenshot below but it gives you an idea of the data you can get on wireless clients.
The UniFi also keeps track of every wireless AP that it has seen. My neighbors seem to have a lot of HP Smart Printers and TVs that need to waste RF spectrum running their own APs for some reason. Cars Have APs? It looks like a lot of cars have their own APs now days? At least I’m guessing these MitsumiE APs are automobiles that have driven by my house.
UniFi Android / iOS App
The Android app is is just as capable (and I presume it is on iPhone as well), I didn’t do a thorough comb through but at a quick glance it appears every screen and configuration setting in the web interface is available in the Android App.
AP Models Comparison
The APs perform well. Since I installed the UniFi we have not had a single wireless connection drop, even if I put the AP power settings at their lowest it has better range than my previous AP. I also setup both APs and my devices had no trouble roaming between the two APs as needed while maintaining connections.
The three main models are:
UAP-AC-LITE – 2×2 MIMO on both bands (budget)
UAP-AC-LR – 3×3 MIMO on 2.4, 2×2 MIMO on 5GHz (middle)
UAP-AC-PRO – 3×3 MIMO on both bands (fastest)
Does 3×3 MIMO make a difference for 2×2 clients? You might get better reception, but probably not a noticeable difference. However, if you do have 3×3 capable clients you should see a benefit going to a 3×3 AP.
UAC-AP-PRO vs UAC-AP-LITE Performance and Coverage with 2×2 Clients
Most wireless clients are only 2×2 MIMO these days, and even though I tend to run the latest hardware I only have 2×2 devices which can connect at a maximum speed of 866.7Mbps. A 3×3 MIMO AP can improve performance of 2×2 MIMO clients because the extra antenna might provide a better signal. That’s the theory anyway.
I can’t really tell a difference between the two routers in my house, in the Android App Wireless Test I get better uploads speeds on the Pro than I do the Lite, which might be due to it’s extra antenna but I don’t see that performance benefit on our laptops when transferring files back and forth between them and my FreeNAS unit.
I do think I get slightly better upload speeds on the Pro model when I’m far away from the AP. This may be due to the extra antenna or it could just be subjective.
As far as real life performance on 5GHz setting the channel width to 80Mhz I get about 50-60Mb/s down and 30-40Mb/s up pretty consistently throughout the house, and that’s with multiple wireless clients connected and a pretty saturated RF spectrum. Here’s an RF Scan at my house… there’s really not a single empty channel even on 5Ghz.
UniFi Managed Switches
Ubiquiti also sells managed switches, ranging from 8 to 48 ports with a variety of PoE options. I’ve been wanting try out managed switches so I picked up their small 8-port. Since I’m running these at home low noise is extremely important. The two switches that fit the bill are the 8-port US-8-60W (with 4 PoE ports) and the 24-port US-24 (without PoE), both of these models are fanless and silent.
The US-24 doesn’t have PoE on it. The US-8-60W has four 802.3af PoE ports. I should note that this switch cannot do passive PoE so it won’t be able to power UniFi’s passive PoE equipment (such as the UAP-AC-LITE).
There are two banks of LEDs, top row is only for the four PoE ports on the right and light up orange if PoE is activated. The bottom row lights up green on gigabit links and orange for 100Mbps links. There’s also a blue/white LED on the far left front of the router that’s off. I do not like blue or white LEDs. Fortunately as soon as I provisioned it the UniFi Controller automatically turned it off based on my site preferences.
After getting a quick primer from a Network Engineer on how VLAN tagging works I decided to start VLAN tagging my network.
Under the UniFi Controller you can setup your VLANs, I programmed all of mine in above. Something that is a bit confusing is there are two Network Purpose types that support VLAN tagging, “Corporate” and “VLAN-Only”. There is no difference between the two, unless you are using the USG (UniFi Security Gateway), which can run a DHCP server for each “Corporate” network type. Since I’m using pfSense instead of the USG I setup mine as vlan-only.
Then it’s fairly trivial to manage the ports, setting up trunking and access ports for certain VLANs. In my case port 2 is my trunk port and goes to my pfSense router. I also ran my Northland Cable connection through the switch so I could get some bandwidth insights.
As always, the UniFi Controller provides some pretty neat insights, it picked up devices not only connected to it but also found devices connected to other switches (notice most of the devices below were found on port 2 which connects up to my VMware vSwitch).
And UniFi provides great statistics and insights into traffic flow on the switch.
Appendix A: Setting an SSL Certificate on the UniFi Controller.
By default the Unifi controller runs on port 8443 with a self-signed SSL certificate. It is ridiculously difficult to set a custom cert… I know how to work with Java keystores but I just couldn’t get the ace.jar Java cert importer to accept my intranet cert. Then I read the CA cert had to be in DER format which also didn’t work…. arrgh. Suddenly it hit me that setting up certs on nginx is easy, it would be much simpler to set up an SSL certificate on an nginx reverse proxy on port 443. I want the UniFi Controller listening on 443 anyway, and even better, I don’t have to touch any UniFi configuration files or certs.
If you’re running an internal CA like I am you can just generate an internal Cert, or if you need a public cert Let’s Encrypt should work just as well. Here’s an example of generating one from FreeNAS.
Export the certificate and key and save them to /etc/nginx/cert.crt and /etc/nginx/cert.key. The configuration is a pretty standard nginx reverse proxy, the only issue I initially ran into was the UniFi controller reported a “WebSocket connection error” warning, so I enabled nginx’s proxy support for WebSockets (which the configuration below takes care of). Other than that it’s a straight forward reverse proxy.
When you request a website, say, b3n.org, your computer needs the IP address. So it sends out packets through your router/firewall, your modem, and out to your ISPs DNS Servers. Your ISP’s DNS server will probably have it cached, if not it queries the authoritative (starting with the Root Name Servers) recursively to find out what the authoritative DNS servers are and then queries those DNS servers. It gets the IP address, and sends it back to your computer. Your computer can then query the server IP for b3n.org. Any latency along this process will result in delays. If you ever type in a url in the address bar and nothing happens for a few hundred milliseconds and then suddenly the website starts to load this is likely the problem.
Is Your DNS Hijacked by Your ISP?
It’s pretty easy for ISPs to hijack DNS queries. A small number of ISPs (Comcast, CenturyLink, Time Warner, Cox, Rogers, Charter, Verizon, Sprint, T-Mobile, Frontier, etc.) have been caught doing exactly that. Want to know why? Advertising revenue. When you misspell a domain some ISPs, instead of returning an NXDOMAIN (does not exist) like any RFC compliant DNS server it will resolve the domain anyway, point it at a page they control, and advertise to you! This is a really bad idea. But there is a way to prevent your ISP from doing this…
Using Google’s Nameservers
If you’re not tech savvy using 220.127.116.11 and 18.104.22.168 is probably better than your ISPs nameservers. It won’t hurt, and will probably help, but it may not help… it’s very trivial for an ISP to route those IPs to their own servers and some do.
Even if your ISP is pure goodness and would never do that, someone could setup a rogue DNS server posing as theirs and intercept all your DNS traffic.
The only solution is to query the Root name servers for authoritative DNS servers and use DNSSEC. Cut out any 3rd party DNS provider and run your own DNS server locally.
Setup an Unbound Server on pfSense
Unbound is a high performance caching DNS server. Unbound queries recursively authoritative DNS servers directly, completely bypassing your ISP. It uses DNSSEC to make sure your queries haven’t been tampered with. And best of all, it caches DNS results locally (like your ISP would) but since it’s on your own network, the cached DNS queries are local!
You can setup a local FreeBSD server and run Unbound on it, but if you’re already using a router like pfSense or OPNsense you can setup an Unbound server in a few clicks.
Open up pfSense, first make sure the forwarder under Services, DNS Forwarder, is disabled. Slowness warning: if you are running a low query lookup network such as on your home network having the forwarder disabled may cause lookups to be slower because you’re having to traverse the DNS servers regularly to get results… this can sometimes take a second or two and result in DNS timeouts while it’s trying to traverse the DNS nameservers. If you find that unbound performance is slow I’d suggest turning on forwarding mode which will use the DNS servers specified in pfSense under system, general setup. In this case I’d recommend pointing them at 22.214.171.124 and 126.96.36.199. If you run with forwarding enabled you should verify that your ISP is not hijacking your DNS results, if they are you should switch ISPs.
Go to Services, DNS Resolver.
Enable the DNS Resolver
Select the Network interfaces that you want Unbound to listen on (do not select ALL, you’ll definitly want to select LAN).
System Domain Local Zone Type: Transparent
Enable DNSSEC Support
Do NOT enable Forwarding Mode
You can also choose to register DHCP addresses in the DNS Resolver which is very handy if you’re using pfSense to manage DHCP.
Under System, General Setup
Make sure all DNS Server fields are empty. DNS Server Override and
Disable DNS Forwarder should be unchecked.
Finally, Under Services, DHCP Server, set your DNS Server to your pfSense’s LAN IP. As your DHCP clients renew their lease they’ll start using pfSense for DNS.
As far as performance if you have low latency to your ISPs DNS you probably won’t notice anything. But if you’re on a high latency connection with 70ms pings like I am, this makes a big difference.
Amazon Lightsail has entered the VPS market, competing directly with DigitalOcean and Vultr. I for one welcome more competition in the $5 cloud server space. I wanted to see how they perform so I spun up 24 cloud servers, 8 for each provider and ran some benchmarks.
$5 Cloud Server Providers Compared
DigitalOcean, Vultr, and Amazon Lightsail offer more expensive plans, but this post is dealing with the low-end $5 plans. Here’s how they compare:
1 CPU Core
20GB HDD (extra block storage @ $0.10/GB/month)
1TB Bandwidth ($0.02/GB overage fee in U.S.).
Best team management – DigitalOcean lets you create multiple-teams and you can add and remove users from those teams.
Ubuntu, FreeBSD, Fedora, Debian, CoreOS, CentOS
1 CPU Core
15GB HDD (extra block storage @ $0.10/GB/month)
1TB Bandwidth ($0.02/GB overage fee in U.S.)
Account sharing – allows you to setup multi-user access.
Floating IPs (currently can’t setup automatically, requires support setup)
Ubuntu, FreeBSD, Fedora, Debian, CoreOS, CentOS, Windows, or any OS with your Custom ISO.
1 CPU Core
20GB HDD (block storage not available)
1TB Bandwidth ($0.09/GB overage fee in U.S.)
3 Free DNS zones (redundancy across TLDs as well).
Amazon Linux or Ubuntu
All three providers have multiple geographic locations worldwide. Vultr has the most locations in the United States, while Amazon has more geographic locations in the world (although only Virginia is available to LightSail at this point in time).
Vultr Global Locations
Amazon Lightsail Global Infrastructure
All providers offer an API. In practice DigitalOcean has been around the longest and thus is more likely to be supported in automation tools (such as Ansible). I expect support for the other APIs to catch up soon.
CPU Test – Calculating Primes
Number of seconds needed to compute prime numbers. On the CPU test Amazon Lightsail consistently outperformed, with Vultr coming in second and DigitalOcean last. CPU1 and CPU2 are 1 and 2 threads respectively calculating primes up to 10,000. CPU4 is a 4-threaded test calculating primes up to 100,000.
Lower is better.
Lower is better.
(I accidentally omitted the memory test from my parser script and didn’t realize it until the last test ran, so this is the average of 4 results per provider)
OLTP (Online transaction processing)
Higher is better.
The OLTP load test simulates a transactional database, in general it measures latency on random inserts, updates, and reads against a MariaDB database. CPU, memory, and storage latency all can effect performance so it’s a good all around indicator. This test measures the number of transactions per second. In this area Vultr outperformed DigitalOcean and Amazon Lightsail in 2 and 4 thread tests, while Lightsail took the lead in the 8-thread test. I don’t know why Lightsail started to perform better under multi-threaded tests, however, my guess is that while Lightsail doesn’t offer the fastest single-threaded storage IOPS it may have better multi-threaded IOPS–but I can’t say for sure without doing some different kinds of tests. DigitalOcean performed the worst in all tests–probably due to it’s slower CPU and memory speed.
Higher is better.
Transactions per second. In random IOPS Vultr provided the best consistent performance, DigitalOcean comes in second place with wide variance, and Lightsail comes in last, but it was by far the most predictable.
Sequential Reads / Writes / Re-writes
Higher is better.
This simply measures sequential read/write speeds on the hard drive. Vultr offers the most consistent high performance, DigitalOcean is all over the place but generally better than Lightsail which comes in last.
Latency (Ping ms) U.S. Locations
Lower is better.
The U.S. latency is all close enough that it doesn’t matter.
Latency (Ping ms) Worldwide Locations
Lower is better.
International Latency, again the results are pretty close.
Download Speed Tests from U.S. Locations
Higher is better.
Downloading data from various locations. It’s really hard to conclude any meaningful analysis from this… the faster peering in New York probably has to do with DigitalOcean and Vultr being located in New York vs Lightsail’s location in Virginia.
Upload Speed Tests to U.S. Locations
Higher is better.
Due to the similarities in the test results I think the bandwidth constraints are on the other side, or at peering.
Download Speed Tests from Worldwide Locations
Higher is better.
Who knows what one could conclude from this, it seems like various providers have different quality peering to different worldwide locations, but there are so many variables it’s hard to say.
Upload Speed Tests to Worldwide Locations
Higher is better.
Similar groupings for the most part.
I spun up 24 x $5 servers, 8 for each VPS provider. I spun up 12 servers yesterday and ran tests, destroyed the VMs, then created 12 new servers today and repeated the tests. All tests were run in the Eastern United States. I chose that region because the only location available currently in Amazon Lightsail is Virginia, so to get as close as I could I deployed Vultr and DigitalOcean servers out of their New York (and New Jersey) data centers. New York is a great place to put a server if you’re trying to provide low latency to the major populations in the United States and Europe without using a CDN.
If the provider had multiple data centers in a region I tried to spread them out.
DigitalOcean – I deployed 4 servers in NYC3, 2 in NYC2, and 2 in NYC1.
Vultr – All 8 servers deployed in their New Jersey data center.
Amazon Lightsail. Deployed in their Virginia location, 2 in each of their four AWS high availability zones.
All the tests I ran are relatively short duration, I did not benchmark sustained loads which may produce different results. My general use case is a web-server or small build server with intermittent workloads. I often spin up servers for a few hours or days and then destroy them once they’re done with their tasks.
The testing scripts I used are available in my GitHub meta-vps-bench repository. The testing scripts are very rudimentary and could be improved. It runs sysbench and speedtest benchmarks. The following commands were run on each server as root:
I tried to stagger starting the tests so that multiple speedtests against the same location had a low risk of occurring at the same time… but it may not always work out that way. I ran all tests twice per server which gives 48 total results (16 for each provider).
This script is for testing. I do NOT recommend running this on production servers.
Looking for a Christmas gift idea for your computer geek? Here’s a short gift guide with a few ideas I think would make great gifts. Unlike a lot of other top gift idea lists written by non-tech people just to make a sale, I’m actually a developer and these are the sort of things that I would enjoy (in fact most of them I own or at the very least had a chance to play with).
Here’s some gifts your geek, hacker, developer, programmer, tech enthusiast, etc. may enjoy:
WiFi RGB LED Light
MagicLight WiFi Smart LED Light Bulb ($). This looks like a normal light-bulb, but it can connect to your WiFi network and be controlled by your SmartPhone, or through home automation software, or Python scripts. This Bulb can change to any color. You can send it HTML Hex Color Codes! If you live up in North Idaho like I do you can program your light to gradually get brighter in the morning to wake you up naturally in the months where the Sun doesn’t rise until late in the day. Or program it to redshift in the evenings before bedtime so the blue light isn’t messing with your circadian rhythm. Or have it turn red as a warning when you’ve left the garage door open after dark! Put a few outside on your house and set them to be certain colors during the Holidays (Red & Green at Christmas, Orange during Halloween, Red, White, and Blue for Independence Day).
Raspberry Pi 3
Raspberry Pi 3 Starter Kit ($$). Every technology enthusiast would enjoy a Raspberry Pi. There are so many projects you could do… build your own weather station, automatic sprinkler system, home automation server, arcade, even a small computer, tiny server, thermometer, etc.
Python Programming for Beginners ($) by Jason Cannon. Yes, the name comes from Monty Python. Python is becoming a well loved language and is growing fast, and is fun to learn and practical. I have been seeing a lot of increase of this language lately. This is one of the best programming languages to learn, even if you’re not a programmer. This book is perfect for someone new to Python or even for someone starting out learning to code for the first time.
MasterKeys Pro Mechanical Keyboard. ($$$). (This is the latest model, I use an older version of this keyboard at work). If your hacker is on the young side there’s a good chance he has never experienced the joy of typing on a mechanical keyboard and may not even know they exist! Does your keyboard let you press every single key on the keyboard simultaneously and they ALL register? This keyboard does. This keyboard has 3 switch options. Cherry MX Red, Brown, or Blue. I linked to the Cherry MX Brown version but there are several different switch types: Cherry MX Reds have no tactile bump, they are linear so great for FPS or RTS gaming where speed matters. Cherry MX Blues provide an audible click and a tactile bump and are great for typing (unless noise is a concern), Cherry MX Browns provide subtle tactile feedback with no audible click making it a great all-purpose keyboard. The MX browns are my favorite Cherry switch and it’s what I recommend starting with for most people if you don’t know what you want.
I should mention, that by “no audible click” I mean no added click noise. Kris tells me the “silent” Browns and Reds are loud compared to a typical keyboard. The Blues are even louder.
Civilization VI ($). This game is one of the longest running series, and in my opinion one of the best turn-based strategy games on the market. Your gamer geek can play single-player, or online with friends. Starting out with a single Settler and building cities… what I like about Civilization is the unique ways to win. Most games are about World Domination through force. But in Civilization that is just one of many ways to win. In addition to Domination you can obtain Victory through Culture, Religion, or Science.
Chicory Coffee & Beignet Donuts
Chicory Coffee & Beignet Donuts ($). If you are ever visiting New Orleans you should stop by the Cafe Du Monde (open 24/7) for some Beignet Donuts and Café au lait. But the next best thing is giving the gift of coffee and donuts for those early mornings or late night programming sessions. This is one of my favorite coffee flavors, it has a unique taste and everyone I’ve brewed it for loves it.
YubiKey Neo ($). If your hacker is concerned about security you might consider getting him the YubiKey Neo. It’s a 2nd Factor Authentication device which works with Android (using NFC), Linux, Mac, and Windows. Everyone should be locking down their accounts (Email, Github, etc.) with a Yubikey. Yubico is one of the more reputable companies. Last year a security bug was discovered in the OpenPGP applet and they offered free replacement (including free shipping) for all the affected devices. Their software to work with they key is open source on GitHub. YubiKey supports such a large variety of MFA authentication methods including FIDO U2F, HOTP, TOPT, Yubico OTP, PIV-Compliant SmartCard, HMAC-SHA1 challenge response, etc. It’s really the only authentication device you need. I can authenticate with just about any service and protocol using a single YubiKey.
ESV MacArthur Study Bible Personal Size ($). Of course, it would be remiss of me not to include a gift that has to do with the very reason we celebrate Christmas. From the Creation and Fall of man, the Son of God coming to earth to die on the cross to take the penalty for our sins, and raising from the dead so that anyone who believes in Him will have eternal life. I received this as a gift a few years ago and it’s to date my favorite Bible. I don’t think you’ll find a higher quality Bible at this price point, it’s even Smyth Sewn which surprised me! MacArthur has some of the most scholarly and practical (easy enough for me to understand) Study Bible notes on the market today. His notes are extensive enough to be helpful, yet the personal edition is still small enough to be portable.
Well, that’s my guide for this year. Wishing everyone a Happy Thanksgiving and a Merry Christmas.