Eli Playing Chess

Magnus Carlsen is the best chess player in the world.  And I’m going to beat him.  – Elijah Bryan

Eli playing chess

Eli playing Chess

Eli practicing Chess speech

 

Is Your WiFi Unstable?

The most Frequently Asked Question from my Family, Friends, and FOAFs…

Laptop Buyer: What Kind of Laptop Should I buy?
Ben: Get one with an Intel Wireless card

WiFi Cards Matter

wifi_cardThe first piece of advice I have is make sure your wireless card is made by Intel.  Do not get anything else.  You might see other tempting wireless cards for so much less by Dell, Broadcom, Ralink, Killer, Realtek, etc.  These WiFi cards might work with most WiFi hotspots, they might work most of the time, but don’t get them.  The problem is they aren’t robust.  I’ve seen them drop connections randomly, not be able to connect to certain wireless APs, drop out the signal when the Microwave is running, etc.  At best case it works fine but later on a driver update might make it worse.  It is not worth saving a few bucks to deal with these issues.  Pay extra for an Intel branded WiFi Card.  It might cost you $20 more and save you months of frustration.  You’ll thank me later when your card isn’t disconnecting randomly.

This brings me to the 2nd most Frequently Asked Question….

My Wireless Keeps Disconnecting.  Help!

Laptop buyer: So, my wireless signal keeps dropping out.
Ben: Did you get an Intel Card like I told you?
Laptop buyer: No….
Ben: Were you trying to save money and went too cheap?
Laptop buyer: Yes…..

And the 3rd most Frequently Asked Question….

Can You Fix My WiFi Stability?

If Eli can fix it, you can fix it.

replacing_wireless

You will need to swap out your WiFi card.

If you’re in the situation where you bought a laptop with a flaky WiFi card, it’s easy to fix!  Grab an inexpensive Intel 7260 WiFi Card from Amazon.  On most laptops the WiFi card is easily accessible from behind the back cover, usually it’s not more work than a memory upgrade.  Unplug the antenna connectors from your unstable wireless card, pop it out, and put the new card in and hook it up.  Your WiFi connections will now be robust.

Back Story

I don’t say this because I’m an Intel fan.  I just want things to work.  Every couple of years I give another brand a try just to make sure my “only Intel” advice is relevant.  I’ve had the same experience with non-Intel brands the last 15 years!

Last year I decided to buy a cheap laptop to watch movies on (we don’t have a TV) and it came with a Dell DW 1704 / Broadcom 4314 Wireless Card.  I bought it just to see if things had gotten better.  They haven’t.  This wireless NIC can’t stream a full length movie from my media server without losing the wireless signal several times.

And it’s not just me, earlier this year several of my colleagues bought Dell XPS laptops with Killer Branded WiFi cards.  They just don’t work reliably in scenarios that Intel chips do.  In their case they couldn’t connect to several APs.  In my case the connection would drop several times a day.  This was both in Windows 10 and in Linux.  And yes, I tried disabling power saving mode on the WiFi adapter.

I’ve had friends and family not be able to even connect to certain APs at all until they swapped out their Broadcom, Killer, or Ralinks for an Intel card.  Now, you might get lucky and find another brand that works.  To me it’s not worth the hassle.

The next time you buy a computer, get one with an Intel WiFi card.

 

 

499th Reformation Day Trivia

This October 31st, 2016, is the 499th Reformation Day, alongside All Hallow’s Eve (or Halloween).

Jack-O-Lantern
Jack-o’-Lantern Eli and I made last year

We owe much to the reformers.  Many were involved, most notably Ulrich Zwingli and later John Calvin, but Martin Luther is said to have started the Reformation by posting the 95 Theses to the door of the Castle Church in Wittenberg.  He intended to start a discussion, but the Church did not take it too kindly.

Martin Luther believed through his study of Scripture, especially Romans, that forgiveness of sins was a gift that God alone could give, and those that taught people could receive forgiveness from the Pope through indulgences (which were used to fund massive projects for the Roman Catholic Church) were in error.

Why does the pope, whose wealth today is greater than the wealth of the richest Crassus, build the basilica of St. Peter with the money of poor believers rather than with his own money?
— Martin Luther, The 95 Theses

Luther hoped he could change (reform) the Roman Catholic Church, but instead he was labeled a heretic and excommunicated.  The Church wrote him a letter saying he did not have their permission to go to heaven which Luther later burned publicly.

I admire Luther the most for the doctrine of Sola Scriptura–that the Scripture is the supreme authority.  It does not mean we don’t have others in authority over us–but all authorities are subject and corrected by the Word of God.

Unless I am convinced by the testimony of the Holy Scriptures or by evident reason-for I can believe neither pope nor councils alone, as it is clear that they have erred repeatedly and contradicted themselves-I consider myself convicted by the testimony of Holy Scripture, which is my basis; my conscience is captive to the Word of God. Thus I cannot and will not recant, because acting against one’s conscience is neither safe nor sound. God help me. Amen.
— Martin Luther, at the Diet of Worms

Here’s some Reformation Day Trivia… feel free to print it out or borrow from it for your Reformation Day Party.

Reformation Trivia

Martin Luther1. The Reformation was started by:

A. Martin Luther
B. Martin Luther King, Jr.
C. John Calvin
D. John Bunyan

2. The Reformation started in ___

A. 1571
B. 1517
C. 1715
D. 1751

3. The reformation is said to start when Martin Luther published

A. The Diet of Worms
B. 95 Theses
C. A Treatise on Good Works
D. An Open Letter to Christian Nobility

4. Martin Luther was an….

A. Austrian Monk
B. Augsburg Monk
C. Austere Monk
D. Augustinian Monk

5. In 1521, Luther was summoned before ____

A. The Diet of Worms
B. Johann Eck
C. Rome
D. Frederick of Saxony

6. Emperor Charles V Commanded Luther to appear in order to….

A. Defend and Debate his ideas.
B. Murder him during passage to the trial
C. Force him to recant
D. Repent of his Heresy

7. What belief was NOT part of Luther’s theology?

A. The Pope has no special relationship to God
B. Scripture is the sole source of authority for Christians.
C. The bread and wine used during Communion only symbolically represents the flesh and the blood of Christ.
D. Salvation is by Faith Alone.

8. The 95 theses were widely distributed thanks to….

A. The Internet
B. The Printing Press
C. The Telegram
D. Lots of Facebook Likes

9. 95 is a prime number.

A. True
B. False

10. What is NOT true about the 95 theses?

A. The 95 Theses are largely concerned with the sale of indulgences
B. The 95 Theses was originally written in German
C. The 95 Theses was a point of argument in a scholarly debate
D. 95 Theses questioned the Pope’s motives
E. 95 Theses were widely distributed thanks to the printing press

11. The NLCS and ALCS series always occur on Reformation Month. They are…

A. Rivaling Lutheran Synods
B. Where the Arminian Lutherean Christian Synergists and the National Lutheran Calvinist Soteriology groups get together to debate theology.
C. A conference of two groups: the Advent Libertarian Catholic Supralapsarianists and the Nicene Liturgist Creed Semi-Pelagianists where they get together and enjoy food, fellowship, and try to understand each other and resolve their differences.
D. Have nothing to do with the Reformation

12. What’s another name for the 95 theses?

A. Disputation Against Scholastic Theology
B. Disputation on the Power of Indulgences
C. On the Papacy in Rome Against The Most Celebrated Romanist in Leipzig
D. The Misuse of the Mass

Indulgences13. What was an Indulgence?

A. A way to reduce the duration or amount of punishment one has to pay for their sins.
B. Self-gratification through wealth, food, sex, or social status.
C. The Catholic luxurious churches
D. Over-tolerance of others that makes Christians passive toward sin

14. Fill in the Blank. “As soon as the coin in the coffer rings, the soul from _______ springs” – Johann Tetzel

A. Hades
B. Purgatory
C. Hell
D. The Nether

15. Luther was excommunicated by the Pope for:

A. Marrying a Nun which was not allowed for monks (or nuns).
B. Calling the Pope an “Antichrist”
C. False Teaching
D. Condemning Infant Baptism

16. In 1515, Pope Leo X granted a plenary indulgence which would:

A. Allow people to suffer less in Hades who bought it
B. Pay for the emperor to send runners to the mountain to get snow for ice cream
C. Reduce the number of worms people have to eat in Hell
D. Finance the construction of St. Peter’s Basilica in Rome

17. At Leipzig, Johann Tetzel was asked by a nobleman if he could buy an indulgence for a future sin. Tetzel agreed as long as the nobleman paid at once. What pre-paid sin did that nobleman later commit?

A. Murder
B. Adultery
C. Eating worms
D. When Tetzel left Leipzig the nobleman attacked him along the way, gave him a beating, and sent him back to Leipzig empty handed.

18. Which of the following is NOT one of the 5 Solas? (pick two)

A. Sola scriptura – Scripture Alone
B. Sola ecclesia – Church Alone
C. Sola fide – Faith Alone
D. Sola gratia – Grace Alone
E. Sola caritas – Love Alone
F. Solus Christus – Christ Alone
G. Soli Deo gloria – Glory to God Alone

19. Which Famous Hymn did Luther write?

A. A Mighty Fortress is Our God
B. Amazon Grace
C. How Great Thou Art
D. When I survey the Wondrous Cross

20. When Luther Went into Hiding, how long did it take him to translate the NT into German?

A. 10 days
B. 10 weeks
C. 10 months
D. 10 years

21. What is something Luther loved to eat?

A. A Diet of Worms
B. Waldorf Salad
C. Meat Pies
D. Jibaritos

22. Who did Martin Luther Marry?

A. Sarah Pierpont
B. Katharina von Bora
C. Idelette Storder de Bure
D. Fanny Crosby

23. What Reformation Anniversary is it?

A. 50th
B. 99th
C. 499th
D. 500th

24. Was Luther Poor or Rich?

A. Poor as a youth and rich later in life
B. Poor later in life and rich as a youth
C. Poor
D. Rich

25. How Old was Luther when he died?

A. 59
B. 60
C. 61
D. 62

26. Which Book is NOT a book Luther tried to remove from the Canon?

A. Philemon
B. Hebrews
C. James
D. Jude
E. Revelation

27. Name all 95 theses….

(answer on the door)

For by grace you have been saved through faith. And this is not your own doing; it is the gift of God, not a result of works, so that no one may boast.
— Ephesians 2:8-9, ESV

Picture of some trees and mountains on Reformation Day

Answers

  1. A. Martin Luther
  2. B. 1517
  3. B. 95 Theses
  4. D. Augustinian Monk
  5. A. The Diet of Worms
  6. A. Defend and Debate his ideas.
  7. C. The bread and wine used during Communion only symboically represents the flesh and blood of Christ.
  8. B. The Printing Press.
  9. B. False
  10. B. The 95 Theses was originally written in German
  11. D. Have nothing to do with the Reformation.
  12. B. Disputation on the Power of Indulgences
  13. A way to reduce the duration or amount of punishment one has to pay for their sins.
  14. B. Purgatory
  15. C. False Teaching
  16. D. Finance the constructoion of St Peter’s Basilica in Rome.
  17. D. When Tetzel left Leipzig the nobleman attacked him along the way, gave him a beating, and sent him back to Leipzig empty handed.
  18. B. Sola Ecllesia and E. Love Alone
  19. A. A Mighty Fotress is Our God.
  20. B. 10 weeks
  21. C. Meat Pies
  22. B. Katharina von Bora
  23. 499th (in 2016)
  24. A. Poor as a youth and rich later in life.
  25. D. 62
  26. A. Philemon
  27. http://www.biblestudytools.com/history/creeds-confessions/luther-95-theses.html

 

Creative Commons LicenseThis post is licensed under a Creative Commons Attribution 4.0 International License.

 

Ben’s Phone Guide (2016 edition)

Phones depreciate in value fast, their useful life is less than their lifespan.  Not because old phones don’t work anymore.  But because manufacturers stop providing security updates after about 3 years (at best!)

old_phone

What If I Told You a Hacker Can Take over Your Phone with One Text… And You Don’t Even Have to Open It?

You might be hacked now and not even know it.

Exploits like like this and like this are real.  Vulnerabilities have been found in the past and exploited.  They will be found in the future and exploited.  Some exploits require you to do nothing but receive (not even open, just receive) an SMS message and a hacker can do what he wants with your phone.  He can install malware, use your phone to launch a DDoS attack against Krebs on Security, he can spy on you (or your kids if your kids have phones) activating the camera and microphone at will listening in on your conversations and reading every message passing through the device.

The only protection against this is either (1) not have a phone (more secure), or (2) if you must have a phone, keep it up to date constantly (not as foolproof but would block all but the most sophisticated hackers).

One of the big problems with phones is security.  For iPhones you get your updates through Apple.  For Android things aren’t as clean.  The Android OS itself gets security updates, but then it has to trickle down through the manufacturer (who often doesn’t provide an update) and then the carrier you bought the phone from.

Calculating Remaining Life Before You Buy

To calculate the real cost of a phone, find out how long the manufacturer and carrier will support security updates for it.  Divide the cost of the phone by the number of months left for security updates and that’s cost of the phone.

monthly cost = cost of phone / remaining life in months

e.g.
cost of phone: $500
remaining life for security updates: 29 months
monthly-cost: $500/29 = $17.24

Oddly, the price of phones doesn’t usually drop that much after the 1st year even though they have lost 1/3rd of their useful life!

There Are Only Two Options

A lot of phone manufacturers / carriers don’t even provide updates to their phones.  They’re unsupported from the moment you bought them!

For the sake of security, I only recommend two phone manufacturers.  Google and Apple.  Both have a track record of providing timely security updates.  Google pushes out a security update every month and Apple doesn’t have a schedule but does a good job getting them out timely.  I also only recommend Apple with the caveat that you trust them because it is a proprietary closed source OS.  You are trusting them to do the right thing and have decent security.

Google Nexus Devices

Nexus 5X

Google stopped selling the Nexus, but they still have 2 years of updates left and are reasonably priced on Amazon.

Google Guarantees Security Patches on Nexus devices 3-years from the release date or at least 18 months from when the Google Store last sold the device (whichever is longer).

As of October 2016, here is the cost per month as I calculate it:

Nexus 5X – security updates until October 2018.  $332. – 16GB.
Ben’s cost over remaining life:  $332/24mos = $13.83/mo
Nexus 6P – security updates until October 2018. $450 – 32GB.
Ben’s cost over remaining life: $450/24mos = $18.75/mo

(If you get a Nexus, note that there are U.S. and International versions of the phone, if you live in the U.S. you’ll want the U.S. version).

Google has not committed to EOL dates on the Pixel line but if it’s similar to Nexus you’re looking at:

Google Pixel – $650 – 32GB – probably until October 2019
Ben’s cost over remaining life: 650/36mos = ~$18.05/mo

Google Pixel XL – $770 – 32GB – probably until October 2019
Ben’s cost over remaining life: 770/36mos ~$21.38/mo

Apple Devices

iPhone 7

iOS is closed source so I consider it less secure and less open than Android, but they do a pretty decent job at keeping hackers out.  Most compromises I hear about are through hooking your iPhone up to a service like iCloud and not the iPhone itself.  I used to use an iPhone, but at the time it was the best phone (better than Blackberry).  Now that we have Android I don’t see a huge need to use a closed proprietary system.  However, it’s always good to have competition.

Here’s a comparison of iPhone models currently getting security updates with a guess of (but not guaranteed) security updates for 3-years.

iPhone 7 Plus – probably until September 2019
Ben’s cost over remaining life: $650/35mos ~$18.57/mo

iPhone 7 Plus – probably until September 2019
Ben’s cost over remaining life: $650/35mos ~$22.00/mo

iPhone 6S – probably until September 2018
iPhone 6 / 6 Plus – probably until September 2017
iPhone 5S / 5C – probably until the next major iOS update

Where Not to Buy a Phone

Mobile carriers typically install a lot of battery sucking bloatware, which can’t be deleted, and often delay pushing out security updates by months, even years, leaving your phone vulnerable to hackers.  Not only that some of the extra software installed introduces vulnerabilities.

Also, phones bought from a mobile carrier are usually locked to that carrier so you can’t switch to someone else without purchasing a new phone.

Mobile Carriers

Having an unlocked phone I avoid the main carriers and instead use MVNOs (Mobile Virtual Network Operator).  These MVNOs use the same network that Verizon, AT&T, Sprint, and T-Mobile have, but most often for a better price.  For great service and prices I like Google Fi (Sprint & T-Mobile Network), Ting (Spring or T-Mobile), and TracFone (Verizon or AT&T) and there are plenty of other MVNO operators to choose from.  You can find one that offers the best plan for your situation.  Using TracFone (which is a pre-paid service) we pay less than $10/month for a voice/data/text plan for a Nexus 5X on Verizon’s network.

Don’t Save Money with a Used Phone

I used to buy used phones off eBay to save money but now I don’t think it’s a good idea with the recent USB firmware hacks and the amount of malware out there.  Used phones are a security risk–you have no idea if a used phone has been compromised, and if it’s been plugged into a compromised USB device that rewrote it’s firmware.  Physical security is paramount.  To be safe, I always buy my phones new.

Personal Data on Work Phones and Work Data on Personal Phones

Think carefully before using your personal phone for work.  If you connect your phone to work email it almost always gives your employer complete control of the device.  They can wipe your phone when you leave, track your location, install software on your phone, and have access to all your personal data.

And similarly, if you put your personal information or your personal email account on a work phone your employer has access to that data.

What Phone Do I Have?

Kris and I both use the Nexus 5X.  I’ve reviewed the Nexus 5X here.  I will likely replace them both when security updates go EOL which will likely be 2018.  Pixel phones are bit expensive so I’m hoping they release some new phones on the Nexus line again next year.

Phone Safety Tips

  1. Always use a phone that’s getting regular (monthly) security updates.  As soon as the phone goes out of support, get a new phone.
  2. Minimize the number of apps you install.  Limit yourself to the official Google Play Store or iOS store and avoid 3rd party stores like the Amazon Store where authors don’t do as good a job at keeping things updated.
  3. Favor installing well known apps with lots of downloads as they’re more likely to be reviewed and have better security practices.
  4. Uninstall apps that you don’t use.
  5. Always buy a new phone.
  6. Don’t use a phone at all.
  7. If you have a Samsung Note 7, you might want to return it before you catch on fire.

 

How to Encrypt Your Email

So, you want to hide your email from the NSA’s prying eyes?  It’s impossible… but here are some steps you can use to make it harder.

This isn’t theoretical.  The NSA has and does intercept this traffic.

Common Points of NSA Interception

The NSA has unlimited resources to compromise your communications.  You’re not going to stop them.  But that doesn’t mean it should be easy. Below are the easy points of NSA interception.  In this example of an email from Mom to Ben the NSA can intercept the email at Mom’s ISP, Mom’s email provider, Ben’s email provider, Ben’s ISP, and any internet hop in between.

no-encryption

 

I’m going to skip over a lot of important stuff, this guide is not intended for security experts or sysadmins of email systems and how to prevent downgrade attacks, etc.  This is meant to be a post about what the average American should do to protect their emails.

Step 1. Client to Server TLS Encryption

client-server-encryption
thunderbird_starttlsEnsure your email client (e.g. Thunderbird) or browser is using a TLS connection to the server.  If you’re using any major provider like Gmail, Office 365, etc. they will be enforcing TLS.  All email providers should be enforcing TLS so if yours is not that’s a good sign you should be switching.

chrome_httpsIf using webmail your browser should show https, if using Thunderbird you should be using STARTTLS for both inbound and outbound connections.

Note, the entire CA (Certificate Authority) system is broken, the NSA could generate a fraudulent certificate from an amicable CA and do a MITM attack and still intercept the email, but now they have to take some effort to do so.  The point is security comes in layers, and we need to start at the basics, we’ll get to more advanced security below.

Step 2. Make sure Your Email Provider is Encrypting Server to Server Traffic

server-serverIn 2013 Google was outraged after finding out the NSA was intercepting it’s server to server traffic.  As a result Google started encrypting all internal traffic between servers (Good for Google).  Most major internet providers provide server to server encryption.  But the problem is not all ISPs use encryption, so it doesn’t do much good if you send an email from a secure service like Gmail to a small-town ISP that has no security whatsoever.  Probably the best way to check is to enter in a recipients email address here: http://checktls.com/ and if their email provider’s MX server’s pass all the test they’re probably secure.

Step 3.  PGP Encrypt Your Emails

openpgp-4096

Now, the NSA can still potentially intercept your emails at rest through a court order, through PRISM, or through hacking into ISPs.  Your email should be encrypted not only in transit, but also at rest.  The best way to do that is to encrypt it using OpenPGP.  This means even if the NSA gets a hold of your email they can’t read it (at least not without spending some serious time and money).

PGP (Pretty Good Privacy) isn’t foolproof.  It doesn’t encrypt the metadata (the NSA can still see that you sent me an email, they can see when you sent it and where you were) but it does encrypt the content.

How go you get OpenPGP?  Right here:  http://openpgp.org/software/ It’s free, open source, and there are plugins for just about everything.  It works on Webmail, Thunderbird, Outlook, etc.  Check the link above for a complete list but here are two common options:

If you use Thunderbird I suggest Enigmail, and if you use Gmail with the webmail interface Mailvelope is a great plugin.

Here’s a very quick getting started guide for Mailvelope below.  If you’re not going to use Mailvelope the concept is pretty much the same nomatter what plugin you choose.  You’ll Generate a Public/Private Keypair, obtain the public key of the person you’re sending an email to, and send them en encrypted email.

How to Setup Mailvelope for Gmail and Chrome

Mailvelope IconHere’s a quick walk-through to set it up.  After installing the plugin you should see this icon on the top-right in Chrome.   Right-click on it and choose Options.

Next Generate a Key….

I should note that “Password” is traditionally called a Pass Phrase, it should be long, but you don’t ever want to forget it or you won’t be able to read any encrypted messages sent to you.  I strongly suggest writing it down and keeping it someplace safe.

mailvelope_key_generation

Now, to send an encrypted email to me, you’ll need to import my key.  Go to “Import Keys” and type in my email address and hit search.  You should click on the keyID: 13E708FC.  A key will pop up, click on it to import my key.

mailvelope_buttonNow, you can send me an encrypted email.  Go to compose a new email in Gmail.  You’ll notice a button in the compose menu.  Click the button.

Write me a message…

compose_email_to_ben

When you receive an encrypted email, it will look like this.  Click on it and enter your passphrase to decrypt.

decrypt

And there you have it.   I wouldn’t say this is foolproof…. it doesn’t protect against a lot of other attack vectors…

XKCD Comic
CC-By-NC 2.5 https://creativecommons.org/licenses/by-nc/2.5/

But I say if the NSA is going to intercept my communications it shouldn’t be easy.  I want them to spend some effort and money to do so.

For further reading I might suggest https://futureboy.us/pgp.html

 

ZFS Dataset Hierarchy | Data Hoarder Edition

OpenZFS LogoZFS is flexible and will let you name and organize datasets however you choose–but before you start building datasets there’s some ways to make management easier in the long term.  I’ve found the following convention works well for me.  It’s not “the” way by any means, but I hope you will find it helpful, I wish some tips like this had been written when I built my first storage system 4 years ago.

Here are my personal ZFS best practices and naming conventions to structure and manage ZFS data sets.

ZFS Pool Naming

I never give two zpools the same name even if they’re in different servers in case there is the off-chance that sometime down the road I’ll need to import two pools into the same system.  I generally like to name my zpool tank[n] where is an incremental number that’s unique across all my servers.

So if I have two servers, say stor1 and stor2 I might have two zpools :

stor1.b3n.org: tank1
stor2.b3n.org: tank2

Top Level ZFS Datasets for Simple Recursive Management

Create a top level dataset called ds[n] where n is unique number across all your pools just in case you ever have to bring two separate datasets onto the same zpool.  The reason I like to create one main top-level dataset is it makes it easy to manage high level tasks recursively on all sub-datasets (such as snapshots, replication, backups, etc.).  If you have more than a handful of datasets you really don’t want to be configuring replication on every single one individually.  So on my first server I have:

tank1/ds1

I usually mount tank/ds1 as readonly from my CrashPlan VM for backups.  You can configure snapshot tasks, replication tasks, backups, all at this top level and be done with it.

freenas_snapshot_pruning
ZFS snaps and pruning recursively managed at the top level dataset

Name ZFS Datasets for Replication

One of the reasons to have a top level dataset is if you’ll ever have two servers…

stor1.b3n.org
   | - tank1/ds1

stor2.b3n.org
   | - tank2/ds2

I replicate them to each other for backup.  Having that top level ds[n] dataset lets me manage ds1 (the primary dataset on the server) completely separately from the replicated dataset (ds2) on stor1.

stor1.b3n.org
 | - tank1/ds1
 | - tank2/ds2 (replicated)

stor2.b3n.org
 | - tank2/ds2
 | - tank1/ds1 (replicated)

Advice for Data Hoarders.  Overkill for the Rest of Us

supermicro_zfs

The ideal is we backup everything.  But in reality storage costs money, WAN bandwidth isn’t always available to backup everything remotely.  I like to structure my datasets such that I can manage them by importance.  So under the ds[n] dataset create sub-datasets.

stor1.b3n.org
 | - tank1/ds1/kirk – very important – family pictures, personal files
 | - tank1/ds1/spock – important – ripped media, ISO files, etc.
 | - tank1/ds1/redshirt – scratch data, tmp data, testing area
 | - tank1/ds1/archive – archived data
 | - tank1/ds1/backups – backups

Kirk – Very Important.  Family photos, home videos, journal, code, projects, scans, crypto-currency wallets, etc.  I like to keep four to five copies of this data using multiple backup methods and multiple locations.  It’s backed up to CrashPlan offsite, rsynced to a friend’s remote server, snapshots are replicated to a local ZFS server, plus an annual backup to a local hard drive for cold storage.  3 copies onsite, 2 copies offsite, 2 different file-system types (ZFS, XFS) and 3 different backup technologies (CrashPlan, Rsync, and  ZFS replication) .  I do not want to lose this data.

Multiple Backup Locations Across the World
Important data is backed up to multiple geographic locations

Spock – Important.  Important data that would be a pain to lose, might cost money to reproduce, but it isn’t catastrophic.  If I had to go a few weeks without it I’d be fine.  For example, rips of all my movies, downloaded Linux ISO files, Logos library and index, etc.  If I lost this data and the house burned down I might have to repurchase my movies and spend a few weeks ripping them again, but I can reproduce the data.  For this dataset I want at least 2 copies, everything is backed up offsite to CrashPlan and if I have the space local ZFS snapshots are replicated to a 2nd server giving me 3 copies.

redshirt_startrek

Redshirt – This is my expendable dataset.  This might be a staging area to store MakeMKV rips until they’re transcoded, I might do video editing here or test out VMs.  This data doesn’t get backed up… I may run snapshots with a short retention policy.  Losing this data would mean losing no more than a days worth of work.  I might also run zfs sync=disabled to get maximum performance here.  And typically I don’t do ZFS snapshot replication to a 2nd server.  In many cases it will make sense to pull this out from under the top level ds[n] dataset and have it be by itself.

Backups – Dataset contains backups of workstations, servers, cloud services–I may backup the backups to CrashPlan or some online service and usually that is sufficient as I already have multiple copies elsewhere.

Archive – This is data I no longer use regularly but don’t want to lose. Old school papers that I’ll probably never need again, backup images of old computers, etc.  I set set this dataset to compression=gzip9, and back it up to CrashPlan plus a local backup and try to have at least 3 copies.

Now, you don’t have to name the datasets Kirk, Spock, and Redshirt… but the idea is to identify importance so that you’re only managing a few datasets when configuring ZFS snapshots, replication, etc.  If you have unlimited cheap storage and bandwidth it may not worth it to do this–but it’s nice to have the option to prioritize.

Now… once I’ve established that hierarchy I start defining my datasets that actually store data which may look something like this:

stor1.b3n.org
| - tank1/ds1/kirk/photos
| - tank1/ds1/kirk/git
| - tank1/ds1/kirk/documents
| - tank1/ds1/kirk/vmware-kirk-nfs
| - tank1/ds1/spock/media
| - tank1/ds1/spock/vmware-spock-nfs
| - tank1/ds1/spock/vmware-iso
| - tank1/ds1/redshirt/raw-rips
| - tank1/ds1/redshirt/tmp
| - tank1/ds1/archive
| - tank1/ds1/archive/2000
| - tank1/ds1/archive/2001
| - tank1/ds1/archive/2002
| - tank1/ds1/backups
| - tank1/ds1/backups/incoming-rsync-backups
| - tank1/ds1/backups/windows
| - tank1/ds1/backups/windows-file-history

 

With this ZFS hierarchy I can manage everything at the top level of ds1 and just setup the same automatic snapshot, replication, and backups for everything.  Or if I need to be more precise I have the ability to handle Kirk, Spock, and Redshirt differently.

 

Intranet SSL Certificates Using Let’s Encrypt | DNS-01

Let's EncryptLet’s Encrypt is a great service offering the ability to generate free SSL certs.  The way it normally works is using http-01 challenge…  to respond to the Let’s Encrypt challenge the client (typically Certbot) puts an answer in the webroot.  Let’s Encrypt makes an http request and if it finds the response to the challenge it issues the cert.

Certbot

Certbot is great for public web-servers.

Generating Intranet SSL Certs Using DNS-01 Challenge

But, what if you’re generating an SSL certificate for a mail server, or mumble server, or anything but a webserver?  You don’t want to spin up a web-server just for certificate verification.

Or what if you’re trying to generate an SSL certificate for an intranet server  Many homelabs, organizations and businesses need publicly signed SSL certs on internal servers.  You may not even want external A records for these services, much less a web-server for validation.

ACME DNS Challenge

Fortunately, Let’s Encrypt introduced the DNS-01 challenge in January of 2016.  Now you can respond to a challenge by creating a TXT record in DNS.

ACME Let's Encrypt DNS-01 Challenge Diagram

 

Lukas Schauer wrote dehydrated (formerly letsencrypt.sh) which can be used to automate the process.  If you need to generate SSL certs for Windows I’ve added the ability to output to PFX / PKCFS 12 in my fork.

Here’s a quick guide on Ubuntu 16.04, but it should work on any Linux distribution (or even FreeBSD).

Install dehydrated / letsencrypt.sh

Hook for DNS-01 Challenge

At this point, you need to install a hook for your DNS provider.  If your DNS provider doesn’t have a hook available you can write one against their API, or switch to a provider that has one.

If you need to pick a new provider with a proper API my favorite DNS Providers are CloudFlare and Amazon Route53.  CloudFlare is what I use for b3n.org.  It gets consistently low latency lookup times according to SolveDNS, and it’s free (I only use CloudFlare for DNS, I don’t use their proxy caching service which can be annoying for visitors from some regions).  Route53 is one of the most advanced DNS providers.  It’s not free but usually ends up cheaper than most other options and is extremely robust.  The access control, APIs, and advanced routing work great.  I’m sure there are other great DNS providers but I haven’t tried them.

Here’s how to set up a CloudFlare hook as an example:

In letsencrypt-cloudflare-hook/hook.py change the top line to point at python3:

Config File

Edit the “/etc/dehydrated/config” file… add or uncomment the following lines:

domains.txt

Create an /etc/dehydrated/domains.txt file, something like this:

The first four lines will each generate their respective certificates, the last line creates a multi-domain or SAN (Subject Alternate Name) cert with multiple entries in a single SSL certificate.

Finally, run

The first time you run it, it should get the challenge from Let’s Encrypt, and provision a DNS TXT record with the response.  When validated the certs will be placed under the certs directory and from there you can distribute them to the appropriate applications.  The certificates will be valid for 90 days.

For subsequent runs letsencrypt.sh will check to see if the certificates have less than 30 days left and attempt to renew them.

Automate

It would be wise to run dehydrated -c from cron once or twice a day and let it renew certs as needed.

To deploy the certs to the respective servers I suggest using an IT Automation tool like Ansible, you can configure an ansible playbook to run from a daily cron job to copy updated certificates to remote servers and automatically reload services if the certificates have been updated.  Here’s an example of an Ansible Playbook which could be called daily to copy certs to all web-servers and reload nginx if the certs were updated or renewed: