This guide will install FreeNAS 10 (Corral) under VMware 6.5 ESXi, then via NFS share ZFS backed storage back to VMware. This is an update of my FreeNAS 9.10 on VMware 6.0 Guide.
“Hyperconverged” Design Overview
FreeNAS is installed as a Virtual Machine on the VMware Hypervisor. An LSI HBA in IT Mode is passed to FreeNAS via VT-d Passthrough. A ZFS pool is created on the disks attacked to the HBA. ZFS provides RAID-Z redundancy and an NFS dataset is then shared from FreeNAS and mounted from VMware which is used to provide storage for the remaining guests. Optionally containers and VM guests can run directly on FreeNAS itself using bhyve.
FreeNAS 10 (now called FreeNAS Corral) is a major rewrite over FreeNAS 9.10, the GUI has been overhauled, it has a CLI interface, and an API. I think the best feature is the bhyve hypervisor and docker support. To some degree for a single all-in-one hypervisor+NAS server you may not even need VMware and be able to get away with bhyve and docker.
Like anything new I advise caution against running it in a production environment. I do see quite a few rough edges and a few missing features that are available in FreeNAS 9.10. I imagine we’ll see frequent updates with polishing and features added. A good rule of thumb is to wait until TrueNAS hardware is shipping with the “Corral” version. I think this is the best release of FreeNAS yet, and it is going to be a great platform moving forward!
1. Get Hardware
This is based on my Supermicro X10SDV Build. For drives I used 4 x White Label NAS class HDDs (see ZFS Hard Drive Guide) and two Intel DC S3700s (similar models between S3500 and S3720 should be fine), which often show up for a decent price on Ebay. One SSD will be used to boot VMware and provide the initial data storage and the other used as a ZIL.
You will need an HBA to pass storage to the FreeNAS guest. I suggest the ServerRAID IBM M1015 flashed to IT mode, or you can usually find the LSI 9210-8i already flashed to IT mode for a decent price on eBay. You will also need a Mini-SAS to 4x SAS SATA Forward Breakout Cable.
2. IPMI Setup
Go ahead and plug in the network cables to the IPMI management port, as well as at least one of the normal ethernet ports.
This should work with just about any server class Supermicro board…. first download the Supermicro IPMIView tool (I just enter “Private” for the company). Once installed run “IPMIView20” from the Start Menu (you may need to run it as Administrator).
Scan for IPMI Devices… once it finds your Supermicro server select it and Save.
Login to IPMI using ADMIN / ADMIN (you’ll want to change that obviously).
KVM Console Tab…
Load the VMware ISO file to the Virtual DVD-ROM drive…
Download the VMware ESXi Free Hypervisor.
Select ISO file, Open Image, select the VMware ISO file which you can download here, and then hit “Plug In”
Hit Delete repeatedly…
Change the boot order, I made the ATEN Virtual CD/DVD the primary boot devices, and my Intel SSD DC S3700 that I’ll install VMware to secondary, and disabled everything else.
Save and Exit, and it should boot the VMware installer ISO.
3. Install VMware ESXi 6.5.0
Install to the Intel SSD Drive.
Once installation is complete “Plug Out” the Virtual ISO file before rebooting.
Once it comes up get the IP address (or set it if you want it to have a static IP which I highly recommend).
4. PCI Passthrough HBA
Go to that address in your browser (I suggest Chrome). Manage, Hardware, PCI Devices, select the LSI HBA card and Enable Passthrough.
5. Setup VMware Storage Network
In the examples below my LAN / VM Network is on 10.2.0.0/16 (255.255.0.0) and my Storage network is on 10.55.0.0/16. You may need to adjust for your network. My storage network is on VLAN 55.
I like to keep my Storage Network separate from my LAN / VM Network. So we’ll create a VM Storage Network portgroup with a VLAN ID of 55.
Networking, Port groups, Add Port Group
Add VM Storage Network with VLAN ID of 55.
(you can choose a different VLAN ID, my storage network is 10.55.0.0/16 so I use “55” to match the network so that I don’t have to remember what VLAN goes to what network, but it doesn’t have to match).
Add a second port group just like it called Storage Network with the same VLAN ID (55).
Add VMKernel NIC
Attach it to the Storage Network and give it an address of 10.55.0.4 with a netmask of 255.255.0.0
You should end up with this…
6. Create a FreeNAS Corral VM
Install it to the DC S3700 Datastore that VMware is installed on.
Add PCI Device and Select your LSI Card.
Add a second NIC for the VM Storage Network. You should have two NICS for FreeNAS, a VM Network and a VM Storage Network and you should set the Adapter Type to VMXNET 3 on both.
I usually give my FreeNAS VM 2 cores, if doing anything heavy (especially if you’ll be running docker images or bhyve under it you may want to increase that count). One rule with VMware is do not give VMs more cores than they need. I usually give each VM one core and only consider more if that particular VM needs more resources. This will reduce the risk of CPU co-stops from occurring. Gabrie van zanten’s How too many vCPUs can negatively affect performance is a good read.
ZFS needs memory. FreeNAS 10 needs 8GB memory minimum. Lock it.
Made the Hard Disk VMDK 16GB. There’s an issue with the VMware 6.5 SCSI controller on FreeBSD/FreeNAS. You’ll know it if you see an error like:
UNMAP failed. disabling BIO_DELETE
UNMAP CDB: 42 00 00 00 00 00 00 00 18 000.
CAM status: SCSI Status Error.
SCSI status: Check Condition.
SCSI sense: ILLEGAL REQUEST asc:26,0 (Invalid field in parameter list).
Command byte 0 is invalid.
Error 22, Unretryable error.
To prevent this, change the Virtual Device Node on the hard drive to SATA controller 0, and SCSI Controller 0 should be LSI Logic SAS
Add CD/DVD Drive, under CD/DVD Media hit Browse to upload and select the FreeNAS Corral ISO file which you can download from FreeNAS.
7. Install FreeNAS VM
Power on the VM…
Select the VMware disk to install to. I should note that if you create two VMDKs you can select them both at this screen and it will create a ZFS boot mirror, if you have an extra hard drive you can create another VMware data store there and put the 2nd vmdk there. This would provide some extra redundancy for the FreeNAS boot pool. In my case I know the DC S3700s are extremely reliable, and if I lost the FreeNAS OS I could just re-import the pool or failover to my secondary FreeNAS server.
Boot via BIOS.
Once FreeNAS is installed reboot and you should get the IP from DHCP on the console (once again I suggest setting this to a static IP).
If you hit that IP with a browser you should have a login screen!
8. Update and Reboot
Before doing anything…. System, Updates, Update and Reboot.
(Note: to get better insight into a task progress head over to the Console and type: task show).
9. Setup SSL Certificate
First, set your hostname, and also create a DNS entry pointing at the FreeNAS IP.
Create Internal CA
Untar the file and click the HobbitonCA.crt to install it, install it to the trusted Root Certificate Authorities. I should note that if someone were to compromise your CA or gain the key they could do a MITM attack on you forging SSL certificates for other sites.
Create a Certificate for FreeNAS
Listen on HTTP+HTTPS and select the Certificate. I also increase the token Lifetime since I religiously lock my workstation when I’m away.
And now SSL is Secured
10. Create Pool
Do you want Performance, Capacity, or Redundancy? Drag the white circle thing where you want on the triangle and FreeNAS will suggest a zpool layout. With 4 disks I chose “Optimal” and it suggested RAID-Z which is what I wanted. Be sure to add the other SSD as a SLOG / ZIL / LOG.
11. Create Users
It’s probably best not to be logging in as root all the time. Create some named users with Administrator access.
12. Create Top Level Dataset
I like to create a top level dataset with a unique name for each FreeNAS server, that way it’s easier to replicate datasets to my other FreeNAS servers and perform recursive tasks (such as snapshots, or replication) on that top level dataset without having to micromanage them. I know you can sometimes do recursive tasks on the entire pool, but oftentimes I want to exclude certain datasets from those tasks (such as if those datasets are being replicated from another server).
If you’d like to see more on my reasoning for using a top level dataset see my ZFS Dataset Hierarcy
Storage, tank3, Datasets, New…
13. Setup Samba
Services, Sharing, SMB, set the NetBIOS name and Workgroup and Enable.
Storage, SMB3, Share, to create a new dataset with a Samba Share. Be sure to set the ownership to a user.
14. Setup NFS Share for VMware
I believe at this time VMware and FreeNAS don’t work together on NFSv4, so best to stick to NFSv3 for now.
Mount NFS Store in VMware by going to Storage, Datastores, new datastore, Mount NFS datastore.
I setup automatic recursive snapshots on the top level dataset. I like to do pruning snapshots like this:
every 5 minutes -> keep for 2 hours
every hour -> keep for keep for 2 days
every day -> keep for 1 week
every week -> keep for 4 weeks
every 4 weeks -> keep for 12 weeks
And SAMBA has Previous Versions integration with ZFS Snapshots, this is great for letting users restore their own files.
16. ZFS Replication to Backup Server
Before putting anything into production setup automatic backups. Preferably one onsite and one offsite.
Peering, New FreeNAS, and enter the details for your secondary FreeNAS server.
Now you’ll see why I created a top level dataset under the pool….
Storage, Tank3, Replications, New, select the stor2.b3n.org Peer, source dataset is your top level dataset, tank3/ds4, and target dataset is tank4/ds4 on the backup FreeNAS server.
Compression should be FAST over a LAN or BEST over a low WAN.
Go to another menu option and then back to Storage, tank3, Replications, replication_ds4, and Start the replication and check back in a couple hours to make sure it’s working. My first replication attempt hung, so I canceled the task and started it again. I also found that adjusting the peer interval from 1 minute to 5 seconds under Peering may have helped.
16.1 Offsite Backups
It’s also a good idea to have Offsite backups, you could use S3, or a CrashPlan Docker Container, etc.
17. Setup Notifications
You want to be notified when something fails. FreeNAS can be configured to send an email or sent out Pushbullet notifications. Here’s how to setup Pushbullet.
Create or Login to your Pushbullet account. Settings, Account, Create an Access Token
Services, Alerts & Reporting, Add the access key (bottom right) and configure the alerts to send out via Pushbullet.
You can use the Pushbullet Chrome extension or Android/iOS apps to receive alerts.
18. bhyve VMs and Docker Containers under FreeNAS under VMware
Add another Port Group on your VM Network which allows Promiscuous mode, MAC address changes, and Forged transmits. You can connect FreeNAS and any VMs you really trust to this port group.
Power down and edit the FreeNAS VM. Change the VM Network to VM Network Promiscuous
Enable Nested Virtualization, under CPU, Hardware virtualization, [x] Expose hardware assisted virtualization to the guest OS.
After booting back up you should be able to create VMs and Docker Containers in FreeNAS under VMware.
Use at your own risk.
More topics may come later if I ever get around to it.