Introduction to Packer and Terraform with Ubuntu on vSphere

Contents

Introduction

Packer and Terraform are both tools created by the company Hashicorp , both of these tools aim to provide automation capabilities. Packer has the goal of creating machine images for multiple different platforms. For example, we can create AMIs for EC2 instances on AWS or VMDK/VMX files for VMware. The whole goal is to automate the ability to create a consistent image that can be used by a specific platform, whether it be AWS or VMware products such as vSphere.

Terraform focuses on the infrastructure itself. Based on a provided configuration file Terraform will be able to build up infrastructure, while also allowing for the ability to update what is running. For example, from an AMI image or a template on vSphere we can launch several of instances and configure them in a specific way.

Goal

In this blog, post a short example will be given of using Packer and Terraform in combination. The goal will be to create a Ubuntu template from an ISO on vSphere using Packer, and then launch a customized instance of that template while configuring Apache on the system.

The sample code used for this post can be found on a repository on my Github.

Packer

To begin, we need to have the ability to run a Ubuntu installation unattended, this functionality is already built into Ubuntu. A Preseed configuration file will be provided to Ubuntu within the boot time parameters, through this configuration file and some other boot parameters Ubuntu will know to perform an automated installation.


Configuration File Rundown

This is just a brief overview of the configuration files present and their general purpose.


Ubuntu Preseed File

The Preseed configuration file used for this example can be seen below. Each section is explained in the comments. It is important to understand this configuration file is specific to Ubuntu, another Linux distribution will have a different configuration file and potently a different method.

Ubuntu Preseed Configuration - preseed.cfg

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
### Base system installation
d-i base-installer/kernel/override-image string linux-server    # Specifies to install the Kernel.

### Account setup
d-i passwd/user-fullname string user                            # Full name of the user to create.
d-i passwd/username string user                                 # Username of the user to create.
d-i passwd/user-password password qwe123                        # Password of the user to create.
d-i passwd/user-password-again password qwe123                  # Repeat password, for the installer.
d-i user-setup/allow-password-weak boolean true                 # Allow weak passwords, this is a test box.
d-i user-setup/encrypt-home boolean false                       # Do not encrypt the users home folder.

### Clock and time zone setup
d-i clock-setup/utc boolean true                                # Set the hardware clock to UTC.                         
d-i time/zone string UTC                                        # Set the timezone to UTC.

### Partitioning
d-i partman-auto/method string lvm                              # Use LVM as partitioning method.
d-i partman-lvm/confirm_nooverwrite boolean true                # Makes partman automatically write to disk without user confirmation.       
d-i partman-auto-lvm/guided_size string max                     # Use all of the volume group for the logical volumes created.
d-i partman/choose_partition select finish                      # Makes partman automatically write to disk without user confirmation.
d-i partman/confirm_nooverwrite boolean true                    # Makes partman automatically write to disk without user confirmation.

### Mirror settings
#d-i mirror/country string US
d-i mirror/http/proxy string                                    # Optional proxy setting, here it is empty.

### Package selection
tasksel tasksel/first multiselect standard                      # Install the standard set of packages.
d-i pkgsel/update-policy select none                            # Do not perform any updates or upgrades.
d-i pkgsel/include string openssh-server open-vm-tools cloud-init # Install some required packages for our template.
d-i pkgsel/install-language-support boolean false                # Don't need any language support.

### Boot loader installation
d-i grub-installer/only_debian boolean true                      # Makes GRUB install to MBR if no other OS is detected.

### Finishing up the installation
d-i finish-install/reboot_in_progress note                      # Prevent the message that install has been completed at the end.

In order to use this file with an automated install the following boot parameter will be specified to Ubuntu:

1
auto=true auto-install/enable=true preseed/url=http://mywebsite.com/mypressed.cfg vga=788 initrd=/install/initrd.gz fb=false debconf/frontend=noninteractive hostname=UbuntuTemplate debian-installer=en_US auto locale=en_US kbd-chooser/method=us keyboard-configuration/modelcode=SKIP keyboard-configuration/layout=USA keyboard-configuration/variant=USA console-setup/ask_detect=false ---

When this boot-time parameter is passed Ubuntu will being installing automatically. However, we do not want to type this in manually. This is where Packer comes in. Its job will be to create a virtual machine, boot up the virtual machine, and then execute this boot-time parameter to complete the installation. There are some other tasks Packer can perform on top of this, read on.

Now take note, we are pulling the configuration file from a remote location here, specially http://mywebsite.com/mypressed.cfg. Other distributions, such as CentOS 7, provide the ability to attach their configuration files into the virtual floppy disk drive created on the virtual machine. This is a convenient way of passing the configuration file when using Packer. This, however, is not possible because Ubuntu has begun removing the drivers for floppy disks.

Required Variables

Next, we begin looking at the configurations that apply directly to Packer. First, vars.json contain multiple variable definitions. Read the comments to get a feel for what is required.

Just to be clear, Packer is responsible for the connection to vSphere, creating a virtual machine, running an automated installation, and then converting that virtual machine to a template.

Variable Values Definitions - vars.json

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
{
    "vcenter_server":"vcenter.example.com",   # vCenter domain.
    "username":"[email protected]",   # Username and SSO domain for vSphere.
    "password":"VMware1!",                    # Password for vSphere.
    "datastore":"Example-Datastore-Name",     # Datastore to store the template on.
    "folder": "",                             # Target path in datastore where template will be created, optional vale.
    "host":"esxi02.example.com",              # ESXi host to use as compute resource.
    "cluster": "Example Lab Cluster",         # The cluster to run the virtual machine in.
    "network": "VM Network",                  # Portgroup to attack to the virtual machine.
    "ssh_username": "user",                   # Username to use for SSH connection.
    "ssh_password": "qwe123",                 # Password to use for SSH connection.
    "seed_file_domain": "192.168.55.163",     # Location of Seedfile configuration.
    "seed_filename": "preseed.cfg",           # Filename of the configuration file.
    "cpu_number": "2",                        # Number of virtual CPU's to use on virtual machine.
    "ram_amount": "2048",                     # Amount of RAM to use on virtual machine.
    "disk_size": "8024",                      # Amount of hard disk space to use for virtual machine (thin provisioned).
    "template_name": "template_ubuntu18"      # Name of the template to be created.
}

These variables can be hardcoded in the main configuration, I just find it cleaner to add it in a separate file.

Packer Configuration

We now move on and look at the core configuration of Packer. The configuration is split into three sections. builders which are responsible for specifying parameters needed to build the template, boot_command which will contain the instructions needed to perform the automated installation, and provisioners which will SSH into the virtual machine and perform any last-minute configurations.

The core Packer configuration beings with the builders section which mainly makes use of the variables that have been defined above. The only real addition of the ISO URL, in this case, we are fetching the URL from a remote location. This can also be set to a Datastore path within vSphere.

Packer Core Configuration - Builders Section - ubuntu-18.json

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
    "builders": [
      {
        "type": "vsphere-iso",                                # Provider to use, in this case vSphere.
  
        "vcenter_server":      "{{user `vcenter_server`}}",   # vCenter domain.
        "username":            "{{user `username`}}",         # Username and SSO domain for vSphere.
        "password":            "{{user `password`}}",         # Password for vSphere.
        "insecure_connection": "true",                        # Allow invalid TLS certificates, such as self signed.
  
        "vm_name": "{{user `template_name`}}",                # Name of the template to be created.
        "datastore": "{{user `datastore`}}",                  # Datastore to store the template on.
        "folder": "{{user `folder`}}",                        # Target path in datastore where template will be created, optional vale.
        "host":     "{{user `host`}}",                        # ESXi host to use as compute resource.
        "convert_to_template": "true",                        # Convert the virtual machine to a template at the end.
        "cluster": "{{user `cluster`}}",                      # The cluster to run the virtual machine in.
        "network": "{{user `network`}}",                      # Portgroup to attack to the virtual machine.
        "boot_wait": "1s",                                    # Wait 1 second before running boot command.
        "boot_order": "disk,cdrom",                           # Set the boot order, dist and then CDROM.
  
        "guest_os_type": "ubuntu64Guest",                     # Set the guest OS type.
  
        "ssh_username": "{{user `ssh_username`}}",            # Username to use for SSH connection.
        "ssh_password": "{{user `ssh_password`}}",            # Password to use for SSH connection.
  
        "CPUs":             "{{user `cpu_number`}}",          # Number of virtual CPU's to use on virtual machine.
        "RAM":              "{{user `ram_amount`}}",          # Amount of RAM to use on virtual machine.
        "RAM_reserve_all": false,                             # Do not reserve all the RAM.
  
        "disk_controller_type":  "pvscsi",                    # Set the disk controller type.
        "disk_size":        "{{user `disk_size`}}",           # Amount of hard disk space to use for virtual machine.
        "disk_thin_provisioned": true,                        # Set virtual disk to thin provisioned.
  
        "network_card": "vmxnet3",                            # Use a VMXNET3 network adapter.
  
        "iso_urls": "http://cdimage.ubuntu.com/ubuntu/releases/bionic/release/ubuntu-18.04.4-server-amd64.iso",   # ISO source URL.
        "iso_checksum": "e2ecdace33c939527cbc9e8d23576381c493b071107207d2040af72595f8990b",                       # Checksum of ISO.
        "iso_checksum_type": "sha256",                                                                            # Hash function for checksum.

The more interesting, boot_command, follows the initial builders. Here a boot command is defined, these are the exact keystrokes to emulate to begin the automated installation processes. We begin by entering the Linux boot parameter console, delete everything that is there by default, and write out our custom boot instructions.

Packer Core Configuration - Boot Command - ubuntu-18.json

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
        "boot_command": [
          "<wait><esc><wait><f6><wait><esc><wait>",              # Sequence to enter the boot command terminal.
          "<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",    # Backspaces.
          "<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",    # Backspaces.
          "<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",    # Backspaces.
          "<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",    # Backspaces.
          "<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",    # Backspaces.
          "<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",    # Backspaces.
          "<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",        # Backspaces.
          "auto=true",                                           # An alias for auto-install/enable, see next line. Just here to be safe.
          " auto-install/enable=true",                           # Delays the locale and keyboard questions until after there has been a chance to preseed them.
          " preseed/url=http://{{user `seed_file_domain`}}/{{user `seed_filename`}}", # URL of preseed configuration file.
          " vga=788",                                                                 # Set screen resolution, was a default parameter.
          " initrd=/install/initrd.gz",                                               # Location of initrd.
          " fb=false debconf/frontend=noninteractive",                                # Sets frontend to non interactive.
          " hostname=UbuntuTemplate",                                                 # Hostname of machine.
          " debian-installer=en_US auto locale=en_US kbd-chooser/method=us",          # Sets locale.
          " keyboard-configuration/modelcode=SKIP keyboard-configuration/layout=USA", # Sets keyboard settings.
          " keyboard-configuration/variant=USA console-setup/ask_detect=false",       # Sets other keyboard settings.
          " ---<enter>"                                                               # Add ending '---' string and press enter.
        ]
      }
    ],

Lastly, the provisioners is defined. This contains instructions to run at the very end before shutting down the virtual machine and converting it to a template.

One important part to note is the execute_command, this definition will help with the use of sudo. We cannot use sudo alone because it will prompt for an interactive password. To solve this the password is piped in through STDIN. Using this method the scripts will be run with sudo.

Packer Core Configuration - Provisioners Section - ubuntu-18.json

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
"provisioners": [
      {
        "type": "shell",
        "execute_command": "echo '{{user `ssh_password`}}'|{{.Vars}} sudo -E -S bash '{{.Path}}'",  # How to run all commands, or shell scripts in this case.
        "scripts": [
          "scripts/update.sh",              
          "scripts/install_perl.sh",
          "scripts/user_no_password_sudo.sh"
        ]
      }
    ]

Extra Scripts

There are three scripts to run at the end of this process, the first is a simple script to update Ubuntu.

Update Script - update.sh

1
2
3
#!/bin/sh

sudo apt-get update ; sudo apt-get upgrade -y ; sudo apt-get dist-upgrade -y; sudo apt-get autoremove -y ; sudo apt-get autoclean -y

The second will install some Perl packages. These are extremely important as Terraform uses a component of VMware Tools named deployPKG. This component is responsible for providing the ability to customize the virtual machine, and it is dependent on Perl. Through testing, I have determined these are the packages needed to use Terraform with customization successfully.

Perl Installation - install_perl.sh

1
2
3
#!/bin/sh

sudo apt-get install perl perl-modules perl-cross-config perl-depends -y

The last script will allow the new user user to run commands using sudo without a password.

Sudo no passowrd script - user_no_password_sudo.sh

1
2
3
#!/bin/sh

sudo echo "user    ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers

Packer Demonstration

Below is a demonstration of Packer running. In the beginning, you notice curl is used on an IP address, this is the IP address that the web server will be on when everything is complete. For the time being it will return a timeout as no web server has been set up yet.

Please note, you will notice that Packer idles on ==> vsphere-iso: Waiting for IP... for some time. This is because during this time Ubuntu is installing, and once it reboots it will get an IP address from DHCP, and Packer will be notified via VMware tools that are installed on the virtual machine.

Terraform

The Terraform configuration spans a few files. Files ending in .tf can all be combined into a single file, however, for readability I like to keep them into different files.


Configuration File Rundown

This is just a brief overview of the configuration files present and their general purpose.


Version File

The versions.tf file contains the minimum version of Terraform that can be used. The required_version is a good way to pin the Terraform module to a specific version of Terraform.

Terraform Version - versions.tf

1
2
3
terraform {
  required_version = ">= 0.12"
}

Variables Definitions

All the variables that will be used are declared in the variables.tf file, they are set to empty values.

Terraform Variable Declaration - variables.tf

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
variable vsphere_user {}              # Username and SSO domain for vSphere.
variable vsphere_password {}          # Password for vSphere.
variable vsphere_server {}            # vCenter domain.
variable vsphere_datacenter_name {}   # Name of vSphere datacenter to run virtual machine.
variable vsphere_resource_pool {}     # Name of resource pool.
variable vsphere_host {}              # ESXi host to use as compute resource.
variable vsphere_datastore {}         # Datastore to store the template on.
variable vsphere_network {}           # Portgroup to attack to the virtual machine.
variable vm_name {}                   # Name of the virtual machine.
variable ip_address {}                # Static IP address of the virtual machine.
variable netmask {}                   # Network netmask.
variable def_gw {}                    # Default gateway.
variable dns_server {}                # DNS server list.
variable domain {}                    # DNS search domain.
variable ssh_username {}              # User with SSH access.
variable ssh_password {}              # Password for user with SSH access.

Variable Values

The terraform.tfvars contains the values to be assigned to each of the variables.

Variable Values - terraform.tfvars

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
# vsphere related params
vsphere_user = "[email protected]"      # Username and SSO domain for vSphere.
vsphere_password = "VMware1!"                   # Password for vSphere.
vsphere_server = "vcenter.example.com"          # vCenter domain.
vsphere_datacenter_name = "Example Lab Datacenter"  # Name of vSphere datacenter to run virtual machine.
vsphere_resource_pool = "Example Lab Cluster/Resources" # If you haven't resource pool, put "Resources" after cluster name
vsphere_host = "esxi02.example.com"             # ESXi host to use as compute resource.
vsphere_datastore = "Example-Datastore-Name"    # Datastore to store the template on.
vsphere_network = "VM Network"                  # Portgroup to attack to the virtual machine.

# VM Setup
vm_name = "Terraform Webserver VM"              # Name of the virtual machine.
ip_address = "192.168.1.168"                    # Static IP address of the virtual machine.
netmask = "24"                                  # Network netmask.
def_gw = "192.168.1.1"                          # Default gateway.
dns_server = ["8.8.8.8"]                        # DNS server list.
domain = "example.com"                          
ssh_username = "user"                           # User with SSH access.
ssh_password = "qwe123"                         # Password for user with SSH access.

Terraform Core Configuration

The webserver_vm.tf contains the core Terraform configuration logic. It beings by specifying much of what was described in the variables, where to connect to, and where to run the virtual machine.

Value Definitions - webserver_vm.tf

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
# Basic configuration withour variables

# Define authentification configuration
provider "vsphere" {
  # If you use a domain set your login like this "MyDomain\\MyUser"
  user           = var.vsphere_user       # Username and SSO domain for vSphere.
  password       = var.vsphere_password   # Password for vSphere.
  vsphere_server = var.vsphere_server     # vCenter domain.

  # if you have a self-signed cert
  allow_unverified_ssl = true             # Allow invalid certificates, such as self signed.
}

#### RETRIEVE DATA INFORMATION ON VCENTER ####

data "vsphere_datacenter" "dc" {
  name = var.vsphere_datacenter_name      # Name of vSphere datacenter to run virtual machine.
}

data "vsphere_resource_pool" "pool" {
  # If you haven't resource pool, put "Resources" after cluster name
  name          = var.vsphere_resource_pool     # If you haven't resource pool, put "Resources" after cluster name
  datacenter_id = data.vsphere_datacenter.dc.id
}

data "vsphere_host" "host" {
  name          = var.vsphere_host              # ESXi host to use as compute resource.
  datacenter_id = data.vsphere_datacenter.dc.id
}

# Retrieve datastore information on vsphere
data "vsphere_datastore" "datastore" {
  name          = var.vsphere_datastore         # Datastore to store the template on.
  datacenter_id = data.vsphere_datacenter.dc.id
}

# Retrieve network information on vsphere
data "vsphere_network" "network" {
  name          = var.vsphere_network           # Portgroup to attack to the virtual machine.
  datacenter_id = data.vsphere_datacenter.dc.id
}

# Retrieve template information on vsphere
data "vsphere_virtual_machine" "template" {
  name          = "template_ubuntu18"           # Name of the template to clone.
  datacenter_id = data.vsphere_datacenter.dc.id
}

The second part of webserver_vm.tf describes the resource that will be created, the web server virtual machine in this case. As can be seen on line 24 we specify a template that we will clone the virtual machine from. There is a set of parameters after that which will customize the virtual machine, including its hostname and static networking information.

Virtual Machine Setup - webserver_vm.tf

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
# Set vm parameters
resource "vsphere_virtual_machine" "webserver-vm" {
  name             = var.vm_name    # Name of the virtual machine.
  num_cpus         = 2              # Number of vCPUs.
  memory           = 4096           # Amount of RAM.
  datastore_id     = data.vsphere_datastore.datastore.id              # Datastore to store the template on.
  host_system_id   = data.vsphere_host.host.id                        # ESXi host to use as compute resource.
  resource_pool_id = data.vsphere_resource_pool.pool.id               # If you haven't resource pool, put "Resources" after cluster name
  guest_id         = data.vsphere_virtual_machine.template.guest_id   # ID of the virtual machine taken from template.
  scsi_type        = data.vsphere_virtual_machine.template.scsi_type  # SCSI type taken from template.

  # Set network parameters
  network_interface {
    network_id = data.vsphere_network.network.id    # Portgroup to attack to the virtual machine.
  }

  # Use a predefined vmware template has main disk
  disk {
    label = "webserver.vmdk"    # Name of the VMDK.
    size = "30"                 # Size of VMDK.
  }

  clone {
    template_uuid = data.vsphere_virtual_machine.template.id

    customize {
      linux_options {
        host_name = "TerraformWebserver"  # New hostname of virtua machine.
        domain    = var.domain            # DNS search domain.
      }

      network_interface {
        ipv4_address    = var.ip_address  # Static IP address.
        ipv4_netmask    = var.netmask     # Netmask.
      }

      ipv4_gateway = var.def_gw           # Default gateway.
      dns_server_list = var.dns_server    # DNS server list.

    }
  }

The last part of the configuration has to do with a script that will be run. After networking is setup Terraform will move this script to the system and run it using SSH. This is the script that will install Apache.

Virtual Machine Customization - webserver_vm.tf

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
  # Execute script on remote vm after this creation
  provisioner "remote-exec" {
    script = "scripts/install-webserver.sh"
    connection {
      type     = "ssh"
      user     = var.ssh_username
      password = var.ssh_password
      host     = var.ip_address
    }
  }

Terraform Demonstration

Below you will see a demonstration of Terraform being run, towards the end you can see curl once again being run with a successful HTTP response.

Resources