AWS + Terraform + Auto Scale Group + User Data Bash Script on Startup to Customize Image

User Data  – On Startup

If you want to customize your VM image on its first start-up, you may want to use “user data”.  You can basically think of this as a script that will be run right after boot-up the very first time.  You can also make it run every reboot apparently (with extra config).

Why would you need this?  Well, in my case, I was spawning up a Presto cluster.  I generally do this in a special HA way… but even if you did it the simple way, you would have 1 coordinator and N workers, and the N workers would have to point at your 1 coordinator.

So, there are 2 interesting things here:

  1. The coordinator and workers are identical barring some slightly different configuration in one file.
  2. The workers need to know about the coordinator in order to use it.

So, for both of these cases, we’d like to run a script on start-up!.

The Terraform Code

When you want to create an auto-scale-group, you have to start by creating a launch template: https://www.terraform.io/docs/providers/aws/r/launch_template.html.

You can use that template to spawn up multiple auto-scale groups when its is done.  The launch template itself has the user data though.  So, you are best off trying to make your user data script generic enough that it can work for all your cases.  It can be a bash file and can use variables, so this isn’t too hard.

If you do need multiple separate user data scripts you’ll have to use separate launch templates, which is not the end of the world either.

The launch template in the link above is very complete, so all I’m going to show you is how to pass a bash script that takes parameters to the user data.

Basically replace:

user_data = "${base64encode(...)}"

In their example with something like this:

user_data = base64encode(templatefile("${path.module}/worker-script.sh", {coordinator_lb = "${aws_lb.coordinator.dns_name}", hive_thrift_csv = "${var.hive_thrift_csv}"}))

Assuming your worker-script has content like this:

#!/bin/bash
echo "Hello World" > /tmp/test-output.txt

and you have the hive_thrift_csv variable defined in your variables file like this:

variable "hive_thrift_csv" {
type = "string"
default = "thrift://ip-addr-1:9083,thrift://ip-addr-2:9083"
}

you should be good. Note, the first variable, definition coordinator_lb = “${aws_lb.coordinator.dns_name}” is a reference to the DNS name from a load balancer created in another part of my terraform config. I left it in as its a good example for a more complex separate variable.

Building Presto Admin

Presto Admin – Is it Worth Using?

I generally deploy Presto clusters using packer and terraform.  Packer builds an image for me with the target presto distribution, some utility scripts, ha proxy (to get multiple coordinators acting in HA), etc.

I kept noticing this presto-admin project though: https://github.com/prestosql/presto-admin.  It allows you to quickly/easily deploy clusters from a central node, and it will handle the coordinator, workers, catalogs, and everything.  That sounds pretty cool.

Advance disclaimer – after I built this, I decided not to use it.  This was because it seems to just deploy a single coordinator and worker set.  For an HA setup, I need multiple coordinators, a load balancer, and workers/users pointing at the load balancer.  So, it’s just not the right fit for me.

Presto Admin – Build

In any case, I did go through the motions of building this – because I could not find a source release.  Fortunately, it’s pretty easy on Centos 7.x (basically RHEL 7.x):

# Download and unzip.
wget https://github.com/prestosql/presto-admin/archive/2.7.tar.gz
tar -xvzf 2.7.tar.gz

# Install pip/etc.
sudo yum install epel-release
sudo yum install python-pip
sudo yum install python-wheel

# Run make file and build web installer.
make dist

After this, just go into the dist folder and find prestoadmin-2.7-online.tar.gz.

I hope this saves you some time; I wasted around 20 minutes trying to find the dist online for download (which I never did).

Using Ansible in Jenkins – Ansible Plugin

Jenkins + Ansible Options

I have been doing a lot of Jenkins, Ansible, Terraform, and similar automations lately and I have seen multiple ways of running Ansible in Jenkins.  These include:

  1. Install Ansible on your Jenkins node and call it with “sh” in a pipeline.
  2. Install Ansible on your Jenkins node and call it with the ansible plugin for Jenkins (not a default plugin).
  3. Run a docker image with Jenkins and configure ansible there.

I think the 3rd option is the most powerful as you can separately configure and version Ansible for your various Jenkins jobs.  But the second option is pretty sleek as well and is what we’ll talk about here.

Jenkins Ansible Plugin

If you go to the Jenkins plugins management page, you can install the ansible plugin pretty easily.  There are some good documents on it right here too.  After installing that plugin, I also suggest you install the AnsiColor plugin.

With both of those in place, you can get ansible running your playbooks with its private key credentials and it will print beautiful colored output to your logs.

Here is an example of how to call it from a pipeline.  Note that this lets you specify your target hosts in a CSV, assuming your playbook uses the “target_hosts” group.  It also disables host key checking (which took me a while to work out as the docs are wrong).  For some reason my ssh wouldn’t work without that setting with this plugin even though it is not ideal.

pipeline {
    agent any

    parameters {
        string(name: 'GIT_BRANCH', defaultValue: 'master', description: 'git branch to work from.')
        string(name: 'TARGET_HOSTS_CSV', defaultValue: 'none', description: 'target deployment hosts.')
    }

    stages {
        stage('Deploy application.') {
            steps {
                sh 'echo [target_hosts] > /tmp/inventory.ini'
                sh 'echo "${TARGET_HOSTS_CSV}" | tr "," "\n" >> /tmp/inventory.ini'

                ansiColor('xterm') {
                    ansiblePlaybook(
                        playbook: './ansible/playbook.yml',
                        inventory: '/tmp/inventory.ini',
                        credentialsId: 'your-jenkins-pk-credential',
                        disableHostKeyChecking: true,
                        colorized: true)
                }
            }

        }
    }
}

Setting up HTTPS on an AWS Load Balancer

Context

This was my first time setting up HTTPS using a real certificate rather than a self-signed one.  This way you get that nice lock symbol + https in your browser without the user seeing the obnoxious “do you trust this site, continue unsafely” warnings.

Steps

Generate a Key Pair

Note that for common name you should use the DNS name that you will be using for the website (or load balancer in front of the website).  This is important.  E.g. *.appname.yourcompany.com.  This one is in case of a wildcard cert to handle anything in front of the app-name as well.

openssl req -new -newkey rsa:2048 -nodes -keyout appname.key -out appname.csr

Do not lose this key pair! Store it somewhere safe and backed up.

Request a Certificate from DigiCert/etc

Go to DigiCert or whatever service your company uses.  Request a standard SSL certificate using the files you generated above.

For the case of a wild-card cert, you will want to add Subject Alternative Names (SANs) for appname.yourcompany.com and *.appname.yourcompany.com.

If you get odd failure messages, these sites tend to have something to validate your CSR.  In my case, I think it made me go remake the CSR without a password before it would accept it.  This was only obvious from the validator; the initial messages were just confusing.

Note: The server name and location fields don’t seem to matter much.  I just put AWS for location and my app name with some environment suffix for the server name (nothing exists with this name).

Add Your Certificate to the AWS Certificate Manager

Once your certificate comes back from Digicert, you can upload it to the Certificate Manager in AWS.  The body is the certificate returned to you from Digicert/etc.  The private key is the one you generated above.  The certificate chain just needed the “intermediate certificate” that Digicert returned to me along with the new certificate (and only that, don’t put your certificate in there as well).

Create Your Load Balancer

Go to EC2 and create a new network load balancer.  Add a TLS (Secure TCP) listener and on the security settings page, pick “Choose a certificate from ACM (recommended)”.  Then you can select your certificate.

Create Your DNS Name

Contact your system admins to create a DNS name for you with the name you used in your certificate request (e.g. appname.yourcompany.com from above).  Point it at the IP of the load balancer.  You will have to resolve the load balancer name to an IP for this (I’m not sure that is best practice, so you may want to read up more before following this last step).

All Done!

Assuming your load balancer points to an application now, you can use https to talk to talk to the DNS name of the load balancer, and the load balancer will take it, terminate the TLS, and forward to your application.  So, you’re good!

 

Ansible – Install AWS CLI and Log in To Amazon ECR Docker Registry via Ansible

There is probably a much cleaner way of doing this using off-the-shelf automations.  But I was just following along with the AWS installation instructions and got this working.

- name: Download AWS CLI bundle.
shell: "cd /tmp && rm -rf /tmp/awscli* && curl 'https://s3.amazonaws.com/aws-cli/awscli-bundle.zip' -o 'awscli-bundle.zip'"

- name: Update repositories cache and install "unzip" package
apt:
name: unzip
update_cache: yes

- name: Unzip AWS CLI bundle.
shell: "cd /tmp && unzip awscli-bundle.zip"

- name: Run AWS CLI installer.
shell: "/tmp/awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws"

- name: Log into aws ecr docker registry
when: jupyterhub__notebook_registry != ''
shell: "$(/usr/local/bin/aws ecr get-login --no-include-email --region us-east-1)"

In order to do the actual login, you need to ensure your EC2 instance has an IAM role assigned to it that has reader privileges. Then you should be all good!

Resolving the ContextualVersionConflict: (cryptography 1.7.1 (/usr/lib/python2.7/dist-packages) error.

I was trying to run ansible over a debian slim docker image and I got this error, which was rather cryptic.  There wasn’t any particularly good information on google for fixing it either.

ContextualVersionConflict: (cryptography 1.7.1 (/usr/lib/python2.7/dist-packages), Requirement.parse(‘cryptography>=2.5’), set([‘paramiko’]))
ERROR! Unexpected Exception, this is probably a bug: (cryptography 1.7.1 (/usr/lib/python2.7/dist-packages), Requirement.parse(‘cryptography>=2.5’), set([‘paramiko’]))
the full traceback was:

Traceback (most recent call last):
File “/usr/local/bin/ansible-playbook”, line 97, in <module>
mycli = getattr(__import__(“ansible.cli.%s” % sub, fromlist=[myclass])…

In my case, running this before my ansible install resolved the issue.  It just forces an upgrade to a newer version which ansible is okay with.  Of course, if you were dependent on the lesser version this may not be an option.

pip install –force-reinstall cryptography==2.7

 

Git – Make File Executable in Repository

When you’re deploying code, you often want the files you have checked out of git to be executable.  For example, if you have a script for automation, you want to be able to do ./script rather than “bash script” or “python script”.  That way the #! line takes care of doing the right thing.

You can actually make a file “executable” in git by doing this command before your push:

git update-index --chmod=+x somefile.sh