JupyterHub – NGINX TLS Termination – Ubuntu

Due to corporate security requirements, I just had to ensure I had TLS both between clients and my load balancer as well as TLS from the load balancer to the back-end application (JupyterHub).

This was a little problematic because I was using a real certificate, so I had intentionally terminated TLS at the load balancer for cost reasons.  So, I used a self-signed certificate between the load balancer and the back-end just now.

If you use this GitHub gist right here for your nginx config, and you modify the certificate paths to point to files you generate from this digital-ocean tutorial, it works out just fine.  Then you just have to point your load balancer to point 443 on your JuptyerHub host(s) and everything works out great.

Here’s an excerpt of the relevant parts of the digital-ocean tutorial. Once you make the files, you can just update the gist yourselves to use them. The Diffie-Hellman group line is not in the gist; so add that yourself based on the digital-ocean one if you are so inclined.

mkdir -p /etc/nginx/ssl && cd /etc/nginx/ssl;
sudo chmod 700 /etc/nginx/ssl
sudo openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -keyout nginx-selfsigned.key -out nginx-selfsigned.crt

Generally fill out all the details for the cert normally, but pay extra attention to common name. This should match your DNS name (e.g. env.yoursite.com). If you deploy to multiple environments and this is an internal app/etc, you may consider *.yoursite.com to avoid needing one per environment).

Once you’re done that, also run the following to create a “strong Diffie-Helman group”. Refer to digital-ocean’s link for this one; I honestly didn’t have the time to look into why this is needed yet.

sudo openssl dhparam -out dhparam.pem 2048

Install PGAdmin Server With Docker

You can get PGAdmin 4 running in server mode with docker very easily.  Using this command will set up the server, set it to always restart in response to reboots or errors, and it will ensure that its data (users, config) is persisted between container runs.

docker pull dpage/pgadmin4

docker run -p 80:80 \
    --name pgadmin \
    --restart always \
    -v "/opt/pgadmin4/data:/var/lib/pgadmin" \
    -v "/opt/pgadmin4/config/servers.json:/servers.json" \
    -e "PGADMIN_DEFAULT_EMAIL=user@domain.com" \
    -d dpage/pgadmin4

You can run this command afterward to see the details and confirm the restart policy as well if you like:

docker inspect pgadmin


AWS CentOS Extend Root Volume with No Downtime

In AWS, you can generally extend the root (or other) volume of any of your EC2 instances without downtime.  The steps slightly vary by OS, file system type, etc though.

On a rather default-configured AWS instance running the main marketplace Centos 7 image, I had to run the following commands.

  1. Find/modify volume in the AWS console “volumes” page under the EC2 service.
  2. Wait for it to get into the “Optimizing” state (visible in the volume listing).
  3. Run: sudo file -s /dev/xvd*
    • If you’re in my situation, this will output a couple lines like this.
      • /dev/xvda: x86 boot sector; partition 1: ID=0x83, active, starthead 32, startsector 2048, 134215647 sectors, code offset 0x63
      • /dev/xvda1: SGI XFS filesystem data (blksz 4096, inosz 512, v2 dirs)
    • The important part is the XFS; that is the file system type.
  4. Run: lsblk
    • Again, in my situation the output looked like this:
      • xvda 202:0 0 64G 0 disk
      • └─xvda1 202:1 0 64G 0 part /
    • This basically says that the data is in one partition under xvda.  Note; mine said 32G to start.   I increased it to 64G and am just going back through the process to document it.
  5. Run: sudo growpart /dev/xvda 1
    • This grows partition #1 of /dev/xvda to take up remaining space.
  6. Run: sudo xfs_growfs -d /
    • This tells the root volume to take up the available space in the partition.
  7. After this, you can just do a “df -h” to see the increased partition size.

Note, your volume may take hours to get out of the “optimizing” stage, but it still can be used immediately.

You can view the raw AWS instructions here in case any of this doesn’t line up for you when you go to modify your instance: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-modify-volume.html.


Presto – Internal TLS + Password Login; Removing Private Key from JKS File


For various reasons, you may have to secure a Presto cluster with TLS, both internally and externally.  This is pretty straight forward following Presto documentation, until you want to also combine that with an LDAP or custom password login mechanism.  Once you have internal TLS, external TLS, and LDAP, you have to play with some extra settings and manipulate your JKS files to get things done.

Internal TLS Settings

For secure internal communication, you should refer to the presto documentation right here: https://prestosql.io/docs/current/security/internal-communication.html.  It will walk you through various configuration settings that enable HTTPS, disable HTTP, and set key stores for TLS.

Part of the instructions have you generate a JKS file (Java Key Store) with a command like this:

keytool -genkeypair -alias example.com -keyalg RSA -keystore keystore.jks
Enter keystore password:
Re-enter new password:
What is your first and last name?
  [Unknown]:  *.example.com (Your site name should go here).

This will get your internal TLS working fine.

Adding External TLS

It would be quite pointless to secure the inside of a cluster if you didn’t secure the connections to the clients using it.  So, you’ve actually set all of the external TLS properties already when you were doing the internal security.  E.g. notice that the properties listed in the LDAP login plugin (which requires external SSL) here: https://prestosql.io/docs/current/security/ldap.html are already referenced in the doc we referred to for internal TLS here https://prestosql.io/docs/current/security/internal-communication.html.

Initially, I figured that I could configure a different JKS file for internal and external communication; but it turns out that this does not work; so don’t try it.   There is some information on that right hereYou need to use the same JKS file in all keystore configurations on the Presto servers.  So, don’t bother trying to tune the properties you already set while doing internal TLS; just keep them.

Given internal and external communication needs the same keystore, a naive second try may be to give clients the same JKS file that you use for internal TLS… but that’s a bad idea for two reasons:

  1. You’re giving away your private key and compromising security.
  2. If you go on to add password-login by LDAP or a custom password authenticator, the private key certificate will bypass it if the clients have it.

So, what you really need to do to allow clients to use TLS safely is use the same JKS file for all the server-side properties, but give clients a copy of that JKS file with the private key removed for use with JDBC/etc.

You can remove the private key from the JKS you made with the internal TLS instructions like this:

keytool -export -alias company.com -file sample.der -keystore keystore.jks
openssl x509 -inform der -in sample.der -out sample.crt
keytool -importcert -file sample.crt -keystore .keystore
The generated .keystore file can be used in JDBC or other connections by referring to it with the SSLTrustStorePath and SSLTrustStorePassword properties.  As it doesn’t have the private key, it will work for SSL, but it will not work as a login mechanism.  So, if you set up password login, clients will have to use it (which is what you want).  You can find JDBC documentation here: https://prestosql.io/docs/current/installation/jdbc.html.

Password Logins

You can do user-name and password login with LDAP out of the box using the documentation I linked earlier.  Alternatively, you can use the custom password plugin documentation I wrote a month ago here: https://coding-stream-of-consciousness.com/2019/06/18/presto-custom-password-authentication-plugin-internal/ to do your own.

In either case, while combining internal TLS and password login, you will have to modify this property:

to say this:
You need this because you have to set the PASSWORD type to make password logins work… but that forces all traffic to require a password.  Internal nodes doing TLS will start asking each other for passwords and failing since they can’t do that.  So, you add CERTIFICATE to allow them to authenticate to each other using their JKS files.
This is why you had to strip the private key out of the file you gave to the clients.  If they had it and used it as a key store, they could have authenticated to the coordinator with the JKS file instead of a user name/password.  But just having the trust store with the public keys allows SSL to work while not allowing it to be used as the CERTIFICATE login mechanism.
I hope this helps you get it working! I spent longer on this than I would like to admit :).
Note: There is some good related conversation here: https://groups.google.com/forum/#!topic/presto-users/R_byjHcIS8A and here: https://groups.google.com/forum/#!topic/presto-users/TYdvs5kGYE8.  These are the google groups that helped me get this working.



Jenkins Pipeline – Maven + Artifactory Example With Secure Credentials

There are always a million ways to do things in Jenkins, but often using the appropriate plugins for common tools pays off a lot.  Here is an example of using multiple maven commands to execute a build, the result of which (a zip file) is then uploaded to Artifactory.

Sample Pipeline

Honestly, there will almost definitely be an artifactory plugin as well; but in this case, I used these 2 plugins:

  • Pipeline Maven Integration Plugin – Allows you to target various maven installations set up in your Jenkins “Global Tool Configuration”.
  • Credentials Binding Plugin – Allows you to pull user names and passwords from credentials into environment variables to use in commands easily.

This was enough to let me run the maven commands I needed and to let me then call the Artifactory REST API safely to upload my one artifact, which is a zip in this case.

I hope this example helps you! 🙂

pipeline {
    agent any
    parameters {
        string(name: 'SOURCE_BRANCH', defaultValue: 'master', description: '...')
        string(name: 'RELEASE_TAG', defaultValue: '', description: '...')
    stages {
        stage('Build and deploy presto zip to artifactory.') {
            steps {

                sh 'echo "Building to version = $RELEASE_TAG"'

                withMaven(maven: 'maven-3') {
                  sh "mvn versions:set -DnewVersion=$RELEASE_TAG"
                  sh "mvn clean install -DskipTests"

                withCredentials([usernamePassword(credentialsId: 'artifactory-token', usernameVariable: 'AUSR',
                    passwordVariable: 'APWD')]) {
                  sh '''curl -X PUT -u $AUSR:$APWD -T presto-server/target/presto-server-$RELEASE_TAG.tar.gz "https://company.com/artifactory/repo/io/presto/$RELEASE_TAG/presto-server-$RELEASE_TAG.tar.gz" '''
                sh 'echo "Done building and tagging to name = $RELEASE_TAG"'

JupyterHub or JupyterLab – Back Up All User Docker Container Files

If, like me, you deployed the JupyerHub docker spawner without persistent volumes and then ended up with tons of users having tons of content to lose, this may help you.

This short bash script will list all containers and then copy out their contents and zip it up to save space.


mkdir -p ~/notebook-backup
cd ~/notebook-backup

CONTAINERS=`docker container ls | grep -v PORTS | awk '{print $14}'`
  echo "Copying out files for ${NAME}; this may take a minute."
  docker cp ${NAME}:/home/notebook ./${NAME}

  echo "Zipping files for ${NAME}."
  tar -zcvf ${NAME}.tar.gz ${NAME}

  echo "Removing source files for ${NAME}."
  rm -rf ./${NAME}


Convert CA Certificate to JKS File (e.g. For Presto)

Many applications require JKS files to enable TLS (Transport Layer Security).  In case you are not sure what a JKS file is, you can read about what a JKS file is and see how to make a self-signed one right here.

Converting a CA Certificate to a JKS File

To convert the files a CA provides you into a JKS file you can do the following, which is lightly modified from this other article I followed.

cat /etc/ssl/certs/ca-bundle.crt IntermediateCA.crt > ca-certs.pem

openssl pkcs12 -export -in ssl_certificate.crt -inkey app.key -chain -CAfile ca-certs.pem -name "*.app.company.com" -out app.p12

keytool -importkeystore -deststorepass Password123! -destkeystore app.jks -srckeystore app.p12 -srcstoretype PKCS12

Note that the domain must match the one specified in the certificate.  Assuming these 3 commands work, you should have a proper JKS file when done.

Given the certificate is from a CA, clients should not need a copy of the JKS file to talk to servers that are using it.   For example, if my Presto server uses this JKS file, JDBC clients on other hosts can talk to it over SSL even though they do not have a copy of the file themselves.

NOTE: The JKS file will only work properly when used against the correct domain.  E.g. if you have a load balancer at https://load-balancer.app2.company.com pointing at your server running your JKS file which is for https://server1.app.company.com, it will not work.  You have to make a CNAME so your load balancer actually looks like it is also under app.company.com (what’s in the cert) and not app2.company.com.