Lookup Tiller Version in Kubernetes Using Kubectl

This is just a simple command to help you in finding the tiller version running in kubernetes.  I made it when trying to make sure my laptop’s helm install matched the cluster tiller install:

TILLER_POD=`kubectl get pods -n kube-system | grep tiller | awk '{print $1}'`
kubectl exec -n kube-system $TILLER_POD -- /tiller -version

Debugging Spring Boot Multi Stage Docker Build – JAR Extract Not Working

Containerized Spring Boot – Multi Stage Build

I was following this good tutorial on deploying spring-boot apps to kubernetes:

https://spring.io/guides/gs/spring-boot-kubernetes/

Their docker file looks like this:

FROM openjdk:8-jdk-alpine AS builder
WORKDIR target/dependency
ARG APPJAR=target/*.jar
COPY ${APPJAR} app.jar
RUN jar -xf ./app.jar

FROM openjdk:8-jre-alpine
VOLUME /tmp
ARG DEPENDENCY=target/dependency
COPY --from=builder ${DEPENDENCY}/BOOT-INF/lib /app/lib
COPY --from=builder ${DEPENDENCY}/META-INF /app/META-INF
COPY --from=builder ${DEPENDENCY}/BOOT-INF/classes /app
ENTRYPOINT ["java","-cp","app:app/lib/*","com.example.demo.DemoApplication"]

So, it is using a multi-sage docker build.  The reasoning for this seems to be described well in this other article here: https://spring.io/blog/2018/11/08/spring-boot-in-a-container.  Basically:

A Spring Boot fat jar naturally has “layers” because of the way that the jar itself is packaged. If we unpack it first it will already be divided into external and internal dependencies. To do this in one step in the docker build, we need to unpack the jar first. For example (sticking with Maven, but the Gradle version is pretty similar):

There are now 3 layers, with all the application resources in the later 2 layers. If the application dependencies don’t change, then the first layer (from BOOT-INF/lib) will not change, so the build will be faster, and so will the startup of the container at runtime as long as the base layers are already cached.

Build Failing

In my case, I was getting an error like this:

Step 9/12 : COPY --from=0 ${DEPENDENCY}/BOOT-INF/lib /app/lib
COPY failed: stat /var/lib/docker/overlay2/72b76f414c0b527f00df9b17f1e87afb4109fa6448a1466655aeaa5e9e358e27/merged/target/dependency/BOOT-INF/lib: no such file or directory

This was a little confusing.  At first I assumed, that the overlay file system path didn’t exist.  When I tried to go there (to the 72… folder), it in fact was not present.  This was a red herring though.

It is failing to copy BOOT-INF/ content.  So, on a hunch, I assumed the 72… folder path was ephemeral.  If that is true, then it did exist at some time, and the path did not have BOOT-INF content in it.

So, the next step was to debug the intermediate container for the multi-stage build.  In my docker file, the first like is:

FROM openjdk:8-jdk-alpine AS builder

So, the name of the intermediate container is “builder”.  We can stop the build at that container to interrogate it by doing this:

docker build . --target=builder
Sending build context to Docker daemon   17.2MB
Step 1/5 : FROM openjdk:8-jdk-alpine AS builder
 ---> a3562aa0b991
Step 2/5 : WORKDIR target/dependency
 ---> Using cache
 ---> 1dbeb7fad4c0
Step 3/5 : ARG APPJAR=target/*.jar
 ---> Using cache
 ---> 67ba4a4a1863
Step 4/5 : COPY ${APPJAR} app.jar
 ---> Using cache
 ---> d9da25b3bc23
Step 5/5 : RUN jar -xf ./app.jar
 ---> Using cache
 ---> b513f7190731
Successfully built b513f7190731

And then interactively running in the built image (b513f7190731):

docker run -it b513f7190731

In my case, I could clearly see that basicalyl no files existed in the /target/dependency folder.  There was only a JAR that was never extracted by the “jar -xf ./app.jar” command.

Debugging the JAR Extraction

I tried to manually run the jar extraction command myself, and it gave no error and did absolutely nothing.  After that, I went to make sure the JAR was valid.  When I looked at its size, it was fine.  When I looked at it with “vi”, I could see that the header was a bash script.

This is because my spring-boot project had the spring-boot-maven plugin in its POM with executable = true; this embeds a bash script in the front to allow it to run as a Linux init.d service.

So, my first move was to remove the plugin altogether.  This did not help as, after that, the JAR did extract but it had no BOOT-INF folder.  Spring boot was not building the fat-jar with dependencies as I removed the plugin.  So, I had to put the plugin back in like this:

<plugin>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-maven-plugin</artifactId>
</plugin>

With executable = true removed.  This builds the fat JAR without the bash file embedded, which allows the “jar -xf ./app.jar” command to properly extract the contents including the BOOT-INF dependencies.

This was not a problem in the tutorial because they’re relying on generated projects from Initializr rather than existing projects you made yourself with possibly different use cases.

Summary

I hope this helped you learn about putting spring-boot in docker, and that it shed some light on how to debug a multi-stage docker build!  I knew a fair bit about docker and spring-boot going into this, but combining them revealed some process differences I had taken for granted over the years.  So, it was a good experience.

Minikube ImagePullBackOff – Local Docker Image

Background Context

Earlier today I was beginning to port our Presto cluster into kubernetes.  So, the first thing I did was containerize Presto and try to run it in a kubernetes deployment in minikube (locally).

I’m fairly new to minikube.  So far, I’m running it with vm-driver=None so that it uses the local docker instance rather than a virtualbox VM/etc.

The Error

So, I got my docker image building well and tested it within docker itself.  That all worked great.  Then I wrote my kubernetes deployment and ran it using the image… but unfortunately, it came up with the pod saying Error: ImagePullBackOff.

I went down a rabbit hole for a while after this because many posts talk about how to enable your minikube to have access to your local docker repo.   But when you’re running vm-driver=None, you are literally running in your local docker – so it should already have access to everything.

The actual error is: “Failed to pull image “qaas-presto:latest”: rpc error: code = Unknown desc = Error response from daemon: pull access denied for qaas-presto, repository does not exist or may require ‘docker login’: denied: requested access to the resource is denied”.

So, the issue is that it’s trying to do a pull and it can’t find the image… but it shouldn’t need to pull because the image already exists locally as it was built locally.

Workaround

I found the workaround in this github entry: https://github.com/kubernetes/minikube/issues/2575. Basically, in your deployment/pod spec/whatever, you just set:

imagePullPolicy: Never

This makes it avoid trying to pull the image, so it never fails to find it.  It just assumes it is present, which it is, and it uses it and moves on.  You may not necessarily want to deploy your config to production with these settings, but you can always template them out with helm or something, so it’s a viable workaround.

 

 

12 Factor App Best Practices – Quick Notes

Why Review This?

The 12 factor app “best practices” are a good overview of most things you should be doing in a modern application.  They provide a good overview of many architectural and dev-ops practices which people tend to learn in a more peace-meal and painful way over time.

Sources / References

While reviewing this recently myself, I read and viewed a few sources.  I found this video https://www.youtube.com/watch?v=94PxlbuizCU on YouTube gave a very practical summary… especially for those who like to work in Java a lot such as myself.  I also found this other one from AWS https://pages.awscloud.com/Building-Microservices-with-the-12-Factor-App-Pattern-on-AWS_0512-CON_OD.html which fills in some areas better than the other, especially when it comes to practical use with containerization and also in the area of logging.  I recommend taking the time to watch both.

The Twelve Factors

Provided with my interpretation/commentary of course based on the various sources I looked at.

  1. Code Base – Should be checked into version control (e.g. Git).  Should have  one application per repository (or sub-module).  E.g. if you have an API and a website, separate them so you don’t end up having to combine releases / so you aren’t inclined to couple them and muddy the waters.
  2. Dependencies – Should be declared.  Should also be external to your code base (e.g. pull from a maven repo, don’t check JARs into your git repo).  Should avoid using anything from outside the project (e.g. if you’re in Node, don’t global install anything, make sure everything is local to the project).  This also kind of leads to the fact that you should not rely on something like an app server (e.g. tomcat) external to your app to host your app.  Your app should be more like a spring-boot application which has is deployed with its own web server bundled in.
  3. Configuration – Passwords/tokens/connections/etc should be separate from your code.  E.g. using profiles and ansible-secrets/etc within your code base to handle multiple environments can be bad because you have to modify your code base to support new environments.  These configurations should be moved out to environment variables or some other mechanism.  The use of environment variables seems to be recommended in many frameworks/technologies online these days, so I think its good to do when possible.  Also note, the use of CNAMES and other flexible configuration mechanisms to abstract the location from which things are running is recommended.
  4. Backing Services – Databases/queues/caches/etc should be decoupled and easily changeable in your configuration.  Again, prefer CNAMEs/etc for abstraction.
  5. Build/Release/Run – A deployment = building code, combining it with config to form a release, and executing that release in the target environment.  This one sounds a little… flexible to me?  E.g. Helm lets you take a template, apply values to it, and deploy the result to kubernetes.  So, in that case, I guess the helm template/vars part is the “release” formation, and running it is the execution of helm?  We don’t need separate docker images built for each environment in this case.  If, on the other hand, we’re taking each environment’s config and generating a new docker image per environment, then we’d have one release per environment (code + config combined), and it would be nicely decoupled from the “run”, in which case this makes more direct sense.  Note that the AWS video calls this later case an anti-pattern and recommends deploying the config within the environment and using one standard docker image (which sounds better).  This could be achieved with config maps, secrets, etc in kubernetes, for example.
  6. Processes – Stateless (stick sessions = bad).
  7. Port Binding – Back to the end of #2/dependencies – apps should be self-contained.  They should provide their own web server and should just expose a port themselves as-is and should not depend on anything else to do so.
  8. Concurrency – Build apps to scale out, not just up.  Of course you can have more threads/pools/etc… but if you can break up your application well, you can have separate apps for each purpose and each can scale at its own pace and use resources effectively.  E.g. if I have an app reading files and dumping records to Kafka, and then it reads the kafka records back out and sends them to an external service, it can be 2 separate apps.  Maybe the first part that reads files needs 4 copies with large memory to keep pace, and the second one needs 10 copies with high CPU and low memory to keep pace.   They can be  dealt with individually and easily once designed well (and this sort of thing is a cake walk when using something like kubernetes).
  9. Disposability“Servers are cattle, not pets”.  You don’t want a long lived server (like an app server) where people know its name and IP and how to debug it.  You want little, ephemeral, easily created and destroyed servers (or rather, containers).  In the cloud, you may use auto-scaling-groups / scale sets (AWS/Azure) to easily scale up and scale down copies of a server (machine image).  But even this isn’t great; it’s heavyweight and slow.  If you have an container orchestrator like kubernetes (or docker swarm, etc) in the cloud, or even on-prem, you can scale up and down docker/etc containers in seconds at large scale.  Either way, you should basically never need to look at a running machine image or a container unless you’re debugging something.  You can create new ones trivially and you don’t care about them.  They are disposable.
  10. Dev/Prod Parity – All your environments should be identical minus their configuration.  Use the same databases, same caches, same web server versions (bundling web-servers into code like spring-boot does helps with the last one).
  11. Logs – View logs as a continuous stream of events and store them in an external place like ELK or Splunk.  Preferably have an agent separate from your app be responsible for uploading the log stream; just let your app write to standard output.
  12. Admin Processes – Things like database pruning, backups, etc. should be treated as any other application you are running and they should be distinct applications separate from the application(s) they are aiding.  E.g. any of those processes should basically follow these steps themselves.

 

 

Checkpoint Linux SSL Network Extender VPN Auto Closes after Connecting.

Not much of a post here… but FYI for everyone – I’m running on Ubuntu Linux.  There is no checkpoint VPN client for Linux, so I have to go through a website and use their “SSL Network Extender”.

A fairly large portion of the time, this seems to hang/break under load (e.g. reading lots of database results).  It also randomly stops working periodically.  I haven’t been able to figure out why honestly.

Eventually, if I keep reconnecting to it, I get into a situation where it auto-closes right after it connects.  I couldn’t fix this until I restarted my laptop.

Anyway, I just figured out that disabling my wireless card and re-enabling it also fixes that issue, and that is much, much faster.  So,  until we figure out the root cause for the other issues, I hope this helps you too!