Pierre Beitz

Dealing with CORS in a Development Environment Using a Reverse Proxy

Pierre Btz — Tue, 24 Jan 2017 00:00:00 GMT

No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://my-site' is therefore not allowed access

As a front-end developer, there are chances you ended up with this message in your console quite a few time. This article will present a solution I used on both professional and personal developments to be able to deal with the CORS. It will also come back on other possible solutions I experimented but I was not satisfied with.

What is CORS anyway?

Same-origin policy

The same-origin policy is a security implemented in browsers that makes sure that a web page A, served from a domain A cannot access ressources from another domain than domain A. For more details, head to the w3c wiki.

CORS

CORS stands for Cross Origin Resource Sharing. They allow a CORS compatible browser (any modern browser) to make cross domain requests to a compatible reverse proxy or a compatible application.

This is not a security feature!

In fact, it relaxes a security feature. As you might have noticed, both ends of the channel need to be compatible: the browser on one side, the server (reverse proxy or application) on the other side.

With the CORS mechanism, the browser automatically adds control headers to the request. Depending on the type of request it can also make a preflight request. Request will be successful if the server’s answer contains a specific header allowing the domain. For more information, you might want to read Making Cross-Domain Requests with CORS.

Issue for the front-end developer

Now, here is the issue: I’m developing a web application that needs to query a RESTful API. The issue is that this RESTful API is not allowing access to my domain (let’s say http://localhost:4200 ).

Exploration of the first solutions

On the Application Side

CORS can be configured at the Application level (RESTful API in our case).

If I have a RESTful API developed with SpringBoot I can decide to set the CORS in SpringBoot using the @CrossOrigin annotation or using a Filter for instance.

This method has several issues:

Purely development stuff lands near production code. There is a chance, somebody, sometimes, will activate the change in production. As most of the time, the configuration will be to set a header accepting any domain ("Access-Control-Allow-Origin", "*"), I will let you decide if this is a risk worth taking.
It implies that you have access to the code of the API. If it is a public API developped by a third party, then you cannot apply it.

On the Reverse Proxy Side

A common scenario is that your server is behind a reverse proxy like nginx or haproxy. You can configure those reverse proxies for CORS.

Again, this method suffers from the same downsides as before.

The development reverse proxy

Now let’s study a purely client solution. If we fire up a reverse proxy on our client, then we can hide both the web application and the RESTfull API behind it. The browser will then always see one domain, therefore the same origin policy will be honored and there won’t be any CORS issue.

This solution has the following advantages:

No modification of any code, the reverse proxy is a development tool like npm or maven.
No need to have access to any server, or to the code of the API running on the server.
In a lot of case, you are in fact in the same conditions as with your production environment which also has a reverse proxy.

Hands On!

The solution relies on docker. It allows to run the reverse proxy as you would any tool. Of course, you could also use it without docker, but there is more work to make everything work…

You only need two files, a configuration file and a startup script. We will use nginx, but you can do the same with any reverse proxy.

Linux

First file you’ll need is the nginx configuration file, nginx.conf:

events {
    worker_connections 1024;
}

http {
    server {
        listen 80;

        location /api {
            proxy_pass http://127.0.0.1:8080/api/;
        }

        location / {
            proxy_pass http://127.0.0.1:3000/;

            # The following is for the websocket connection of the webpack dev server (https://gist.github.com/simongfxu/ea128160c296f31e41e6)
            proxy_redirect off;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
        }
    }
}

Of course you’ll need to adapt, this example is the one I use to develop a web application with a spring boot back end. The web application is served on port 3000. The back end is served on port 8080. What I’m telling nginx is that if the path of the request starts with /api, then the request should be sent to the backend, otherwise, it should be sent to the front end. The last section of the configuration is a special configuration I had to add so that the webpack dev server works correctly.

Now, let’s add a startup script:

#!/usr/bin/env bash

docker run --rm --name dev-nginx -p 80:80 --net="host" -v $PWD/nginx.conf:/etc/nginx/nginx.conf:ro  nginx:1.11.8-alpine

With this script, we are telling docker to boot up a container based on a minimal official nginx image. The configuration that nginx should use is bind mounted using the previous configuration file. The container is running on the same network stack as the host. Finally we are exposing the port 80.

Once everything is set up, you can start up your front end server, your backend server and finally your reverse proxy. Then direct your browser to localhost and you should be able to work.

OsX

Sadly, the previous configuration won’t work on OsX because basically the host mode of Docker is not working like on Linux.

The fastest solution I found was to use my hostname instead of localhost in the nginx.conf:

events {
    worker_connections 1024;
}

http {
    server {
        listen 80;

        location /api {
            proxy_pass http://hostname:8080/api/;
        }

        location / {
            proxy_pass http://hostname:3000/;

            # The following is for the websocket connection of the webpack dev server (https://gist.github.com/simongfxu/ea128160c296f31e41e6)
            proxy_redirect off;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
        }
    }
}

And the startup script is not using the host network mode:

#!/usr/bin/env bash

docker run --rm --name dev-nginx -p 80:80 -v $PWD/nginx.conf:/etc/nginx/nginx.conf:ro  nginx:1.11.8-alpine

Note that this method also implies that your server can bind to your IP (as you are using not using localhost anymore).

What about Windows?

I did not have the opportunity to test either solution on Windows. I’ll gladly edit this article with any feedback on Windows.

That’s it for this post. I hope it will help you configure your development environment. If you use any other method that you would like to share or discuss, please leave a comment!

Writing a Jenkinsfile: Tips and Tricks

Pierre Btz — Tue, 17 Jan 2017 00:00:00 GMT

Jenkins Pipeline are great. They allow the developer to keep their pipeline and their build near their code.

But developing a Pipeline can quickly become cumbersome:

The text area of the Jenkins UI is not really convenient when you start writing pipelines a bit more complex than your basic hello world.
Your code can grow and become hard to maintain (yeah, we end up having the same problems in our builds than in the rest of our code!).
Validation is not easy…

This post will sum up some technics I found useful to help me be efficient with pipelines. Of course the list is not exhaustive. I am more than interested by any new technic you would like to share in the comments.

Writing the Jenkinsfile

This part will be focused on the tools you can use to assist you. As already mentionned, the Jenkins UI is not really the best place to comfortably write a huge script. Nevertheless, never forget its existence. When I have doubts, I often create a quick dummy job to validate/invalidate some part of a script.

The IDE

Well, I won’t go into an IDE war, this is far from being the point of the article. Your favorite java IDE (with potentially a groovy support plugin) will do the trick.

That being said, you can do more than having a simple groovy IDE, you can also teach your IDE to understand the DSL used by Jenkins! This works thanks to the GroovyDSL (GDSL). A GDSL is basically a file that will describe the meta language that a DSL uses. By putting this file in the classpath of a groovy project, your IDE will be able to assist you write code that follows the DSL. The good news is that the Jenkins pipeline exposes a GDSL. Go to $jenkins_base_url/pipeline_syntax/gdsl to get it. Then add this file in the classpath of your project and voilà, your IDE can now autocomplete!

Note that I only tested this feature in Intellij, but this should work in other groovy compatible IDEs. Also, plugins might enrich the DSL without being described in the GDSL, so don’t expect the autocomplete to be absolutely complete.

The linter

During development of a pipeline, it’s easy to make a typo in your Jenkinsfile and end up with an invalide Jenkinsfile. This is a pain, as you lose time understanding the mistake, fix it, commit the fix and push it. The good news is that there is a way to shorten this feedback loop.

Jenkins offers a linter for your Jenkinsfile. Best way to use the linter is with the JenkinsCLI. You will need the Pipeline Model Definition version 0.7 at least. You can then ask Jenkins to lint your Jenkinsfile:

curl -O $jenkins_base_url/jnlpJars/jenkins-cli.jar
java -jar jenkins-cli.jar -s $jenkins_base_url login --username $username --password $password
java -jar jenkins-cli.jar -s $jenkins_base_url declarative-linter



The Pipeline Syntax

Even with the IDE assisting you, it is sometimes hard to correctly write what you wish to do:




Plugins are not exposed in the GDSL


The syntax might be cumbersome


Some parameters might seem obscure




There is a Pipeline Syntax page in Jenkins that is here to:




Help you understand the different options you have


Generate some code snippets for you




You can reach this page by browsing $jenkins_base_url/pipeline_syntax.
I won’t go into details, the usage of the page is pretty self explanatory, you choose the step you wish to generate, then you fill in the options as you would in a good old freestyle job (click on the quotemarks if you need assistance), and finally you click the Generate Pipeline Script button to get your snippet.
On the following screenshot, I generated a junit step.
Note the presence of the help button (right red arrow) that allowed me to quickly understand the 'Retain long standard output/error' (which finally translated into keepLingStdio in the code):







The final part of the Pipeline Syntax page (circled in red in the screenshot) is a reference to the many variables you have access to in the pipeline.


Like for the GDSL, plugin sadly do not always support that page.



Best Practices

CloudBees proposed a good article with ten best practices for Jenkins Pipeline Plugin.
This should help you getting started on the right track.



Organizing your pipeline


With pipelines, you can (and you should!) now treat your build as you treat your code!
This means for instance that you should keep your pipeline DRY.


Clean coding can now be applied to your pipelines!


Extracting functions

As for the rest of the code, try to extract methods and reuse your code.
Do not just copy and paste.
Of course functions will not help you from a pipeline to another, but it is a good starting point.
If you have the same function duplicated in multiple pipelines, it will help you to go to the next step.
If the logic is duplicated throughout your pipelines, the next step will take more time (and will be more error prone).



Pipeline Shared Libraries

The Pipeline Shared Libraries Plugin allows a developer to share and reuse code between pipelines.
I won’t go into details on how to use it here, it is worth a complete article, but here is a simple example that I encountered that will convince you of its power.
All our builds had to notify an SCM.
The only way to correctly catch all the cases (build success, build error), was by building inside a try/catch block.
So we ended up starting all our builds duplicating the following logic:



#!/usr/bin/groovy

try {
    notify(start)
    //put here all your pipeline
    notify(success)
} catch(exception) {
    notify(failure)
}

def notify(state) {
    //notify scm
}



As you can see, there is a lot of duplicated code between the builds, with a huge maintenance problem.


What is happening is the SCM API is changing for instance?
I would need to iterate over all the pipelines to update the code of the notify function.


Same issue if we can finally get rid of this try/catch structure (side note, that might soon be the case as we will see in the last part of this article).
I would have to manually change all the Jenkinsfiles.


With the Shared Libraries Plugin, I was able to enrich the DSL with our particular use case.
In the library I wrote:



//vars/notifiableBuild.groovy
#!/usr/bin/groovy

def call(body) {
    try {
        notify(start)
        body()
    	notify(success)
    } catch(exception) {
        notify(failure)
    }
}

def notify(state) {
    //notify scm
}



And finally, in a Jenkinsfile, I end up writing:



#!/usr/bin/groovy

notifiableBuild {
    //put here all your pipeline
}



To learn more about this, you should visit the official documentation of the plugin, which will help you getting started.





The Experimental Corner


In this part, I will share two projects you should keep an eye on as they should help a bit more spreading the usage of the pipelines.
Note that the two projects are at a very early stage of development.


Jenkins Blue Ocean Pipeline Editor Plugin

This plugin is using the Blue Ocean interface.
Side note, if you don’t know Blue Ocean, you should definitively have a look as it is far superior to the Jenkins 'classic' interface to display pipelines.
Note that it is also work in progress (it is in a beta stage), but I use it in production everyday without any issue (except for the missing features of course).


That being said, this new Pipeline Editor plugin proposes a new interface to graphically design your pipeline.
This is at very alpha stage, but it was demonstrated in the last Jenkins Meetup (note that the rest of the meetup is also worth your time if you are into Jenkins) and seems very promising.



Declarative Pipeline

The official name of this plugin is the Pipeline Model Definition Plugin.
Its aim is to give a more config like touch to the pipeline.
Code vs Config, here we are again (let’s not replay maven vs gradle and gulp vs grunt all over again…)!
Again, the plugin was demoed in the last Jenkins Online Meetup.
The API does not seem stable for the moment, but some stuff seems very interesting.
For instance, the notification issue I raised before could be adressed with something like:



#!/usr/bin/groovy

post {
    always {
    }

    success {
    	notify(success)
    }

    failure {
    	notify(failure)
    }
}



That’s it for this post, do not hesitate to post any comment you might have!



Docker starter kit part 3: Commits, Volumes and Data Management
Pierre Btz — Sun, 11 Dec 2016 00:00:00 GMT



In the first part of this blog series on Docker, you had the opportunity to follow a Docker basic tutorial and to have an overview of the lifecycle of a Docker container.
In the second part you then studied what a Docker Image and a Docker Container are made up.
This third post of the series will dive into how you can manage your data with Docker.




UnionFS and persistence


Thin Layer Lifecycle

If you remember correctly what we saw until now, an image is the union of multiple read only layers.
When we create a container, Dockers adds a new read/write thin layer.
Everything that happens during the life of the container occurs in this thin layer.
When the container is destroyed this thin layer is destroyed with it and everything that happened during the life of the container is lost.



Docker commit

Let me reassure you right now, there is a way to avoid losing everything: the docker commit command.
Basically what this command does is commit the thin layer into a new image, a bit like you would commit your staged files into your git repository.
To illustrate this, we will use the debian:jessie image we already encountered.
We will run it starting an interactive bash session:



docker run --name demo -it debian:jessie bash



We will get back to this on a later post, for the moment, let’s just say to simplify that we told Docker to attach to the container so that we can run commands interactively.
You should now have something like this:



root@:/#



You are ready to interact directly with your container.
Let’s create a simple dummy file to simulate data:



cd home
touch hello



Now, if you remember correctly, we just have created a file named hello in the thin layer of the container.
First let’s check that if we stop and restart the container the file is still here.
We will exit the container:



exit



The container should be stopped (you can check with docker ps -a), let’s restart it and attach to it:



docker start demo
docker exec -it demo bash



We are now attached to the container, let’s make sure our file is still here:



ls /home/



You should see hello.
Now, let’s exit the container again (exit) and instanciate a new container from debian:jessie:



docker run --name demo2 -it debian:jessie bash



Let’s check the content of the home directory of this container:



ls /home/



It’s empty as we expected.
This means that if we destroy the demo container, we lose our hello file.
Fortunately, the docker commit command is here to help us.
Let’s exit the demo2 container again.
Then we can commit our demo container:



docker commit demo dummyimage



That’s it, we have an image ready and we can create new containers which will contain our hello file.
I’ll let you make sure any container created from our dummyimage is containing the hello file.
You can also check the history of the image to see that we indeed commited a new layer:



IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
387fe6f1f01a        15 hours ago        bash                                            71 B
73e72bf822ca        4 weeks ago         /bin/sh -c #(nop)  CMD ["/bin/bash"]            0 B
           4 weeks ago         /bin/sh -c #(nop) ADD file:41ea5187c501168...   123 MB




The `docker commit` command is useful to commit changes you do not want to lose.
You can also leverage on it to move a custom container to another machine for instance.
However, for maintenability purpose, it is recommended to build your images using a Dockerfile as you saw in the first part of this series.
We will get back to the building of image using Dockerfile in more details later on.






Volumes


At this point, you might starting to understand that the UnionFS is not really what we wish for data persistence.
With it, data are bound to a container lifecycle.
You can quickly lose some data by just destroying a container.
Data sharing between containers is not easily possible either.
Now, volume comes into play.


Definition

A volume is a directory that bypasses the UnionFS.
A volume decouples the data from the lifecycle of the container.


Among its properties are:




A volume is initialized at container creation.


A volume can be shared and reused between different containers.


If you delete a container, its volumes are not deleted (except if you explicitely ask for their deletion).





Data Volumes


The following assumes you are using at least Docker v1.9.
This is the first release that is using named volumes.



Named volumes

Creation and Mounting

There are two ways of creating named volumes:




either explicitely with the docker volume create command.


or implicitely when creating a new container.




So both of the following will do the same:



## explicit method
docker volume create --name my_data
docker run -it --name my_container -v my_data:/data debian:jessie bash
## implicit method
docker run -it --name my_container -v my_data:/data debian:jessie bash



I’ll let you run one of the two methods (not both!).
You should be attached to a container.
We will then write a file in our volume:



touch /data/hello
exit



Then, let’s mount a new container.
This time, the my_data volume will be mounted under /home:



docker run -it --name my_container2 -v my_data:/home debian:jessie bash



In the container, let’s check that our volume is correctly mounted under /home:



ls /home
exit




Mounting volumes used by other containers

You can mount volumes used by other containers.
For instance, the following would mount a /data and a /home directory, both pointing to the my_data volume:



docker run -it --name my_container3 --volumes-from my_container --volumes-from my_container2 debian:jessie bash



In the container, we can check that both /data and /home contain the hello file:



ls /home
ls /data
exit




Be careful with what you do when manipulating volumes.
It may be not a good idea to mount the same volume on two different mount points in the same container!
Same thing with concurrent modifications: if you are mounting the same volume in different containers, you most probably want to check for concurrent modifications.




Mounting on an existing directory

If a container contains data in the mount point, then this data is copied onto the new volume.



Find the volume(s) used by a container

You can use the docker inspect --format "{{ .Mounts }}" command to locate the volume(s) used by a container.
Running this command on my_container3, we have:



[{ my_data /var/lib/docker/volumes/my_data/_data /data local  true }
 { my_data /var/lib/docker/volumes/my_data/_data /home local  true }
]




Removing a volume

One thing important to know, is that removing a container won’t remove its volumes (even if they are not used by any other container).
This is coherent with the fact that volumes should lived independently from containers.
So if we do the following:



docker rm my_container
docker rm my_container2
docker rm my_container3
docker volume ls | grep my_data



We see that we still have a our my_data volume.
To finally delete the volume:



docker volume rm my_data




Read only mode

Until now, we did not specify any option when mounting our volume.
It was mounted as read/write.
We could chose to mount it in read only mode:



docker run -it --name my_container -v my_data:/data:ro debian:jessie bash



In the container:



root@325a440f674b:/# touch /data/hello
touch: cannot touch '/data/hello': Read-only file system





Host directory mounting

Declare the mount

Docker also allows to mount a directory of your host.
You can mount a host directory by specifying a full path.
Create a file called hello, containing the string hello world on your host.
Then issue:



docker run -it --name host_mounting -v $PWD:/data debian:jessie bash




Carreful, if you are running under Windows or OsX, you need to let Docker access the shared directory by going into Docker options under the Shared Drives category.
Under Windows, you'll also need to specify the path manually as the PWD variable does not exist.



Once you booted up your container, we will install vim and then open the hello file:



apt-get update && apt-get install vim -y
cd /data
vim hello



You should see the hello world string.
Now edit the file and add something to it.
For those not familiar with vim, you want to type i to enter edition mode, then type something, then type esc and finally :wq.
Now, open the file on a text editor in your host to see the changes.



Read only mode

It works like volumes, check the previous section.



Find the volume(s) used by a container

It works like volumes, check the previous section.



Mounting on an existing directory

If a container contains data in the mount point, then this data is kept, and the host directory content overlays the data in the container mount point.



Mounting a file

Host mount also works with files. The previous example could have been writen:



docker run -it --name host_mounting -v $PWD/hello:/data/hello debian:jessie bash





About Data Containers

I made a choice to present you only named volumes and to avoid data containers.
Here is why: when you use data container, Docker really creates unnamed containers behind your back.
If you destroy your data container, the volume it was using become dangling and its management is not easy.


Here is an example that will show you a bit the usage of data containers and the limitation I was mentionning.


We will first run interactively a container with an unnamed volume:



docker run -it --name unnamed_volume -v /data debian:jessie bash



And in the container:



touch /data/hello
exit



Let’s inspect the container for mounted volumes:



docker inspect --format "{{ .Mounts }}" unnamed_volume



You end up with:



[{volume 6faa8a43f80bb550b11aa8f3f6093273782fbf5a5674fd5067ba7c7c27942314 /var/lib/docker/volumes/6faa8a43f80bb550b11aa8f3f6093273782fbf5a5674fd5067ba7c7c27942314/_data /data local  true }]



So you really have a volume stored.
Now, usually, what you would do to be able to reuse this data container from another container is to use the --volumes-from command:



docker run -it --name unnamed_volume2 --volumes-from unnamed_volume debian:jessie bash



Here unnamed_volume2 shares the same unnamed volume with unnamed_volume.
This volume is mounted under /data.
Let’s check this:



ls /data
exit



You should see our dummy hello file.
So far so good.
Why do I prefer avoiding this technic might you ask.
Well, let’s now delete both our containers after having taken care of remembering our unnamed volume id:



docker inspect --format "{{ .Mounts }}" unnamed_volume
docker rm unnamed_volume
docker rm unnamed_volume2



And let’s look for our unnamed volume:



docker volume ls | grep 



It’s still there!
Of course, you could still mount it by its id, to check what it contains, but it’s far from perfect from a maintenance point of view.



You could also have issued a `docker rm -v` command to delete any anonymous volumes.



I’m not saying that unnamed volumes and data containers might not be useful sometimes, but most of the case you will probably end up using named containers.
So, unless you are stuck with a pre 1.9 version of Docker, I strongly suggest you consider named volumes first.
In any case, you might want to review the Manage data in containers page of the official documentation.


That’s it for this post, next one will show you how you could back up and restore your data, how you can extend volumes with drivers, and  a practical usage of the mounting of a host directory.







Docker starter kit part 2: Anatomy of an Image and a Container
Pierre Btz — Sun, 04 Dec 2016 00:00:00 GMT



In the first part of this blog series on Docker, you had the opportunity to follow a Docker basic tutorial and to have an overview of the lifecycle of a Docker container.
In this post, we will have an overview on what a Docker Image and a Docker Container are made up.




Anatomy of an image


Union File Systems


Union file systems, or UnionFS, are file systems that operate by creating layers, making them very lightweight and fast. Docker uses union file systems to provide the building blocks for containers.


— Docker Official Documentation



We will see next how those UnionFS are exploited in Docker.



UnionFS and Docker Images

A Docker image is made of several layers.
Each layer represents a filesystem difference with the layer below it.
The stack of all the layers makes up the filesystem of an image.
Each layer is read only and shared between the images (and the containers, as we will see in the next part).


As you may have guessed, all of this means that you need to be very careful when building new images to maximize the reuse of layers.
We will discuss this in details in another blog post of this series.



Hands On!

Let us take a concrete example.
Let us pull a maven image:



docker pull maven:3.3.9-jdk-8



Do not bother with the output of the command yet, we will get back to it later.
Instead, let’s now use the history commands that displays all the layers of an image:



docker history maven:3.3.9-jdk-8



And the output:



IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
b2ef69e414f5        3 weeks ago         /bin/sh -c #(nop)  CMD ["mvn"]                  0 B
           3 weeks ago         /bin/sh -c #(nop)  ENTRYPOINT ["/usr/local/bi   0 B
           3 weeks ago         /bin/sh -c #(nop)  VOLUME [/root/.m2]           0 B
           3 weeks ago         /bin/sh -c #(nop) COPY file:b3fc14e8337e0079a   327 B
           3 weeks ago         /bin/sh -c #(nop) COPY file:e3aa30a24fcf60a59   1.042 kB
           3 weeks ago         /bin/sh -c #(nop)  ENV MAVEN_CONFIG=/root/.m2   0 B
           3 weeks ago         /bin/sh -c #(nop)  ENV MAVEN_HOME=/usr/share/   0 B
           3 weeks ago         |2 MAVEN_VERSION=3.3.9 USER_HOME_DIR=/root /b   10.03 MB
           3 weeks ago         /bin/sh -c #(nop)  ARG USER_HOME_DIR=/root      0 B
           3 weeks ago         /bin/sh -c #(nop)  ARG MAVEN_VERSION=3.3.9      0 B
           3 weeks ago         /bin/sh -c /var/lib/dpkg/info/ca-certificates   418.2 kB
           3 weeks ago         /bin/sh -c set -x  && apt-get update  && apt-   351.5 MB
           3 weeks ago         /bin/sh -c #(nop)  ENV CA_CERTIFICATES_JAVA_V   0 B
           3 weeks ago         /bin/sh -c #(nop)  ENV JAVA_DEBIAN_VERSION=8u   0 B
           3 weeks ago         /bin/sh -c #(nop)  ENV JAVA_VERSION=8u111       0 B
           3 weeks ago         /bin/sh -c #(nop)  ENV JAVA_HOME=/usr/lib/jvm   0 B
           3 weeks ago         /bin/sh -c {   echo '#!/bin/sh';   echo 'set    87 B
           3 weeks ago         /bin/sh -c #(nop)  ENV LANG=C.UTF-8             0 B
           3 weeks ago         /bin/sh -c echo 'deb http://deb.debian.org/de   55 B
           3 weeks ago         /bin/sh -c apt-get update && apt-get install    1.287 MB
           3 weeks ago         /bin/sh -c apt-get update && apt-get install    122.6 MB
           3 weeks ago         /bin/sh -c apt-get update && apt-get install    44.3 MB
           3 weeks ago         /bin/sh -c #(nop)  CMD ["/bin/bash"]            0 B
           3 weeks ago         /bin/sh -c #(nop) ADD file:41ea5187c50116884c   123 MB



That is a lot of layers right?
Well, let’s try to sort this out.
First let’s review the Dockerfile used to build this image, available on github:



FROM openjdk:8-jdk

ARG MAVEN_VERSION=3.3.9
ARG USER_HOME_DIR="/root"

RUN mkdir -p /usr/share/maven /usr/share/maven/ref \
  && curl -fsSL http://apache.osuosl.org/maven/maven-3/$MAVEN_VERSION/binaries/apache-maven-$MAVEN_VERSION-bin.tar.gz \
    | tar -xzC /usr/share/maven --strip-components=1 \
  && ln -s /usr/share/maven/bin/mvn /usr/bin/mvn

ENV MAVEN_HOME /usr/share/maven
ENV MAVEN_CONFIG "$USER_HOME_DIR/.m2"

COPY mvn-entrypoint.sh /usr/local/bin/mvn-entrypoint.sh
COPY settings-docker.xml /usr/share/maven/ref/

VOLUME "$USER_HOME_DIR/.m2"

ENTRYPOINT ["/usr/local/bin/mvn-entrypoint.sh"]
CMD ["mvn"]



When you build an image, you obviously build the layers from bottom to top.
So if we read the Dockerfile from bottom to top, we should find the list of commands that the history command output:



IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
b2ef69e414f5        3 weeks ago         /bin/sh -c #(nop)  CMD ["mvn"]                  0 B
           3 weeks ago         /bin/sh -c #(nop)  ENTRYPOINT ["/usr/local/bi   0 B
           3 weeks ago         /bin/sh -c #(nop)  VOLUME [/root/.m2]           0 B
           3 weeks ago         /bin/sh -c #(nop) COPY file:b3fc14e8337e0079a   327 B
           3 weeks ago         /bin/sh -c #(nop) COPY file:e3aa30a24fcf60a59   1.042 kB
           3 weeks ago         /bin/sh -c #(nop)  ENV MAVEN_CONFIG=/root/.m2   0 B
           3 weeks ago         /bin/sh -c #(nop)  ENV MAVEN_HOME=/usr/share/   0 B
           3 weeks ago         |2 MAVEN_VERSION=3.3.9 USER_HOME_DIR=/root /b   10.03 MB
           3 weeks ago         /bin/sh -c #(nop)  ARG USER_HOME_DIR=/root      0 B
           3 weeks ago         /bin/sh -c #(nop)  ARG MAVEN_VERSION=3.3.9      0 B



It seems correct, up until the FROM openjdk:8-jdk command of the Dockerfile.
So, what about those remaining layers?



IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
           3 weeks ago         /bin/sh -c /var/lib/dpkg/info/ca-certificates   418.2 kB
           3 weeks ago         /bin/sh -c set -x  && apt-get update  && apt-   351.5 MB
           3 weeks ago         /bin/sh -c #(nop)  ENV CA_CERTIFICATES_JAVA_V   0 B
           3 weeks ago         /bin/sh -c #(nop)  ENV JAVA_DEBIAN_VERSION=8u   0 B
           3 weeks ago         /bin/sh -c #(nop)  ENV JAVA_VERSION=8u111       0 B
           3 weeks ago         /bin/sh -c #(nop)  ENV JAVA_HOME=/usr/lib/jvm   0 B
           3 weeks ago         /bin/sh -c {   echo '#!/bin/sh';   echo 'set    87 B
           3 weeks ago         /bin/sh -c #(nop)  ENV LANG=C.UTF-8             0 B
           3 weeks ago         /bin/sh -c echo 'deb http://deb.debian.org/de   55 B
           3 weeks ago         /bin/sh -c apt-get update && apt-get install    1.287 MB
           3 weeks ago         /bin/sh -c apt-get update && apt-get install    122.6 MB
           3 weeks ago         /bin/sh -c apt-get update && apt-get install    44.3 MB
           3 weeks ago         /bin/sh -c #(nop)  CMD ["/bin/bash"]            0 B
           3 weeks ago         /bin/sh -c #(nop) ADD file:41ea5187c50116884c   123 MB



Let’s have a look at the content of the Dockerfile used to build openjdk:8-jdk (I removed the comments to the readibility):



FROM buildpack-deps:jessie-scm
RUN apt-get update && apt-get install -y --no-install-recommends \
		bzip2 \
		unzip \
		xz-utils \
	&& rm -rf /var/lib/apt/lists/*

RUN echo 'deb http://deb.debian.org/debian jessie-backports main' > /etc/apt/sources.list.d/jessie-backports.list
ENV LANG C.UTF-8
RUN { \
		echo '#!/bin/sh'; \
		echo 'set -e'; \
		echo; \
		echo 'dirname "$(dirname "$(readlink -f "$(which javac || which java)")")"'; \
	} > /usr/local/bin/docker-java-home \
	&& chmod +x /usr/local/bin/docker-java-home

ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64

ENV JAVA_VERSION 8u111
ENV JAVA_DEBIAN_VERSION 8u111-b14-2~bpo8+1
ENV CA_CERTIFICATES_JAVA_VERSION 20140324

RUN set -x \
	&& apt-get update \
	&& apt-get install -y \
		openjdk-8-jdk="$JAVA_DEBIAN_VERSION" \
		ca-certificates-java="$CA_CERTIFICATES_JAVA_VERSION" \
	&& rm -rf /var/lib/apt/lists/* \
	&& [ "$JAVA_HOME" = "$(docker-java-home)" ]

RUN /var/lib/dpkg/info/ca-certificates-java.postinst configure



If you read the Dockerfile from bottom to top, you’ll match every command with a layer.
We could continue again with the Dockerfile of the next base image (buildpack-deps:jessie-scm) but I guess you get the point.



A Docker image is a stack of layers.
When you build on top of a base image, you are stacking layers on top of all the layers of the base image.




Base image

At this point, you might wonder which is the mother of all images.
If you continue working up the base images from the previous example, you will get to the debian:jessie image:



FROM scratch
ADD rootfs.tar.xz /
CMD ["/bin/bash"]



The scratch image is a reserved minimal image used to signal Docker that you are building an image from scratch.
The next command of your Dockerfile will be the first layer of your container.
The scratch image has a special status in Docker.
You cannot pull it, push it or tag an image with the name.
Building a base image is clearly out of the scope of this blog post.
If you wish to build a base image, you can have a look to the official documentation on base images.



Back to the output of docker pull

Remember, I told you that we would get back to the output of docker pull maven:3.3.9-jdk-8.
My output looked like this:



3.3.9-jdk-8: Pulling from library/maven

386a066cd84a: Already exists
75ea84187083: Already exists
88b459c9f665: Already exists
690dbea0e4ca: Already exists
7e401cdd6f18: Already exists
a58186ddf9a0: Already exists
49999ed55bc4: Already exists
eb40561aad8f: Already exists
4ce0e24588f2: Pull complete
35430242cb99: Pull complete
d4af041dcf95: Pull complete
Digest: sha256:a7fd540bc273b7c4f1193fbcd46127ad3912fd095a251382f4e4312b9ac85e9d
Status: Downloaded newer image for maven:3.3.9-jdk-8



The output of the docker pull shows you all the image layers that the maven:3.3.9-jdk-8 image is made of.
If you compare this output to the output of the docker history command, you will notice that we have 11 image layers here for 24 layers in the image.
This is due to the fact that during the build process, docker is able to make optimizations by squashing layers into an image.



The previous statement is right since Docker 1.10.
Before version 1.10, there was a 1:1 ratio between image layers and layers.



Of course this does not change anything regarding what you download.
You won’t have to download any image if you wish to download the openjdk:8-jdk image for instance:



openjdk:8-jdk: Pulling from library/openjdk

386a066cd84a: Already exists
75ea84187083: Already exists
88b459c9f665: Already exists
690dbea0e4ca: Already exists
7e401cdd6f18: Already exists
a58186ddf9a0: Already exists
49999ed55bc4: Already exists
eb40561aad8f: Already exists
Digest: sha256:dd0fc686a5584c0c7f3e50dd84ddc42fae400c27a21d8ca98dad190aff5e9d52
Status: Downloaded newer image for openjdk:8-jdk






Anatomy of a container


Now, how does the previous apply to containers?
Well, it is pretty simple actually.
When you create a container, Docker will add a new layer on top of the stack of layers that makes up the image.
Contrary to the image layers, this layer is writable.


If you decide to create another container from the same image, a new writable layer will be created, the rest of the layer stack will be shared.


This diagram from the official documentation pretty much sums it up:







If you wish to go further in your understanding of containers, particurlarly on the topic of the writing strategies in containers, I suggest this page of the official documentation.


That is it for this post, you can jump to the next one Docker starter kit part 3: Commits, Volumes and Data Management





Docker starter kit part 1: Getting Started, Container's lifecycle
Pierre Btz — Sun, 27 Nov 2016 00:00:00 GMT



Docker is a major change in the way we develop software.
But it can be a bit challenging for a developer to enter the container world.
This series of articles is an answer to a developer wanting to onboard the Docker train.
It is not designed to be a complete docker training (these already exist), but rather a starter kit to quickly get up to speed.


There is not a fixed number of parts for this series as I am not sure yet how far I want to go (for instance I am not sure yet if I will talk about Docker compose or Docker Swarm).
Of course, do not hesitate to point any mistake or omission.




Getting Started


OS

OS does not matter anymore.
Docker is now available natively on Windows 10 (from 1511 November update, Build 10586), OSX and Linux.
I use it without any problem on OSX and a debian at work, and on Windows 10 and a Linux Mint at home.
What I recommand though is to avoid the non native solutions for older OSX and Windows 7/10.
My experience with them was just awful: a lot of issues, of instabilities as well as some strange behaviors.



Let’s go!

Well, this part will be pretty simple.
Go to the official page, and follow the official tutorial:




Installation


Images and Containers


Run your first image


Build your first image


Create your Docker Hub account this part can be skipped if you have access to a private repository


Push and Pull an image if you are using a private repository, the name of the image becomes $private_repo/docker-whale




In the following, I will assume you have the basic understanding that the tutorial gave you.


EDIT (04/12/2016): You can also have a look to the blog of Johnny Mkhael who wrote a Hello Docker blog series.





Lifecycle of a container


Overview

A Docker Container can have multiple states.
Here is a complete list of the different states as well as the associated commands from the official documentation:







As you can see, there are multiple state possible.



Main States

The main states are:




created: your container has been created.
This means you have created an instance of an image.
You get to this state by running the command docker create










Remember what you saw in the tutorial, an image does not have a state and never changes.







running: The container is running which means it has been created (docker create) and started (docker start).
You can also directly reach this state from an image by running docker run.


paused: A paused container is in a frozen state but its main process has not been stopped (the SIGTERM signal was not sent from docker to PID1 process).
The container has not released its ressources (eg memory).
From a running container you reach this state with the docker pause.
It can be unfrozen using the docker unpause.


stopped: When stopping a container, the SIGTERM signal has been sent from docker to the main process.
After a waiting time (10 seconds by default), the main process will receive a SIGKILL signal.
You can stop a container using docker stop.
NB: You can also directly send a SIGKILL signal using docker kill.
It is probably not a good idea if you wish to restart your container.
You can move from a stopped container to a running container using the docker start command.




That is it for this post, you can jump to the next one: Docker starter kit part 2: Anatomy of an Image and a Container






Running fish on Windows 10
Pierre Btz — Sun, 09 Oct 2016 00:00:00 GMT



You may already know that the latest Windows 10 builds offer a bash in Windows. This is a great feature. But what is even more interesting is that Microsoft did not do an emulation of linux commands but they partnered up with Canonical instead and integrated a real Ubuntu image on top of Windows. This means that you can run commands like…apt-get in this bash.




'Any Shell for Windows'


In other words, this means that potentially, the 'bash for Windows' feature, could be understood like an 'any shell for Windows' feature! I decided to give it a try with fish. Fish is an amazing shell a colleague of mine showed me some time ago. I am now using it every day at work and I was missing it at home on my Windows machine. It is really simple to use, it has a powerful autosuggestion engine, and it is easily extensible with scripts…and a lot of people distribute great scripts!




How to install fish?


I will assume here that you already know how to install bash for Windows. If not, there are plenty of blogs that have detailed step by step guides. Once you are running bash, it is simply a matter of following the fish installation instruction for Ubuntu, that is to say:



sudo add-apt-repository ppa:fish-shell/nightly-master
sudo apt-get update
sudo apt-get install fish



Now you can just type fish in your prompt, and you are done. Unfortunately, next time you will launch an Unix shell from Windows it will have to be a bash. Good news is that it is easy to customize bash startup with the .bashrc file. To do this, first go to your unix home. Be careful, when you launch a bash, Windows puts you under your Windows' home (/mnt/c/Users/YourUser) and not under Unix’s home (~). To be sure, type:



cd
nano .bashrc



Then add a the beginning of your .bashrc file the following:



if [ -t 1 ]; then
  exec fish
fi



Be careful not to just exec fish, you could break some programs and scripts relying on bash.


EDIT(04/12/2016): Despite the following script in my .bashrc, I had issues with scripts relying on bash.
I found another strategy which was to simply make a shortcut to launch fish.
To do this, you just have to locate the shortcut you have to launch bash, make a copy of this shortcut and edit it:




target: Replace C:\Windows\System32\bash.exe ~ by C:\Windows\System32\bash.exe -c /usr/bin/fish.


start in: Set your preferred starting point. For the windows home set %userprofile%.




Be sure to remove the modifications of the .bashrc as they are now useless (and maybe not harmless…).




Scripting


I installed fisherman to manage the scripts of my fish installation. No ad intended, I don’t have enough feedback on it to give advice yet. Just know that there are other plugin managers out there, like oh my fish. Here again, no need to adapt the installation instructions, everything worked as it would on Unix. For instance, I decided to install a useful teleporter script that allows to set labels on directories and jump back and forth:



#install fisherman
curl -Lo ~/.config/fish/functions/fisher.fish --create-dirs git.io/fisher
#install the tp script
fisher install simonrelet/tp-fish-function



And here is the result:







I hope this was helpful. I’m still new to fish, feel free to drop a comment with tips or useful scripts! Of course, the main focus of the article was on fish, but I’m pretty sure that you can run any Unix shell you like using the same method.





Welcome Blogging World!
Pierre Btz — Sun, 02 Oct 2016 00:00:00 GMT



The idea of blogging has been making its way in my head for some time now. I’m already used to blogging on the internal blog of my company, but the audience is really small…and so is the feedback.




Who am I?


By day, I am a UI developer working on multiple stacks going from Swing to Angular 2.0. I am also in charge of the Continuous Integration in my team.


I am interested in a lot of development topics, especially the subjects of Continuous Integration, Continous Delivery, Testing and Clean Coding.




What to expect from this blog?


The main idea behind this blog is to share some thoughts about my readings, or the conferences I saw. I would also like to share some of my experiment. As the title suggests, I see this blog as a notepad, for my future self…and for anyone who might have some interest on what I have to say.