CICD with Docker Swarm Mode using your PC or Cloud - Part-1

Introduction:

As of June 20, 2016 as DockerCon 2016, Docker has yet again won the heart of millions by releasing Docker Swarm Mode which has all the required functionalities of a full-fledged container clustering and orchestration! 

With Swam Mode, all of the popular Swarm Clustering ecosystem is now simply a part of single/same old) Docker Engine with no dependence on any 3rd party software! And still compatible with all the existing Docker CLI and Tooling (ex: docker cli, docker compose, dokku etc. etc.). 

As always Docker focused on following three features for Swarm Mode:

• Distributed: Swarm Mode uses the Raft Consensus Algorithm in order to coordinate and does not rely on a single point of failure to perform decisions.
• Secure: Node communication and membership within a Swarm are secure out of the box. Swarm Mode uses mutual TLS for node authentication, role authorization and transport encryption, automating both certificate issuance and rotation.
• Simple: Swarm Mode is operationally simple and minimizes infrastructure dependencies. It does not need an external database to operate. It uses internal distributed State store.

With Docker Swarm Mode, I am sure it'd be much more favorable as compare to Kubernetes (another popular open-source container clustering and orchestration system), which rather uses Docker itself as container and most of the 3rd party systems created by CoreOS community (ex: etcd as service discovery, fluentd as clustering etc.) along with it's own ecosystem!

Excited about Swarm Mode? Let's see how can we make 1-click deployment or CICD with Docker Swarm Mode and Go Pipelines using your own PC or Cloud! Here we go...

Software	My Setup	Other Options	Source
Docker 1.12+	MacBook OSX*, VirtualBox	Windows* PC, Cloud instances (AWS, Azure, Google etc)	Mac: https://docs.docker.com/docker-for-mac/ Windows: https://docs.docker.com/docker-for-windows/ Linux: https://www.docker.com/products/docker#/servers
Go Pipeline Server	MacBook OSX	Windows PC, Cloud instances (AWS, Azure, Google etc)	https://www.go.cd/download/
Oracle VirtualBox	MacBook OSX	Windows PC, Cloud instances (AWS, Azure, Google etc)	https://www.virtualbox.org/wiki/Downloads

* As of Aug 2016: Docker version 1.12+ is in Beta for OSX and Windows.

PART-1 (Installation, Setup and Testing of Docker Swarm Mode Cluster)

Installation: 

Step-1: Download and Install Docker 1.12+

Check that it works and is above version 1.12+.

nverma@macbook-pro$ docker version
Client:
 Version:      1.12.0
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   8eab29e
 Built:        Thu Jul 28 21:04:48 2016
 OS/Arch:      darwin/amd64
 Experimental: true

Server:
 Version:      1.12.0
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   8eab29e
 Built:        Thu Jul 28 23:54:00 2016
 OS/Arch:      linux/amd64

Step-2: Download and install "docker-machine"

This is required to create and manage Virtual Machines (on the Oracle VirtualBox) which would act as Docker nodes. I am using it to crate VirtualBox VMs locally on my Mac but you can use it to directly create VMs on Cloud from your PC (ex: AWS EC2 instance).

1. curl -L https://github.com/docker/machine/releases/download/v0.8.0-rc2/docker-machine-`uname -s`-`uname -m` >/usr/local/bin/docker-machine
2.  chmod +x /usr/local/bin/docker-machine
   

• Check if it is installed properly:

nverma@macbook-pro$ docker-machine version
docker-machine version 0.8.0, build b85aac1
nverma@macbook-pro$

Step-3: Download and Install VirtualBox

I have version 5.0.20 r106931 on my Mac, but it doesn't matter, try and use latest version!

Setup:

Step-1: Create required docker machines

We will create 3 machines (can work with 2 too) where we will call one as master and others as workers


nverma@macbook-pro$ docker-machine create -d virtualbox master
Running pre-create checks...
Creating machine...
(master) Copying /Users/nverma/.docker/machine/cache/boot2docker.iso to /Users/nverma/.docker/machine/machines/master/boot2docker.iso...
(master) Creating VirtualBox VM...
(master) Creating SSH key...
(master) Starting the VM...
(master) Check network to re-create if needed...
(master) Waiting for an IP...
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with boot2docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Checking connection to Docker...
Docker is up and running!
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env master

• Similarly create other required VMs.

 nverma@macbook-pro$ docker-machine create -d virtualbox worker1
...
 nverma@macbook-pro$ docker-machine create -d virtualbox worker2
...

• Check VMs

$ docker-machine ls
NAME      ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER    ERRORS
master    -        virtualbox   Running   tcp://192.168.99.101:2376           v1.12.0  
worker1   -        virtualbox   Running   tcp://192.168.99.102:2376           v1.12.0  
worker2   -        virtualbox   Running   tcp://192.168.99.103:2376           v1.12.0  

Step-2: Setup Swarm Mode - Cluster

Docker has made it so simple to setup the Swarm Cluster with their Swarm Mode that it is as simple as 1-2-3... 

Initialize the machine named "master" as swarm master.

Set environment:

nverma@macbook-pro$ eval $(docker-machine env master)
 
Initialise the Swarm with the external IP of the master VM:

nverma@macbook-pro$ docker swarm init --advertise-addr 192.168.99.101

Swarm initialized: current node (bo83ch825l2t1mbo8n6z4xkxj) is now a manager.

To add a worker to this swarm, run the following command:
    docker swarm join \
    --token SWMTKN-1-4zqhbw2mylo2b9ajaix9n06nmjxjx00wlvzv8kyv22olyx4ls1-0h27p8ndvzlyyzdcyrmlm5iz6 \
    192.168.99.101:2377

To add a manager to this swarm, run the following command:
    docker swarm join \
    --token SWMTKN-1-4zqhbw2mylo2b9ajaix9n06nmjxjx00wlvzv8kyv22olyx4ls1-99v82agtft162d4i6qga0omlq \
    192.168.99.101:2377 


Swarm cluster is ready! Now join the workers/nodes into this cluster 


nverma@macbook-pro$ eval $(docker-machine env worker1)
nverma@macbook-pro$ docker swarm join \
  --token SWMTKN-1-4zqhbw2mylo2b9ajaix9n06nmjxjx00wlvzv8kyv22olyx4ls1-0h27p8ndvzlyyzdcyrmlm5iz6 \
  192.168.99.101:2377
 
This node joined a swarm as a worker.

Repeat same on the 3rd mode/worker2. And that's it, my complete 3 nodes Swarm Cluster is ready to rock!

This is what I have now:

nverma@macbook-pro$ eval $(docker-machine env master)nverma@macbook-pro$ docker node ls
ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
6rarliq49ehaoi5olk85ztpfe    worker1   Ready   Active       
7i4s01v67u3faa1nof8nzf2g0    worker2   Ready   Active       
bo83ch825l2t1mbo8n6z4xkxj *  master    Ready   Active        Leader

Step-3: Test run your first service

Following steps should help you get some flavor of the swarm mode.

• Create a new service:

nverma@macbook-pro$ docker service ls
ID  NAME  REPLICAS  IMAGE  COMMAND

nverma@macbook-pro$ docker service create --replicas 1 --name mypingapp alpine ping google.com
8qjvnp8efw4fw1uhmfe388fbk

nverma@macbook-pro$ docker service ls
ID            NAME       REPLICAS  IMAGE   COMMAND
8qjvnp8efw4f  mypingapp  0/1       alpine  ping google.com


Here name of my service/application is "mypingapp", it will use a very small "alpine" image from the Docker Hub Repository and will run "ping google.com" command during start.

• Get more details:

nverma@macbook-pro$ docker service ps mypingapp
ID                         NAME         IMAGE   NODE    DESIRED STATE  CURRENT STATE          ERROR
ed1m1atgwm250brc9lvxlgi6h  mypingapp.1  alpine  master  Running        Running 3 seconds ago
nverma@macbook-pro$


It shows that my service is running 1 container as desired and running on the "master" node.

 

• Check logs:

nverma@macbook-pro$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
652c368e4316        alpine:latest       "ping google.com"        8 seconds ago       Up 5 seconds                                 mypingapp.1.ed1m1atgwm250brc9lvxlgi6h

nverma@macbook-pro$ docker logs 652c368e4316
PING google.com (172.217.25.46): 56 data bytes
64 bytes from 172.217.25.46: seq=0 ttl=61 time=11.134 ms
64 bytes from 172.217.25.46: seq=1 ttl=61 time=10.046 ms
.....

• Inspect your service:

nverma@macbook-pro$ docker service inspect --pretty mypingapp

• Scale your service (to handle more load or parallel processing) :
  

nverma@macbook-pro$ docker service scale mypingapp=3
mypingapp scaled to 3

 nverma@macbook-pro$ docker service ls
ID            NAME       REPLICAS  IMAGE   COMMAND
8qjvnp8efw4f  mypingapp  3/3       alpine  ping google.com

nverma@macbook-pro$ docker service ps mypingapp
ID                         NAME         IMAGE   NODE     DESIRED STATE  CURRENT STATE           ERROR
ed1m1atgwm250brc9lvxlgi6h  mypingapp.1  alpine  master   Running        Running 11 minutes ago
4k9ek6cejk36a4gxgic3cqgmr  mypingapp.2  alpine  master   Running        Running 10 seconds ago
bzv3s3td44tz03uqd6imcxm1u  mypingapp.3  alpine  worker2  Running        Running 8 seconds ago  


Once you’re happy, remove your service and it's the time to create some more meaningful service and move to PART-2.

nverma@macbook-pro$ docker service rm mypingapp