Monday 29 August 2016

CICD with Docker Swarm Mode using your PC or Cloud - Part-2

Part-2 (Installation, Setup and Testing of Java SpringBoot Application in Docker Container)

In the Part-1 we setup the latest Docker Swarm Mode cluster and run basic service. In this part we will build a docker image for a Spring Boot java app and will push the image stored in a local docker repository.






Step-2: Build a Java Springboot App and Dockerise it 

  • Get  a sample Java Springboot application

Clone a sample Java Springboot application from my github site:
https://github.com/n1t1nv3rma/springboot-maven.git

nverma@macbook-pro$ pwd
/Users/nverma/Documents/test


nverma@macbook-pro$ls -l
nverma@macbook-pro$


nverma@macbook-pro$ git clone https://github.com/n1t1nv3rma/springboot-maven.git
Cloning into 'springboot-maven'...
remote: Counting objects: 50, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 50 (delta 0), reused 0 (delta 0), pack-reused 47
Unpacking objects: 100% (50/50), done.
Checking connectivity... done.
 

nverma@macbook-pro$ ls -l
total 0
drwxr-xr-x  6
nverma  staff  204 24 Aug 22:54 springboot-mavennverma@macbook-pro$ cd springboot-maven/nverma@macbook-pro$ ls -l
total 16
-rw-r--r--  1
nverma  staff    18 24 Aug 22:54 README.md
-rw-r--r--  1
nverma  staff  2188 24 Aug 22:54 pom.xml
drwxr-xr-x  3
nverma  staff   102 24 Aug 22:54 srcnverma@macbook-pro$

If you are familiar with Maven, then you can notice that in the pom.xml file (which is a build file for Maven) is using "spring-boot-maven-plugin" and "docker-maven-plugin" plugins. 

As per name the "spring-boot-maven-plugin" is responsible for building my spring boot app and the "docker-maven-plugin" is responsible for packagin the spring boot app into a Docker image based upon my Dockerfile.

The source code for sample Spring Boot application is under the "src/main/java" directory.

Dockerfile is listed under src/main/docker/Dockerfile. Here are it's contents

nverma@macbook-pro$ cat src/main/docker/Dockerfile 
FROM frolvlad/alpine-oraclejdk8:slim
VOLUME /tmp
ADD gs-spring-boot-docker-0.1.0.jar app.jar
RUN sh -c 'touch /app.jar'
ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]


So what it has is to use the standard "alpine-oraclejdk8:slim" image from the Docker hub repo. and add our Spring Boot app (a JAR file) into it and run it via Java.


  • Build the application and docker image
nverma@macbook-pro$ pwd
/Users/nverma/Documents/test/springboot-maven

nverma@macbook-pro$ ls -l
total 16
-rw-r--r--  1 nverma  staff    18 24 Aug 22:54 README.md
-rw-r--r--  1 nverma  staff  2188 24 Aug 22:54 pom.xml
drwxr-xr-x  3 nverma  staff   102 24 Aug 22:54 src


Set the Docker env. to point to the “master” VM i.e. the Swarm Mode Cluster master.

nverma@macbook-pro$ eval $(docker-machine env master)

nverma@macbook-pro$ mvn package docker:build
[INFO] Scanning for projects...
[INFO]                                                                       
[INFO] ------------------------------------------------------------------------
[INFO] Building gs-spring-boot-docker 0.1.0
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ gs-spring-boot-docker ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Copying 0 resource
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ gs-spring-boot-docker ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 3 source files to /Users/nverma/Documents/test/springboot-maven/target/classes
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ gs-spring-boot-docker ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /Users/nverma/Documents/test/springboot-maven/src/test/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ gs-spring-boot-docker ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-surefire-plugin:2.18.1:test (default-test) @ gs-spring-boot-docker ---
[INFO] No tests to run.
[INFO]
[INFO] --- maven-jar-plugin:2.6:jar (default-jar) @ gs-spring-boot-docker ---
[INFO] Building jar: /Users/nverma/Documents/test/springboot-maven/target/gs-spring-boot-docker-0.1.0.jar
[INFO]
[INFO] --- spring-boot-maven-plugin:1.4.0.RELEASE:repackage (default) @ gs-spring-boot-docker ---
[INFO]
[INFO] --- docker-maven-plugin:0.4.11:build (default-cli) @ gs-spring-boot-docker ---
[INFO] Copying /Users/nverma/Documents/test/springboot-maven/target/gs-spring-boot-docker-0.1.0.jar -> /Users/nverma/Documents/test/springboot-maven/target/docker/gs-spring-boot-docker-0.1.0.jar
[INFO] Copying src/main/docker/Dockerfile -> /Users/nverma/Documents/test/springboot-maven/target/docker/Dockerfile
[INFO] Building image master:5000/gs-spring-boot-docker:1.6
Step 1 : FROM frolvlad/alpine-oraclejdk8:slim
 ---> ea24082fc934
Step 2 : VOLUME /tmp
 ---> Using cache
 ---> 0b7db19ef34d
Step 3 : ADD gs-spring-boot-docker-0.1.0.jar app.jar
 ---> 854f755a5bb0
Removing intermediate container 1909c71ce1fb
Step 4 : RUN sh -c 'touch /app.jar'
 ---> Running in 569c609bb1cb
 ---> 8c1e4c9979ca
Removing intermediate container 569c609bb1cb
Step 5 : ENTRYPOINT java -Djava.security.egd=file:/dev/./urandom -jar /app.jar
 ---> Running in 4d265484903f
 ---> 4bc97947a91f
Removing intermediate container 4d265484903f
Successfully built 4bc97947a91f
[INFO] Built master:5000/gs-spring-boot-docker:1.6
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 21.539 s
[INFO] Finished at: 2016-08-24T23:10:23+10:00
[INFO] Final Memory: 42M/196M
[INFO] ------------------------------------------------------------------------
nverma@macbook-pro$

 

We can see that first Maven compiled and built the Sprint Boot App and bundled it into a JAR as “target/gs-spring-boot-docker-0.1.0.jar”. One can test the App locally by running it as “java -jar target/gs-spring-boot-docker-0.1.0.jar” and by hitting the “http://localhost:8080/greet?name=World” url in your browser.

We also noticed that with the help of “docker-maven-plugin” Maven further transformed the Sprint Boot app into a full-fledged Docker image as per the instructions given in the standard Dockerfile under the src/main/docker directory. We should find that image built and present locally on our “master” server.


nverma@macbook-pro$ docker images
REPOSITORY                             TAG                 IMAGE ID            CREATED             SIZE
master:5000/gs-spring-boot-docker      1.6                 e8c983e1ee9e        7 seconds ago       195.3 MB
frolvlad/alpine-oraclejdk8             slim                ea24082fc934        2 days ago         167.1 MB
alpine                                 latest              4e38e38c8ce0        1 week ago         4.799 MB


A question might be churning in the mind as to why the docker image is called "master:5000/gs-spring-boot-docker:1.6"? Yes, it is like this for a reason! In this format, the image name reflects the name of the local Docker repository server reachable via “master:5000” fqdn.

But currently we have only named/tagged the image as it lives on the “master:5000” server. But we still need to build a local local Docker repository server and push/server the newly created image from there.

  • Build a local Docker repository server and push images into it

We will be building an insecure local registry for ease and learning purposes.

Login into the “master” node where we will run this service.

nverma@macbook-pro$ docker-machine ssh master
docker@master:~$


Edit the content of the file “/var/lib/boot2docker/profile” as below:

EXTRA_ARGS='                                           
--label provider=virtualbox
--insecure-registry 0.0.0.0:5000


Restart “docker service”:

docker@master:~$ sudo /etc/init.d/docker restart
Need TLS certs for master,127.0.0.1,10.0.2.15,192.168.99.101
-------------------
docker@master:~$



Verify:

docker@master:~$ docker info | grep -A2 Regis
Registry: https://index.docker.io/v1/
Labels:
 provider=virtualbox
Insecure Registries:
 0.0.0.0:5000
 127.0.0.0/8
docker@master:~$



Once done, just start the Docker repository service as a docker container itself! And that’s the beauty and power of Docker!

docker@master:~$ docker run -d -p 5000:5000 --restart=always --name registry registry:2

docker@master:~$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
2fc0354d46a9        registry:2          "/entrypoint.sh /etc/"   2 minutes ago         Up 2 minutes        0.0.0.0:5000->5000/tcp   registry
docker@master:~$


That’s it! Lets just push our image into this repository so that all the Swarm Cluster Nodes can fetch the required image from this local server instead of downloading from internet every time.

docker@master:~$ docker push master:5000/gs-spring-boot-docker:1.6
The push refers to a repository [master:5000/gs-spring-boot-docker]
decea895b03e: Pushed
64a80a295fba: Pushed
f3d2c9a5ee9f: Pushed
017f469bdc32: Layer already exists
4fe15f8d0ae6: Layer already exists
1.6: digest: sha256:39129303cdfee6d6b674db97e5ab7a6df77310a2494fc16c2c659ee4e154c7a3 size: 1375
docker@master:~$



  • Update all Swarm Nodes to be able to be able to pull images from the local Docker repository server (i.e. Master in this case)
Login into the “worker1” node where we will run this service. 

nverma@macbook-pro$ docker-machine ssh worker1
docker@worker1:~$ sudo vi /etc/hosts

Update "/etc/hosts" with the IP address of the repo. server i.e. "master" in our case.

192.168.99.101   master

Edit the content of the file “/var/lib/boot2docker/profile” as below:

EXTRA_ARGS='                                           
--label provider=virtualbox
--insecure-registry master:5000


Restart “docker service”:

docker@worker1:~$ sudo /etc/init.d/docker restart
Need TLS certs for master,127.0.0.1,10.0.2.16,192.168.99.102
-------------------
docker@worker1:~$



Repeat for all worker nodes. 

Ok. Now we have the Docker Swarm Mode cluster built and have the required Application as well as it’s Docker Image available locally. So lets continue our CICD journey by using Go Pipeline, while exploring the potential of Docker Swarm Mode, in next PART-3.

CICD with Docker Swarm Mode using your PC or Cloud - Part-3

PART-3 (CICD with Go Pipelines and Docker Swarm Mode Cluster)

 

Installation:

In the Part-1 and Part-2 we setup the Docker Swarm Mode cluster, built a Docker Image for an app and pushed that image into a local docker repository. In this part we will install and setup a popular CICD tool called Go Pipeline, and use it to perform 1-click deployments and rolling updates.

Step-1: Download and Install Go Pipeline Server and Agent (for CICD)

It'd require Java. So make sure to install Java too. Below is what I have on my Mac.

nverma@macbook-pro$ java -version
java version "1.8.0_102"
Java(TM) SE Runtime Environment (build 1.8.0_102-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)



I installed both the Go Pipeline Server and Agent locally on my Mac/PC. Once the Go Server started, it became accessible via http://localhost:8153/go/pipelines


Tip: For the Go Agent to run the commands as needed, we need to set the PATH variable like below:

nverma@macbook-pro$ pwd
/Users/nverma/Library/Application Support/Go Agent

nverma@macbook-pro$ cat overrides.env
PATH=$PATH:/usr/local/bin:/Users/nverma/Documents/apache/apache-maven-3.3.9/bin
nverma@macbook-pro$


Step-2: Setup the Pipeline


To make it easier to test and setup, let us create a single Pipeline called “mypipe”, which has single Stage called “mystage” with three Jobs called “1_build_job”, “2_push_job” and “3_deploy_job”.

As per the names:

- 1_build_job: is to fetch the code from Github, run the Maven build and create the Docker Image.

- 2_push_job: is to run the Docker push command to push the image into the local image repository (i.e. master:5000)

- 3_deploy_job: is to create a Docker service if first time, else update the image with the latest image and let Docket swarm do a rolling update to the live application.

To connect to the Swarm “master” node, set the following variables at the pipeline (“mypipe”) level under the “Environment Variables” tab.

'DOCKER_HOST' to 'tcp://192.168.99.101:2376'
'DOCKER_MACHINE_NAME' to 'master'
'DOCKER_TLS_VERIFY' to '1'
'DOCKER_CERT_PATH' to '/Users/nverma/.docker/machine/machines/master'


Example:



Under the “Materials” tab, add material of type “Git” with the URL of the application source code, in this case use my sample app’s GIT repo i.e. “https://github.com/n1t1nv3rma/springboot-maven.git”.


Example:


Update the Jobs:

a) Update “1_build_job” with a new ‘Task’ of type “Custom Command”. Delete the default Task of type “Ant” and add a Custom Type by selecting option “more”
    - Under the Command type “mvn”
    - Under the ‘Arguments’ type “package” and “docker:build” in new lines (in Go even argument is written in new line).

Example:



b) Update “2_push_job” with a new ‘Task’ of type “Custom Command”. Delete the default Task of type “Ant” and add a new one of type “more”
    - Under the Command type path to a custom script “/Users/nverma/Documents/Maven/scr/push-image.sh”

The contents of this script are:

------
#!/bin/bash
eval $(docker-machine env master)
IMG=`docker images | grep master:5000 |  sort -k2,1 -nr | head -1 | awk '{print $1":"$2}'`
docker push $IMG
------



c) Update “3_deploy_job” with a new ‘Task’ of type “Custom Command”. Delete the default Task of type “Ant” and add a new one of type “more”
    - Under the Command type path to a custom script “/Users/nverma/Documents/Maven/scr/run-service.sh”

The contents of this script are:

------
#!/bin/bash
eval $(docker-machine env master)

IMG=`docker images | grep master |  sort -k2,1 -nr | head -1 | awk '{print $1":"$2}'`

SERVICE="myspringapp"

if [ `docker service ls -f NAME=$SERVICE -q | wc -l` -gt 0 ]
then
    echo "updating service $SERVICE..."
        docker service update --image $IMG $SERVICE
else
    echo "ceating service $SERVICE..."
    docker service create --replicas 1 --name $SERVICE --update-delay 20s -p 8080:8080 $IMG
fi
------



We are all set for the basic automation. For more information about the Go Pipeline refer: “https://docs.go.cd/current/introduction/concepts_in_go.html”.


Step-3: Run the Pipeline



Click on the "Play" button or make changes to code. If all goes well, all these 3 Jobs should fetch the code, build the image and run the required service using it. In case of any issues, click the Pipeline (status bar) or each Job (status bar) to view the Console log for further troubleshooting.

To initiate the pipeline, I’d just update the application code and commit/push the code into the Github, and the pipeline should start on it’s own.

For example, as a simple change, let’s update the version to “<app.version>1.7</app.version>” in the pom.xml, say under the “/Users/nverma/Documents/test” directory (where I cloned the application files in Part-2) and push the changes to the Github repo.

One should have the new Application running with newer docker images 1.7.

Detailed example:

nverma@macbook-pro$ grep app.version pom.xml
        <app.version>1.7</app.version>
                    <imageName>${docker.image.prefix}/${project.artifactId}:${app.version}</imageName>
nverma@macbook-pro$

nverma@macbook-pro$ git add .

nverma@macbook-pro$ git commit -m "initial"
[master 3aa0ddf] initial
 1 file changed, 1 insertion(+), 1 deletion(-)
nverma@macbook-pro$

nverma@macbook-pro$ git status
On branch master
Your branch is ahead of 'origin/master' by 1 commit.
  (use "git push" to publish your local commits)
nothing to commit, working directory clean
nverma@macbook-pro$

nverma@macbook-pro$ git push -u origin master
Counting objects: 3, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 341 bytes | 0 bytes/s, done.
Total 3 (delta 1), reused 0 (delta 0)
remote: Resolving deltas: 100% (1/1), completed with 1 local objects.
To https://github.com/n1t1nv3rma/springboot-maven.git
   ed8d069..3aa0ddf  master -> master
Branch master set up to track remote branch master from origin.
nverma@macbook-pro$



This should initiate the Pipeline “mypipe” and all 3 Jobs should run one by one.


Following is what I see now:

nverma@macbook-pro$ docker images | grep 1.7
master:5000/gs-spring-boot-docker      1.7                 79d919940ce3        25 seconds ago      195.3 MB


nverma@macbook-pro$ docker node ls
ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
2d9ngt8phrhkdm21cwjv1tjss    worker2   Ready   Active       
6rarliq49ehaoi5olk85ztpfe    worker1   Down    Drain        
bo83ch825l2t1mbo8n6z4xkxj *  master    Ready   Active        Leader

nverma@macbook-pro$ docker service ls
ID            NAME         REPLICAS  IMAGE                                  COMMAND
3cu2kq6z0zx9  myspringapp  1/1       master:5000/gs-spring-boot-docker:1.7 
nverma@macbook-pro$

nverma@macbook-pro$ docker service ps myspringapp
ID                         NAME           IMAGE                                  NODE     DESIRED STATE  CURRENT STATE               ERROR
0inz6b7b1z76jw62091ajwgcl  myspringapp.1  master:5000/gs-spring-boot-docker:1.7  worker2  Running        Running about a minute ago 
nverma@macbook-pro$


nverma@macbook-pro$ docker service scale myspringapp=2
myspringapp scaled to 2

nverma@macbook-pro$ docker service ps myspringapp
ID                         NAME           IMAGE                                  NODE     DESIRED STATE  CURRENT STATE          ERROR
0inz6b7b1z76jw62091ajwgcl  myspringapp.1  master:5000/gs-spring-boot-docker:1.7  worker2  Running        Running 2 minutes ago 
5ndjck2hnxmhabd7c00bkp1pk  myspringapp.2  master:5000/gs-spring-boot-docker:1.7  master   Running        Running 8 seconds ago 
nverma@macbook-pro$



Further evaluation:


Perform following steps to further enhance your docker experience!

*) Scale up/down, and try hitting the service on a Swarm node which is not running service currently. For example in the above output, service is not running on the “worker1”, so hitting the service on “worker1:8080/greet?name=Testing” should still work! Magic? Yeah… Docker Swarm Mode has implemented full internal network mesh, routing and load-balancing! So cluster nodes are smart enough to route the request to the node running it!

*) Run “docker service create --help” and check out the “—update-delay duration” and “—update-parallelism uint” options and perform Rolling Updates to your App.

*) Create your own Overlay Network in your Swarm and use it to laugh another Application/Service! And see if you can reach Service from one network to a Service from another network. It shouldn’t and that the beauty of complete isolation.

More info: “$docker network --help” and https://docs.docker.com/engine/swarm/networking/


That’s it for now... Hope you enjoyed the 3 blogs and Docker Swarm Mode! Have fun!

CICD with Docker Swarm Mode using your PC or Cloud - Part-1

Introduction:


As of




  • Distributed: Swarm Mode uses the Raft Consensus Algorithm in order to coordinate and does not rely on a single point of failure to perform decisions.
  • Secure: Node communication and membership within a Swarm are secure out of the box. Swarm Mode uses mutual TLS for node authentication, role authorization and transport encryption, automating both certificate issuance and rotation.
  • Simple: Swarm Mode is operationally simple and minimizes infrastructure dependencies. It does not need an external database to operate. It uses internal distributed State store.





SoftwareMy SetupOther OptionsSource
Docker 1.12+MacBook OSX*, VirtualBox Windows* PC, Cloud instances (AWS, Azure, Google etc)Mac: https://docs.docker.com/docker-for-mac/
Windows: https://docs.docker.com/docker-for-windows/
Linux: https://www.docker.com/products/docker#/servers
Go Pipeline ServerMacBook OSX Windows PC, Cloud instances (AWS, Azure, Google etc)https://www.go.cd/download/
Oracle VirtualBoxMacBook OSX Windows PC, Cloud instances (AWS, Azure, Google etc)https://www.virtualbox.org/wiki/Downloads

* As of Aug 2016: Docker version 1.12+ is in Beta for OSX and Windows.



PART-1 (Installation, Setup and Testing of Docker Swarm Mode Cluster)












5.0.20 r106931 on my Mac, but it doesn't matter, try and use latest version!

Setup:

Step-1: Create required docker machines


We will create 3 machines (can work with 2 too) where we will call one as master and others as workers


nverma@macbook-pro$ docker-machine create -d virtualbox master
Running pre-create checks...
Creating machine...
(master) Copying /Users/nverma/.docker/machine/cache/boot2docker.iso to /Users/nverma/.docker/machine/machines/master/boot2docker.iso...
(master) Creating VirtualBox VM...
(master) Creating SSH key...
(master) Starting the VM...
(master) Check network to re-create if needed...
(master) Waiting for an IP...
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with boot2docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Checking connection to Docker...
Docker is up and running!
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env master



  • Similarly create other required VMs.

 nverma@macbook-pro$ docker-machine create -d virtualbox worker1
...
 nverma@macbook-pro$ docker-machine create -d virtualbox worker2
...


  • Check VMs

$ docker-machine ls
NAME      ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER    ERRORS
master    -        virtualbox   Running   tcp://192.168.99.101:2376           v1.12.0  
worker1   -        virtualbox   Running   tcp://192.168.99.102:2376           v1.12.0  
worker2   -        virtualbox   Running   tcp://192.168.99.103:2376           v1.12.0  



Step-2: Setup Swarm Mode - Cluster


Docker has made it so simple to setup the Swarm Cluster with their Swarm Mode that it is as simple as 1-2-3... 

Initialize the machine named "master" as swarm master.

Set environment:

nverma@macbook-pro$ eval $(docker-machine env master)
 
Initialise the Swarm with the external IP of the master VM:


nverma@macbook-pro$ docker swarm init --advertise-addr 192.168.99.101

Swarm initialized: current node (bo83ch825l2t1mbo8n6z4xkxj) is now a manager.

To add a worker to this swarm, run the following command:
    docker swarm join \
    --token SWMTKN-1-4zqhbw2mylo2b9ajaix9n06nmjxjx00wlvzv8kyv22olyx4ls1-0h27p8ndvzlyyzdcyrmlm5iz6 \
    192.168.99.101:2377

To add a manager to this swarm, run the following command:
    docker swarm join \
    --token SWMTKN-1-4zqhbw2mylo2b9ajaix9n06nmjxjx00wlvzv8kyv22olyx4ls1-99v82agtft162d4i6qga0omlq \
    192.168.99.101:2377 



Swarm cluster is ready! Now join the workers/nodes into this cluster 


nverma@macbook-pro$ eval $(docker-machine env worker1)
nverma@macbook-pro$ docker swarm join \
  --token SWMTKN-1-4zqhbw2mylo2b9ajaix9n06nmjxjx00wlvzv8kyv22olyx4ls1-0h27p8ndvzlyyzdcyrmlm5iz6 \
  192.168.99.101:2377
 

This node joined a swarm as a worker.

Repeat same on the 3rd mode/worker2. And that's it, my complete 3 nodes Swarm Cluster is ready to rock!

This is what I have now:

nverma@macbook-pro$ eval $(docker-machine env master)nverma@macbook-pro$ docker node ls
ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
6rarliq49ehaoi5olk85ztpfe    worker1   Ready   Active       
7i4s01v67u3faa1nof8nzf2g0    worker2   Ready   Active       
bo83ch825l2t1mbo8n6z4xkxj *  master    Ready   Active        Leader



Step-3: Test run your first service

Following steps should help you get some flavor of the swarm mode.

  • Create a new service:

nverma@macbook-pro$ docker service ls
ID  NAME  REPLICAS  IMAGE  COMMAND

nverma@macbook-pro$ docker service create --replicas 1 --name mypingapp alpine ping google.com
8qjvnp8efw4fw1uhmfe388fbk


nverma@macbook-pro$ docker service ls

ID            NAME       REPLICAS  IMAGE   COMMAND
8qjvnp8efw4f  mypingapp  0/1       alpine  ping google.com



Here name of my service/application is "mypingapp", it will use a very small "alpine" image from the Docker Hub Repository and will run "ping google.com" command during start.



  • Get more details:

nverma@macbook-pro$ docker service ps mypingapp
ID                         NAME         IMAGE   NODE    DESIRED STATE  CURRENT STATE          ERROR
ed1m1atgwm250brc9lvxlgi6h  mypingapp.1  alpine  master  Running        Running 3 seconds ago
nverma@macbook-pro$



It shows that my service is running 1 container as desired and running on the "master" node.

 

  • Check logs:

nverma@macbook-pro$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
652c368e4316        alpine:latest       "ping google.com"        8 seconds ago       Up 5 seconds                                 mypingapp.1.ed1m1atgwm250brc9lvxlgi6h

nverma@macbook-pro$ docker logs 652c368e4316
PING google.com (172.217.25.46): 56 data bytes
64 bytes from 172.217.25.46: seq=0 ttl=61 time=11.134 ms
64 bytes from 172.217.25.46: seq=1 ttl=61 time=10.046 ms
.....

  • Inspect your service:

nverma@macbook-pro$ docker service inspect --pretty mypingapp
  • Scale your service (to handle more load or parallel processing) :

nverma@macbook-pro$ docker service scale mypingapp=3
mypingapp scaled to 3

 nverma@macbook-pro$ docker service ls
ID            NAME       REPLICAS  IMAGE   COMMAND
8qjvnp8efw4f  mypingapp  3/3       alpine  ping google.com

nverma@macbook-pro$ docker service ps mypingapp
ID                         NAME         IMAGE   NODE     DESIRED STATE  CURRENT STATE           ERROR
ed1m1atgwm250brc9lvxlgi6h  mypingapp.1  alpine  master   Running        Running 11 minutes ago
4k9ek6cejk36a4gxgic3cqgmr  mypingapp.2  alpine  master   Running        Running 10 seconds ago
bzv3s3td44tz03uqd6imcxm1u  mypingapp.3  alpine  worker2  Running        Running 8 seconds ago  



Once you’re happy, remove your service and it's the time to create some more meaningful service and move to PART-2.

nverma@macbook-pro$ docker service rm mypingapp




Wednesday 15 June 2016

Is DevOps and Agile same?


Well, it is a valid question that would churn in anyone's mind!

Ideally, both Agile methodology and DevOps methodology promotes the software development practices, where the ultimate aim is to realize the business value much faster through iterative, smaller and continuous software deliveries.

Looking at the Agile Principles, we can say that DevOps is nothing but a subset of the Agile software development methodology.

While Agile is an over arching IT software development methodology, DevOps have more emphasis on the collaboration between IT Development and Operational organisations with automation as key principle, in order to improve overall efficiency and productivity.

Although, one of the Agile principles says "Business people and developers must work
together daily throughout the project", it sort of neglects the operational side of IT, which as a matter of fact much more relevant for bigger and mature IT organisations. As a result IT Ops practices were often "not" in align with those of Development team and hence the business suffered.

So IMO, an evolution of the DevOps methodology took place in around year 2008, for Development and Operational organisations to collaborate with each other and together realise the business value faster!

DevOps tools - One stop!

The purpose of this post is to create a one-stop shop for all the popular tools and frameworks promoting DevOps

Development - Collaboration
Architecture & Development standards:
Togaf http://www.opengroup.org/subjectareas/enterprise/togaf/
Zacman http://www.zachman.com/
Dodaf http://dodcio.defense.gov/Library/DoDArchitectureFramework.aspx
Agilie methodology:
Atlassian JIRA https://www.atlassian.com/software/jira
Atlassian Confluence https://www.atlassian.com/software/confluence
Scrum https://en.wikipedia.org/wiki/Scrum_(software_development)
Kanban https://en.wikipedia.org/wiki/Kanban_(development)
Other https://en.wikipedia.org/wiki/Agile_software_development
Testing automation:
JMeter http://jmeter.apache.org
Selenium http://seleniumhq.org/
HPQC http://www.hp.com/go/qualitycenter
Cucumber https://cucumber.io/
Other https://en.wikipedia.org/wiki/List_of_web_testing_tools
Deployment Automation:
IDE (Netbeans, Eclipse, Visual Studio) https://netbeans.org/
https://eclipse.org/ide/
https://www.visualstudio.com/
Vagrant, Docker, Packer etc. https://www.vagrantup.com/
https://www.docker.com/
https://www.packer.io/
Virtual Box, VMware etc. https://www.virtualbox.org/
http://www.vmware.com
Cloud (AWS, Azzure, Google etc) https://aws.amazon.com/application-management/
https://aws.amazon.com/codedeploy
https://aws.amazon.com/codepipeline
https://www.visualstudio.com/
https://azure.microsoft.com/en-us/documentation/scenarios/devtest/
https://cloud.google.com/deployment-manager
Other Scripting?
Project Management & Service Transition:
Project Management (Prince2, PMP/PMBOK) https://www.prince2.com
http://www.pmi.org/PMBOK-Guide-and-Standards.aspx
Release/Change Management (ITIL) https://en.wikipedia.org/wiki/ITIL
CAB (ITIL) https://en.wikipedia.org/wiki/Change_advisory_board
Knowledge Transfer (ITIL) https://en.wikipedia.org/wiki/Knowledge_transfer
Other https://en.wikipedia.org/wiki/IT_service_management
Development - Automation
Coding:
IDE (Netbeans, Eclipse, Visual Studio) https://netbeans.org
/
https://eclipse.org/ide/
https://www.visualstudio.com/
Source Control (Git, Subversion) https://git-scm.com
/
https://en.wikipedia.org/wiki/Apache_Subversion
TDD, BDD (Test / Behaviour Driven Development) https://en.wikipedia.org/wiki/Test-driven_development
https://en.wikipedia.org/wiki/Behavior-driven_development
Other https://en.wikipedia.org/wiki/Build_automation
Build automation:
Gradle, Maven, Ant/Ivy https://gradle.org
https://maven.apache.org
http://ant.apache.org
Jenkins, Go Pipelines, Bamboo, TeamCity, Cruise Control https://jenkins-ci.org
https://www.go.cd/
/https://www.atlassian.com/software/bamboo
http://www.jetbrains.com/teamcity/
http://cruisecontrol.sourceforge.net/
Other List_of_build_automation_software#Continuous_integration_tools
Package Management:
Artifactory, Nexus, Apache Archiva http://www.jfrog.com/open-source
http://www.sonatype.org/nexus/go/
http://archiva.apache.org/
Maven, NPM, NuGet https://maven.apache.org
http://npmjs.org/
https://nuget.org/
Other List_of_software_package_management_systems#Application-level_package_managers
Unit/Integeration Testing:
Junit http://junit.org
RSpec http://rspec.info
Nunit http://www.nunit.org
TestNG http://testng.org
Spock https://code.google.com/p/spock
Other List_of_unit_testing_frameworks
Code coverage:
SonarQube http://www.sonarqube.org
Cobertura http://cobertura.github.io/cobertura/
Clover https://www.atlassian.com/pt/software/clover
NCover https://www.ncover.com
Simplecov https://github.com/colszowka/simplecov
Other https://en.wikipedia.org/wiki/Code_coverage
Operations - Collaboration
Agile practices:
Atlassian JIRA (Issues, Feedback) https://www.atlassian.com/software/jira
Atlassian Confluence (Documentation) https://www.atlassian.com/software/confluence
Scrum (Daily standups) https://en.wikipedia.org/wiki/Stand-up_meeting
Kanban (Agility) https://en.wikipedia.org/wiki/Kanban_board
Infrastructure as a Service:
Private Cloud (DC, VMware) Traditional DC: Microsoft HyperV, VMware vCloud Suite, Oracle, Cisco, Redhat etc.
Public Cloud (AWS, Azure, RackSpace etc.) http://aws.amazon.com
/
https://azure.microsoft.com
http://www.rackspace.com
Platform as a Service:
Private Cloud (DC, VMware) Traditional DC: Microsoft HyperV, VMware vCloud Suite, Oracle, Cisco, Redhat etc.
Public (AWS, SalesForce, Azure, Google, IBM, OpenShift etc. ) http://aws.amazon.com/
http://www.salesforce.com/au/platform/overview/
https://azure.microsoft.com
https://cloud.google.com
https://www.openshift.com
Service Transition:
Project Management (Prince2, PMP/PMBOK) https://www.prince2.com
http://www.pmi.org/PMBOK-Guide-and-Standards.aspx
Change Management, Incident Management (ITIL / ServiceNow) https://en.wikipedia.org/wiki/ITIL
https://www.servicenow.com
CAB (ITIL) https://en.wikipedia.org/wiki/Change_advisory_board
Knowledge Transfer (ITIL) https://en.wikipedia.org/wiki/Knowledge_transfer
Other https://en.wikipedia.org/wiki/IT_service_management
Continuous Service Improvement:
Issues/Feedback Manangement (JIRA, ServiceNow, BMC Remedy) https://www.atlassian.com/software/jira
https://www.servicenow.com
http://www.bmc.com/it-solutions/remedy-itsm.html
Risks and Reporting (ITSM) https://en.wikipedia.org/wiki/IT_service_management
http://www.bmc.com/it-solutions/remedy-itsm.html
Service Continuity (ITSM) https://en.wikipedia.org/wiki/IT_service_management
http://www.bmc.com/it-solutions/remedy-itsm.html
Other https://en.wikipedia.org/wiki/Continual_improvement_process
Operations - Automation
Configuration Management:
Chef http://www.chef.io/
Puppet http://www.puppetlabs.com/
Ansible http://www.ansible.com/
Salt http://saltstack.com/
CFEngine http://www.cfengine.com/
Other Scripting?
Deployment Automation:
Chef, Puppet, Ansible, Salt http://www.chef.io/
http://www.puppetlabs.com/
http://www.ansible.com/
http://saltstack.com/
Cloud Services (AWS, Azure, Heroku, Google, Custom) http://aws.amazon.com
http://www.salesforce.com/au/platform/overview/
https://azure.microsoft.com
https://cloud.google.com
Other Scripting?
Monitoring:
Nagios, CA UIM, BMC, Solarwinds https://www.nagios.org
http://www.ca.com/us/opscenter/ca-unified-infrastructure-management.aspx
http://www.bmc.com/it-solutions/truesight-operations-management.html
http://www.solarwinds.com
Zabbix, NewRelic, ScienceLogic, AppDynamics http://www.zabbix.com
/
http://newrelic.com/
https://www.sciencelogic.com/
http://www.appdynamics.com/
Cloud Services (AWS, Azure, Heroku, Google, Custom) http://aws.amazon.com
http://www.salesforce.com/au/platform/overview/
https://azure.microsoft.com
https://cloud.google.com
Other https://en.wikipedia.org/wiki/Application_performance_management
Security & Analysis:
Splunk http://www.splunk.com
Sumo Logic https://www.sumologic.com
Loggly https://www.loggly.com
ArcSight http://www.hp.com/go/ArcSight
Cloud Services (AWS, Azure, Heroku, Google, Custom) http://aws.amazon.com
http://www.salesforce.com/au/platform/overview/
https://azure.microsoft.com
https://cloud.google.com
Other https://en.wikipedia.org/wiki/Security_information_and_event_management


Let's keep it up-to-date for mankind!

Sunday 5 June 2016

AWS vs DevOps

DevOps is not limited to the IT organizations operating in the traditional Data Centres but it is very aggressively being adopted by organisations running on cloud. When we talk about cloud, it is hard to pass by AWS (Amazon Web Services).

Even world biggest cloud providers like AWS are aligning their suite of services and product line according to the industry accepted DevOps practices!

Following table provides a quick insight into how AWS services/products are aligned to the DevOps practices:

DevOps Pactise AWS Services Comments
Architecture & Development standards Architecture

Whitepapers

Solutions
A great mix of Architectures, Whitepapers and Solutions based upon various different Application and Use types.
Code Management Codecommit Works with AWS IAM by default. Though it's obvious integration with other DevOps related services like AWS CodePipeline, CodeDeploy etc. could be a game-changer!
Build automation Codepipeline Integrates nicely with GitHub, Jenkins, AWS CodeDeploy, Lambda, IAM etc. to provide a nice CI/CD experience.
Package Management Way to go... But AWS CodePipeline can be used with AWS S3 service to store archives. Refer to our DevOps Tools - One Stop page for more info.
Testing automation Way to go... But AWS CodePipeline can be used create Custom Actions for testing automation. Refer to our DevOps Tools - One Stop page for more info.
Code coverage AWS CodeGuru Refer to our DevOps Tools - One Stop page for more info.
Infrastructure as a Code Cloudformation

Servicecatalog
Build entire AWS infrastructure via code/template
Platform as a Service AWS EC2
AWS EC2 Container Service (ECS)
Use AWS AMI to pre-package and create AWS EC2 Compute Instances. Whereas ECS is a Docker based clustering solution. Can use Packer to generate AMIs or Docker images.
Deployment Automation
Elasticbeanstalk Provide a good range of pre-built platform (Java, Ruby, Python, Docker, NodeJS, GO etc) with choice of pre-configured Auto Scaling Group and Load Balancer etc. Most easy to use and hence less choice for customization. Just bring your code.
Opsworks OpsWorks is Chef based currently and can be used for Deployment Automation. Just bring your own recipes or select from built-in platform, without the need to maintain your own Chef server. Allows more customization and control as compared to Beanstalk.
Codedeploy Use your existing automated scripts to simply deploy the code. Nicely integrates with AWS S3, GitHub and AWS CodePipeline. You have full control on the underlying platform choice and deployment. Has good lifecycle and deployment management.
Cloudformation Allows complete control over the underlying infrastructure and code installation with the help of built-in Metadata. Hence most complex and powerful option.
Configuration Management Opsworks AWS OpsWorks is Chef based currently and can be used for Deployment Automation as well as Configuration Management
Monitoring Cloudwatch

Cloudtrail
CloudWatch is for basic monitoring, alerting and logs management, whereas CloudTrail is for auditing
Security and Reporting AWS IAM

AWS Config

Cloudtrail
IAM is core Ideantity and Access management that integerates with almost every AWS service.