in docker devops ~ read.
Splendors and Miseries of Docker Network

Splendors and Miseries of Docker Network

A bit less than two weeks ago Docker 1.9 was released. When I read release notes which contains all rainbow colors about new cool superdooper Docker Network with multi-host support I was excited. I wanted to try this. I believed that many problems with network will go away. I was wrong. No one problem was solved.

Life before

Before Docker 1.9 there are four possible network options or network configurations for containers:

  • none
  • container
  • bridge
  • host

When network is not needed "none" is used. Only one loopback interface is available. Nginx container may be launched as show below. Argument -p is ignored.

docker run -d --net=none -p=9000:80 \  
  --name nginx_none nginx

But possibility to use network is still available if other container use container network option:

docker run -ti --net=container:nginx_none \  
  --name alpine \
  alpine sh

After launch you may get nginx content inside of alpine container:

wget -O - localhost //not :9000  

The most popular network option was bridge. It gives possibility to incapsulate network through bridge interface and mapping ports inside of container to any port that you want on host interface include random port:

docker run -d --net=bridge -p=9001:80 \  
  --name nginx_bridge nginx

curl $(docker-machine ip vm):9001  

But main problem of this network configuration is fact that some kind of software may think that it has ip inside of container and try to use it. Internal ip is not equal to external and there are many problems of production usage cause this reason. Look at this fragment from docker-compose file:

consul:  
  image: gliderlabs/consul-server
  ports:
    - "8500:8500"
  command: "-bootstrap -advertise 192.168.99.100"

We should pass advertise ip address manually cause consul thinks that it must provide internal ip address. We can solve this problem if we use host network option:

docker run -d --net=host -p=9002:80 \  
  --name nginx_host nginx

But we can't control port now. And we can't launch several nginx containers on same host if we want.

So, any option for production has pros and cons. And it was a high hopes on new network docker feature.

Do right things wrong

First about a good parts. Docker provides simple and unified interface for all its products. And Docker Network is not the exception.

It has simple commands with network subcommand:

docker network ls     //see list of networks  
docker network create //create new user defined network  
docker network rm     //remove network

and so on (doesn't matter)  

And with single-host point-of-view things becomes better. Simple new bridge interface creation for more network security and isolation:

docker network create \  
   --driver bridge new_bridge

You also may connect or disconnect any network to any container. But not always. If container expose ports on 0.0.0.0 interface then you get an error while trying connect network to this container. Be careful.

Anyway, single-host networks is not something new. Syntax sugar and nothing more. The ice on the cake is multi-host network that docker calls overlay.

For multi-host network you must have key-value backend storage such as Consul or etcd. Then you should create new swarm cluster:

docker-machine create \  
  --driver virtualbox \
  --swarm --swarm-image="swarm" --swarm-master \
  --swarm-discovery="consul://$(docker-machine ip consul):8500" \
  --engine-opt="cluster-store=consul://$(docker-machine ip consul):8500" \
  --engine-opt="cluster-advertise=eth1:2376" \
node1

docker-machine create \  
  --driver virtualbox \
  --swarm --swarm-image="swarm" \
  --swarm-discovery="consul://$(docker-machine ip consul):8500" \
  --engine-opt="cluster-store=consul://$(docker-machine ip consul):8500" \
  --engine-opt="cluster-advertise=eth1:2376" \
node2

docker-machine create \  
  --driver virtualbox \
  --swarm --swarm-image="swarm" \
  --swarm-discovery="consul://$(docker-machine ip consul):8500" \
  --engine-opt="cluster-store=consul://$(docker-machine ip consul):8500" \
  --engine-opt="cluster-advertise=eth1:2376" \
node3

eval $(docker-machine env --swarm node1)  

Next you should create new overlay network:

docker network create \  
   --driver overlay dn

For test it use this simple docker-compose file:

web:  
    image: bfirsh/compose-mongodb-demo
    environment:
        - "MONGO_HOST=mongo" //magic host :)
    net: "dn" //overlay network
    ports:
        - "80:5000"
mongo:  
    image: mongo
    container_name: mongo
    net: "dn"

If launch this compose file nodeX/mongo and foldername_web_1 containers are created. Ok, lets explain what is the magic host in environment variable. Try this command:

cat /etc/hosts  

and see inside:

10.0.1.2 mongo  
10.0.1.2 mongo.dn  

Ok, DNS is cool and rocks. Lets see inside of mongo container:

10.0.1.3 foldername_web_1  
10.0.1.3 foldername_web_1.dn  

Not so cool, right? Looks awful and unusable in real life.
Maybe we can get information from consul that we use for swarm backend? Not. It has not any useful information or I don't know how to get it.

But we still have an overlay network and maybe real project can be launched quickly and simply.

Let's have this configuration:

consul:  
  image: gliderlabs/consul-server
  container_name: consul
  ports:
    - "8400:8400"
    - "8500:8500"
    - "53:8600/udp"
  net: "dn"
  command: "-bootstrap -ui-dir /ui"

fee-service:  
  image: fee-service
  ports:
    - "8080"
  net: "dn"
  environment:
    - "SERVICE_NAME=fee-service"
    - "SERVICE_CHECK_HTTP=/health"
    - "SPRING_PROFILES_ACTIVE=consul"

Consul for service discovery and one service that is written on Spring Cloud stack. Fee service have an option that describe consul host:

spring.cloud.consul.host: consul  

Not so good cause we know about _1 and _2 if we want more than one consul container in production in our cluster, in our network. But in example we shut eyes on this assumption. If we launch this configuration we'll have broken health checks and wrong endpoint of service, cause Spring gets first non-loopback interface. And it is not our overlay network. It's a bridge. And it's a not default bridge, it's a special docker_gwbridge that has different subnet. Very funny I know.

Ok, maybe plugins help us? Strange but on the official docker plugin page (http://docs.docker.com/engine/extend/plugins/) information about any (at least one please) network plugin is not presented. Why? I googling weave plugin (I just remember this guys from dockercon). It did not work with first try. With second try I could create network but DNS did not work (cause two different bridges, remember that?). Other overlay network problems was still alive. Last plugin update have the unfavourable explanation about DNS problems:

WeaveDNS not supported

Weave's built-in service discovery mechanism is not currently supported by the plugin. However, Docker provides a rudimentary discovery mechanism by writing all user-provided container names and hostnames into every container's /etc/hosts.

I want rant and rave.

And last chance is swarm REST API. Yep, we can get all information about containers with API. But there are no integration with popular client load balancers such as a ribbon now. No health checks except of rudimentary node-alive. And maybe some surprises that I don't know at the moment.

Why Docker? Why you do right things wrong?

Conclusion

One try with real code is more informative than thousand examples with busybox. I hope you understand.

comments powered by Disqus