We recently switched (like 2 weeks ago) our project from deployment on ubuntu servers via 'git pull' managed with supervisord to docker/coreos/fleet, and it's been epic. While coreos is built for large clusters, we run a 3 host cluster in ec2, and couldn't be happier. We switched from multiple servers running 1 instance of each service to load balancing all instances on these 3 hosts. This increased uptime, made deployment and management easier, and gave us the benefits of docker as well (verifying things work locally).
There's only 2 real problems, both of them very minor:
* fleet managing state. We've had to manually kill containers sometimes, and destroy systemd services before we could start it again.
* all EC2 amis use ebs backed instances. We haven't used a higher-IOPs ebs backed instance because the only delay we see are in startup times (which doesn't matter, just longer rolling deploys). But an instance-backed ami would be nice.
How do you handle persistent storage in a fault-tolerant way?
You could technically pin your MySQL container to one host, but that seems to defeat the point of fleet. I considered trying to mount an iSCSI target to run the database from, but CoreOS doesn't have a working iSCSI initiator.
I guess I could just run all the persistent stuff on a more traditional OS, but then why mess with CoreOS at all?
- one to attach the EBS volume (wrapping the above container)
- one to mount the device to a directory on the host
- one to run the DB container, with a bind mount to the host directory
If the CoreOS host terminates, the cluster will reschedule the units to another host.
CoreOS seems to do a decent job as the application layer for something operating in a larger cloud that provides database and file storage as services, like Amazon or OpenStack. However, I think Docker has the potential to eliminate the need for infrastructure-level "clouds" to be so complex. The basic unit can be the container instead of the instance, so you don't need fancy automated instance management and all that. It'd be nice if you could also obviate the need for the other external services, and just run everything in containers on metal.
I was really excited by CoreOS as a way to do this, but it seems to fighting me at every turn. It seems much easer to just wire together the REST APIs of several Ubuntu/CentOS nodes running Docker.
That explains how they chose to minimize deadlocks.
Yes, if you use it incorrectly and/or for the wrong problem space, you will have issues. It isn't a magical solution to all problems. It is, however, an effective solution for a specific problem space.
I've been using it in our development environment. It locks like crazy.
This is after configuring HAProxy to have a primary node which all connections go to.
OpenStack seems to be OK with it though, it chugs along without fully breaking down. I'm sure I would see a pretty big performance delta if I ran a benchmark against it with and without Galera.
Using CoreOS as a "base image" under docker would serve little purpose because CoreOS isn't really designed to run anything other than its core services. People usually use Ubuntu or CentOS.
He's saying that there isn't a point of running CoreOS as the base image inside of a container, which is true for most people. There isn't a good reason to run CoreOS containers on a CoreOS host.
Oh! I had a different sense of "base ... under Docker" in mind.
My interpretation was that Docker was running on top of an OS, which you could call the "base", and say it was "under" Docker in the sense of "underlying".
Whereas he was using the opposite spatial metaphor: the Docker image, which is a "base" in the Docker sense because you derive other images from it, is "under" Docker in that it's within Docker, or lower in the process tree.
George Lakoff would be thrilled by this example of conflicting metaphors.
Have you written up this in a blog post anywhere? Would be an interesting read and very valuable for dev/test scenarios in addition to production ones.
There's only 2 real problems, both of them very minor:
* fleet managing state. We've had to manually kill containers sometimes, and destroy systemd services before we could start it again.
* all EC2 amis use ebs backed instances. We haven't used a higher-IOPs ebs backed instance because the only delay we see are in startup times (which doesn't matter, just longer rolling deploys). But an instance-backed ami would be nice.