Docker and MOGs

	Author:	“No Bugs” Hare  Follow:
	Job Title:	Sarcastic Architect
	Hobbies:	Thinking Aloud, Arguing with Managers, Annoying HRs, Calling a Spade a Spade, Keeping Tongue in Cheek

[rabbit_ddmog vol=”7″ chap=”Chapter 25(b) from “beta” Volume VII”]

When speaking about DevOps in 2017, we cannot avoid discussing Docker (it is a darling of way too many DevOps practitioners out there). These days, everyone and their dog goes crazy about Docker, but what exactly Docker is?

Strictly speaking, Docker is an application container, but then we run into a problem defining “application container”. At this point, while [Coleman] argues that “Containers are not VMs” – the first approximation to describe Docker is still as a “lightweight VM”.¹ Sure, there are significant differences between VMs and containers, but still – from a high-enough-viewpoint we can think of Docker containers as of “lightweight VMs”; continuing analogy from [Coleman], while indeed VMs are analogous to houses, and containers are analogous to apartments, when I (as an application) sleep in a room, I don’t care much whether the room is within an apartment or within a house.

Encapsulated Environment – A Real Saver (for Really Poorly Written Programs, That Is)

Theorizing aside, let’s see what Docker can do for us (as MOG DevOps).

Probably the most touted advantage of Docker is an ability to run our program in exactly the same environment with exactly the same components-our-program-depends-on – which in turn can save us lots of trouble on potential incompatibilities of different components.

For quite a few programs out there this is indeed The Big Thing™. I’ve seen programs (by Very Big Vendors™ BTW) which required several dozens of dependencies, with quite of the dependencies required as “exactly version X” (opposed to usual “at least version X”). Anybody who tried to deploy this kind of things, is quite sure to remember the pain.

In another example (and repeating myself from Vol. [[TODO]]): once upon a time, I have worked with a very nice developer (his name was Andrew), who was responsible for developing an installer for some commercial software (which was a total dependency mess). While usually he was very polite, but when speaking about his job, he described the process in terms of “Our software is a pile of **it, and I need to take this **it into my hands, and carry it to the end-user’s server, without spilling it on the way”.

“When facing such monstrosities as those pieces of... software discussed above, I perfectly understand admins who admire DockerWhen facing such monstrosities as those pieces of… software discussed above, I perfectly understand admins who admire Docker; and Andrew would obviously be delighted to prepare a Docker container and say “hey, please install this container instead, and it will just work because we have carried all our dependencies with us”.²

On the other hand, as a developer, I would be extremely ashamed of releasing such a program myself. As it was noted above – I am sure that we (as developers) are responsible not only for making a program which passes tests, but also for making sure that our program is usable in the real world. This includes avoiding producing abominations such as the ones described above; moreover – as it was discussed at length in Vol. I’s chapter on DIY-vs-Reuse –

In particular, from my experience, 99% of dependencies which require an exact version of some software (directly or indirectly), 95% of dependencies which demand a dozen of other dependencies, and 80% of all the dependencies I’ve seen in the software – are better to be avoided (the reasons to avoid specific dependencies vary, see discussion in Vol. II).

LOC Source lines of code (SLOC), also known as lines of code (LOC), is a software metric used to measure the size of a computer program by counting the number of lines in the text of the program's source code.— Wikipedia —While I certainly don’t want to say that all dependencies are bad (in particular, writing your own database, your own TCP stack, or your own JPEG library are rarely good ideas), IMNSHO, these days re-use is badly overused <sad-face />. As one example: if you’re about to use a component which merely performs a very basic e-mail address format validation, and requires a very specific version-of-.NET to do it – chances are that you’ll be much better in the long run writing these 50 LOCs yourself.

Sure, there will be quite a bit of resistance to the approach of dependency fighting (coming from the camp of the developers thinking along the lines of “I’ve got this task; what is the library I can use which does exactly what I need?”), but if we want our sizeable project to live a long and successful life – this is the only way.

And coming back to our current discussion on Docker:

If you’re still in doubt of “hey, what will happen if the next version of the software breaks our app?” – I have to say that in theory, it can happen with or without Docker (with Docker or without, you DO need to apply security patches to the dependencies). However, in practice, as I wrote and maintained quite a few real-world systems designed along the lines discussed in this book (in particular, fighting those sneaky dependencies vigilantly) – I have to say that I don’t remember of one single case of a program failing in production due to dependency version mismatch; those very few cases when such a mismatch would be a problem – were detected during routine testing.

A word of caution: as your Server-Side grows to, say, a million lines of code, it is almost inevitable that some part of it will become a “dependency hell” and will require Docker to run. Still – make sure that such occurrences are very occasional, are obviously frowned upon, and most importantly – do not allow them to become a common practice (allowing it will become the beginning of the end for your system).

Other Benefits of Docker

As discussed above, I strongly prefer to write programs without going into “dependency hell” – and for such programs, benefits of encapsulating components do not apply.

“Contrary to popular belief, there are benefits of Docker containers which go beyond providing poorly written programs with exact versions of their dependenciesStill, contrary to popular belief, there are benefits of Docker containers which go beyond providing poorly written programs with exact versions of their dependencies. These benefits³ include:

• Isolated Configuration.
  • While a well-written software doesn’t really need Docker to deal with “dependency hell” – Docker may help to ensure consistent configuration between QA and production (and production misconfigurations are known to cause quite a bit of trouble). As with anything else – it is not a silver bullet, but it may prevent a problem or two.
• Resource Quotas.
  • They can easily save your bacon from one runaway process eating up the whole RAM of your server (which in turn can bring down the whole game if you’re unlucky enough).
• Docker containers do provide a bit of additional security (protecting your host OS from an attacker who managed to take over your app and/or container).
  • Details are beyond the scope of this book; a good place to start researching this vast topic is [Docker.Security].
• Binary Version Control. To be perfectly clear: I am not speaking about using Docker instead of your usual source control system. Rather, I mean keeping previous (binary) versions of your containers within Docker’s layered file system.
  • On the other hand – with multiple similar and/or identical servers running (and this is exactly what we expect from an MOG, at least for Game World servers); using layered file system for binary version control purposes is not really convenient.⁴
• Automated Scaling and Load Balancing. Now we’re speaking.
  • To achieve it, you need to use another layer over usual Docker containers, such as Docker Swarm or Kubernetes.
  • Make sure to understand what exactly Swarm is doing, and make sure to avoid terminating your stateful instances (IIRC, Swarm terminates your containers on “redeployment”, so this is one thing you want to avoid while your Game Servers are running). We’ll discuss in-memory state in a bit more detail below, but for now let’s note that whatever you’re doing – make sure that your stateful instances are not terminated forcefully by your scaler/balancer.

Docker Costs

From what I’ve seen and heard, Docker costs are pretty much negligible. TBH, even VM costs are pretty much negligible (that is, if you control the whole physical host and avoid noisy neighbors⁵) – but with Docker the costs are down to pretty much zero.

On the other hand – make sure not to use crazy-to-start-with configurations using things such as remote-disk-shared-over-NFS; they don’t work without Docker, and while adding Docker won’t make things observably worse – it won’t improve things either.

Docker with an In-Memory State (Immutable but non-Ephemeral)

“I do not buy a rather popular argument that “to achieve scalability, apps and middleware should be stateless”As we discussed in Vol. III’s chapter on Scalability, I do not buy a rather popular argument that “to achieve scalability, apps and middleware should be stateless” (it is the only one way to skin the scalability cat – and, depending on circumstances, it is often not the best one).

A brief recap. In fact – as business requirements very rarely allow for purely stateless implementations of the whole system – going after stateful middleware means that by declaring our apps and middleware stateless, we’re merely pushing the state (as well as all the scalability issues) down to the database (and for most of the environments out there, databases are NOT trivially scalable, in spite of what-your-DB-sales-person-may-tell-you). Moreover, in certain use cases (including, but not limited to, MOGs) blindly making middleware-and-apps stateless often causes increasing DB load by a factor of 10x-100x(!) – which in turn often makes the task of DB scaling pretty much hopeless.⁶

In other words – for our MOG purposes much more often than not, we do want our Game World Apps to have an in-memory state (and yes, this state goes beyond simple caching).

Now back to Docker. While Docker containers are indeed better to be kept immutable, it doesn’t necessarily mean that they cannot hold an in-memory state (i.e. they don’t necessarily need to be ephemeral).

The difference here is that when we’re speaking about Docker images being immutable – we’re actually speaking about a prohibition on persistent state, and in-memory states are perfectly fine. From a practical perspective, even if our app/container has in-memory state, after graceful shutdown of the app/container, we still can avoid storing all those images, and can still upgrade seamlessly.⁷

On the other hand – having substantial in-memory state means that we cannot arbitrarily restart / move our apps and containers (i.e. our containers, while being immutable, will go against Docker’s advice in [Docker.Guidelines], and won’t be ephemeral) <sad-face />. Still, while an ability to dispose of any container just because we don’t like it, looks interesting (and useful too), it is not strictly required; moreover, it seems that Docker Swarm can also be used in a manner which does allow for stateful apps with an in-memory state.

Let’s summarize this in the following table:

Docker Swarm and In-Memory State

BTW, if considering the usage our immutable-but-not-ephemeral containers in Docker Swarm, we need to keep in mind a few peculiarities which aren’t really too common for mostly-web-oriented ephemeral Docker deployments:

• If we’re using rented-per-hour hosts (such as cloud servers) to run our containers, we won’t be able to retire the whole host until all the containers there are terminated. And as we cannot force-terminate our non-ephemeral containers – it leaves us with two choices:
  • Use Swarm‘s DRAIN on the host-to-be-retired (preventing new containers to be created there) – and wait until all the containers on the host terminate naturally. If all our containers have lifetime limited by a small upper bound such as 30 minutes – it might fly.
  • Otherwise – we’ll need to do it in a (semi-)manual manner, identifying those containers which prevent us from retiring host, and implementing our own app-level app migration between containers.⁸ BTW, as it was mentioned in Vol. II’s chapter on (Re)Actors, deterministic containers can be migrated without incurring the latency penalty of full_serialization-transfer-full_deserializiation.
• “Keep in mind that Swarm-provided DNS-based load balancing as such doesn’t apply to game-like loadsKeep in mind that Swarm-provided DNS-based load balancing as such doesn’t apply to game-like loads (as a rule of thumb, with our containers being stateful, we need to balance containers, not incoming requests; not to mention big inherent problems with DNS-based balancing due to DNS caching). For more discussion on Load Balancing in general – see Vol. III’s chapter on Server-Side Architecture.

Summary

To summarize the discussion above:

• Docker MAY be useful for MOG Server-Side deployments.
  • OTOH, most-touted Docker feature of Isolated Components is not really useful provided that we as developers do our job properly
  • Still, other Docker features – such as Security, Resource Quotas, and Isolated Configuration, MAY be useful
  • One especially potentially interesting for us Docker feature is Docker Swarm
  • Keep in mind that primarily Docker is aimed for stateless web apps, so while it can be used for MOGs – take all 3^rd-party advice on Docker with a big pinch of salt (asking yourself – does it really apply to apps with in-memory state?)
• If using Docker, we’ll be aiming for:
  • At least as a first step – we’ll be using Docker for Game World Servers
  • Our Game World Containers will be immutable (i.e. without persistent state) – but will have an in-memory state (i.e. they won’t be ephemeral)
    • As a result – we’ll need to make sure that Docker (including Swarm if we’re using it) doesn’t terminate containers just because it feels like it.
    • Retiring the whole host may be difficult – DRAIN doesn’t always help, and semi-manual migration mechanisms might be necessary.
  • Keep in mind that most of the Docker-related advice available out there, is intended for web-like services without an in-memory state (and with ephemeral containers), so take all 3^rd-party advice with a big pinch of salt.
    • In particular – I do insist that at least for games and game-like services, we should have quite a few our Docker containers non-ephemeral (non-disposable) – otherwise we’ll get very serious performance penalties (see Vol. III’s chapter on Server-Side Architectures for discussion). OTOH – our Docker containers can and should be kept immutable (i.e. each new instance of the container of the same type, can be re-created from the same original image).

[[To Be Continued…

This concludes beta Chapter 25(b) from the upcoming book “Development and Deployment of Multiplayer Online Games (from social games to MMOFPS, with social games in between)”.

Stay tuned for beta Chapter 25(c), where we’ll discuss basics of system monitoring]]

Don't like this post? Comment↯ below. You do?! Please share:

Acknowledgement

Cartoons by Sergey Gordeev from Gordeev Animation Graphics, Prague.