Configuring Hardware for MOG. Part I (Switches, Firewalls, Game Servers)

 
Author:  Follow: TwitterFacebook
Job Title:Sarcastic Architect
Hobbies:Thinking Aloud, Arguing with Managers, Annoying HRs,
Calling a Spade a Spade, Keeping Tongue in Cheek
 
 
Configuring Servers, Switches, and Firewalls
#DDMoG, Vol. VII

[[This is Chapter 23(d) from “beta” Volume VII of the upcoming book "Development&Deployment of Multiplayer Online Games", which is currently being beta-tested. Beta-testing is intended to improve the quality of the book, and provides free e-copy of the "release" book to those who help with improving; for further details see "Book Beta Testing". All the content published during Beta Testing, is subject to change before the book is published.

To navigate through the "1st beta" of the book, you may want to use Development&Deployment of MOG: Table of Contents.]]

After you rented your hardware (the one which we discussed a bit earlier) – you need to configure it. Whether we like it or not,1 after we got the hardware – we need to configure it. In this regard, some things are obvious – but some are not; in this section, we’ll try to concentrate on those not-so-obvious things (which, if done wrong, can cost you lots of time – and often even customer loyalty – down the road).

Hare pointing out:this section is NOT intended for serious adminsBTW, a disclaimer – this section is NOT intended for serious admins; 90% of the stuff you’ll see here – is perfectly trivial for any reasonably good admin (but is still very non-obvious and counter-intuitive for most of developers <sigh />). If you’re an admin – feel free to skip to section [[TODO]] on database servers (and there, I will mention some things which are not that obvious).


1 Accidentally, I happen to like it <wink />

 

Network Equipment

Before starting to discuss interesting stuff such as servers – let’s discuss issues which are related to network equipment.

Overall, in this book, we won’t go into serious discussions on “how to setup Ethernet trunking between your switches”, “how to run your own BGP router” and so on (when2 you need this kind of stuff – you WILL need a really serious admin on your team). Instead – we’ll concentrate on simple things – which can be done if you need to run your system all by yourself, without serious admins to do it for you; apparently – it is possible (I’ve seen a game with tens of thousands of simultaneous players which was running very successfully without dedicated admins for years, though I would suggest not to do it and to hire an admin as soon as you can afford it).


2 not “if”

 

Ethernet Switch

For Ethernet switches – configuring is fairly simple. In addition to asking-your-ISP-to-connect-cables, and besides those VLANs shown on Fig 21.1-21.2, there is basically only one thing which you really need to keep in mind when dealing with Ethernet switches – it is auto-negotiation.

The problem here is that whenever we’re connecting two Ethernet devices – they need to decide which speed (100Mbit/s, 1Gbit/s, …) and which duplex-mode (half-duplex or full-duplex) they need to use. If the devices are in “auto-negotiation mode” – then they’re supposed to find out the best mode by negotiating it; if device is in “manual” mode – it just does what it is told to do (BTW, it means that having auto-negotiation at one end, and manual mode at the other end of connection is very likely to cause trouble).

The problem we’re dealing with here, is that if sides of Ethernet connection, for whatever reason are using different settings – the connection will kinda-work, but it will cause weird stuff such as packet loss and so on.

Femida hare:With regards to configuring auto-negotiation, among admins there are two schools of thoughtWith regards to configuring auto-negotiation, among admins there are two schools of thought: the first one says that “smart auto-negotiation protocols will do it reliably itself, so any kind of setting things up manually will cause a potential for misconfiguration – hence, just leave everything at ‘auto’, and it will magically work”; quite obviously – switch vendors firmly belong to this camp (doing otherwise would be interpreted as an admission that supposedly-working stuff doesn’t really work on their switches). Another school of thought says that “hey, in real world auto-negotiation often misfires, creating really crazy scenarios of half-working Ethernet connections with lots of packet loss – hence, let’s configure everything manually”.

From my experience – I am seriously leaning towards the second camp; moreover – from what I’ve seen, when facing Ethernet parameter mismatch after year 2005 or so, we’re not really speaking about incompatible implementations, but rather about not-ideal cables (and BTW, Ethernet cables tend to be pretty bad in rented ISP environments <sad-face />). If the cable is not ideal (like poorly crimped, or having been bent too sharply once) – it can intermittently cause trouble (and in general – it should be replaced, but the problem is how to know that it should be replaced – see also about monitoring below). However, the cost of this trouble-caused-by-somewhat-broken-cable tends to be very different during normal operation (where most of the cables will only cause very occasional packet loss) – and during negotiation (if we get a problem during auto-negotiation, then the whole communication will be mismatched and will cause lots of trouble).

As a result, the following list of guidelines tends to be a good starting point:

  • Choose ONE of the following strategies:
    • either keep everything (=”both switch and servers”) at auto-negotiation
    • or set everything as manual (this includes all the switches/firewalls and all the servers)
    • Any other approach is pretty much doomed from the start
  • Personally – I prefer “set everything as manual”; it is especially good if we can say that all (or at least almost-all) our connections are the same (and this is pretty much the case for modern Server-Side setups, where all the Ethernet connections are 1Gbit/full-duplex; sure – you can get an occasional 10Gbit/s link or a 100Mbit/s uplink, but they’re not frequent enough to cause significant configuration trouble).
  • For uplink connections (such as those going to the ISP) – make sure to ask your ISP what they’re using on their ports.
  • Most importantly, however – make sure to monitor your switches and links on continuous basis (more on monitoring in [[TODO]] chapter). It will allow to detect Ethernet-level problems (and 80+% of these problems will be problems with cables) as early as possible.

[[TODO: Ethernet-level redundancy (here or in Fault Tolerance?)]]

 

Firewall

As we discussed above, in MOG environments there are two types of firewall: simple packet filters (usually used for game traffic3), and complicated stateful UTM-like boxes. Here, we won’t go into discussions about the differences between these two classes of firewalls; however, there are a few things which need to be remembered when a newbie-like-me tries to configure a firewall:

  • Make sure to create several “trust zones”; at the very least – you need to have zones shown on Fig 21.1: “untrusted” zone (Internet), “trusted zone” (most of your servers), and “DMZ” zone (for your web/mail server(s)).
    • Each of “trust zones” should be connected to its own VLAN on your switch (see Fig 21.1 for an example)
  • Your Internet-facing firewall should do some kind of NAT, essentially hiding IPs in your internal network from the outside.
    • CIDR notation is a compact representation of an IP address and its associated routing prefix. The notation is constructed from an IP address, a slash ('/') character, and a decimal number.— Wikipedia — Within your private network – make sure to use ONLY “private” IP addresses such as 10.x.x.x, 192.168.x.x., or 172.16.x.x-172.31.x.x (BTW, if you want to get brownie points when communicating with admins – don’t forget to write these ranges – as well as all the other IP ranges – in so-called CIDR notations as 10.0.0.0/8, 192.168.0.0/16, and 172.16.0.0/12 respectively)
  • Default behavior for the firewall (at least for the packets coming from the Internet) should be DROP (or REJECT4).5 In other words – if your firewall doesn’t recognize what-to-do – make sure that packet doesn’t come through.
    • As for outgoing connections – strictly speaking, for production environment you should also have “default is DROP/REJECT” policy, but usually for outgoing connections/packets having an “ACCEPT by default” policy is not too bad (unless you’re running stock exchange, casino, or a well-established game with artifacts-sold-for-real-$20k).
  • As for ICMP packets – make sure that at least ICMP with (Type 3, Code 4), a.k.a. ICMP Fragmentation Needed is allowed (at least within the context of established TCP connections) on all your firewalls, which handle some kind of TCP.
    • PMTUD Path MTU Discovery (PMTUD) is a standardized technique in computer networking for determining the maximum transmission unit (MTU) size on the network path between two Internet Protocol (IP) hosts, usually with the goal of avoiding IP fragmentation.— Wikipedia — This is necessary to support so-called PMTUD – which is currently used by default by pretty much all the TCP stacks out there; while it is easy to disable PMTUD on your servers – disabling it on Clients can easily become too ugly (as a rule of thumb, even if we’re allowed to do it – we don’t want to mess with system-wide settings on Client devices)
    • These days, most of the firewalls enable these ICMPs-required-for-PMTUD by default, but if you happen to have a developer who just read that “all ICMPs are evil, and MUST be disabled” – he might be able to find a way to filter out all ICMPs, including “Fragmentation Needed” ones over established TCP connection, and doing so will lead to problems which can be very-difficult-to-identify to the untrained eye.
    • As for the other types of ICMP – it is debatable whether to block them or to let them through (in particular, ICMP ping – “Echo Request/Reply” in ICMP-speak – is more often disabled than not); these I don’t care much about, and at your first deployment, you can do whatever suits your fancy with them

Once again – by all means it is NOT an exclusive list; however – it is a list of mistakes which are pretty common when firewall is configured by developers, and these mistakes are better to be avoided.


3 BTW for UDP, you cannot really do more filtering anyway
4 for the purposes of this Chapter – I will use terminology from iptables; while other firewalls may name the same things differently – they will support the same concepts for sure
5 I don’t want to go into discussion on “what is better – DROP or REJECT”, it is debatable and not really important at our current level of discussion

 

Servers

With regards to your Server boxes – as we discussed above, there are three common types of them in the context of an MOG: (a) Game Servers (such as Game World Servers, Matchmaking Servers, etc.), (b) OLTP Database Server and (c) other Storage Servers (including DB Replicas, analytics, backup storage, etc. etc.). Let’s discuss them one by one.

Game Servers

The most common server you’ll have – is a Game Server (to run your Game Worlds, Matchmaking, and so on). Usually – Game Servers are only about CPU+RAM+NIC (and any persistent storage is used just for OS plus apps plus some kind of local logging). As a result – most of the time, Game Servers are using 2S/1U boxes (which tend to be the best ones price/performance-wise); sometimes, though, 2S/2U boxes can compete with them pricewise.

Assertive hare:last but certainly not least, we’re removing everything we can remove without hurting that-part-of-OS-functionality-which-we-really-needConfiguring Game Server is usually fairly simple: first, we choose our OS and install it, then – we configure a few minor items such as RAID/swap/etc., and last but certainly not least, we’re removing everything we can remove without hurting that-part-of-OS-functionality-which-we-really-need.

BTW, here we’re NOT speaking about “choosing OS” as “choosing between Windows and Linux” (this choice has been made long ago, when you just started to write your apps for the Server-Side). On the other hand – the choice between different versions and/or distributions is still usually available at this point.

 

Linux (with FreeBSD on the side)

As it was discussed at length in Vol. III’s chapter on Server-Side Architecture – most of the time, we should use Linux for the Server-Side; in particular, even if we consider everything else pretty much equal,6 Windows license costs can hurt you pretty badly, especially if you need to run several hundred Game Servers.

When deploying a reasonably-written Linux app-level program, it shouldn’t depend on the specific distribution. This essentially leaves a choice of distribution to the deployment time (which is a Good Thing™; the less things we carve in stone – the more flexibility we have in production, and all kinds of things can affect our choices over multiple years in production).


6 and I would argue that in spite of security gap being reduced, security-wise Linux is still significantly ahead of Windows

 

Choosing Linux Distro

As for choosing your Server-Side Linux distro:

  • For Game Servers, distro you choose doesn’t matter too much – but you still need to make a choice
    • I know that I will be pummeled for saying this – but as an honest hare, I have no option to keep it from you: for Game Servers, file system doesn’t matter (just because there are very few things mostly-in-memory-processing Game Server does with disk). [[TODO: avoid jumping the-most-recent-but-not-really-mature-fs bandwagon; as of 2017 – btrfs still gets enough complaints to be avoided (even if it is default by your distro)]]
  • First, you need to decide between RedHat-related group of distros vs Debian-based one.7 Having a free choice (and it is usually the case for Game Servers) I tend to prefer Debian (though RedHat-based ones are also perfectly workable).
    • Wtf hare:there are distros beyond the RedHat/Debian dichotomy – but if you’re reading this section looking for my advice on it – most likely you still have too little experience running Linux to look in that directionthere are distros beyond this RedHat/Debian dichotomy (notably Arch Linux) – but if you’re reading this section looking for my advice on it – most likely you still have too little experience running Linux to look in that direction
    • In addition, there is also a FreeBSD; however – keep in mind that FreeBSD is not Linux (though it is Unix), so depending on whatever-your-game-app uses, you may need to adjust your game app a bit; still – usually it is not a big deal (especially if you’re following advice from Vol. III on separating your infrastructure code from your game logic). OTOH, once again: if you’re still reading this advice – you’ll probably be better with mainstream RedHat-/Debian-based distros
  • Within RedHat-related distros – the most popular Server-Side choices are RedHat, CentOS, and ClearOS (server-oriented). Out of these, I know that RedHat and CentOS work fine, but would certainly like to try more recent and supposedly-more-lightweight ClearOS.
  • If you’re more of a Debian fan8 – then you have Debian stable, Debian testing, Debian unstable, and Ubuntu. Out of these, my order of preference is Debian stable – Debian testing – Ubuntu – Debian unstable. The rationale for this preference is simple: most of the time – Debian stable will do everything your game app needs, and with Debian stable being the most rock-solid distro I know – there is no reason to introduce any unnecessary risks into your production.

7 mostly, it is a choice of rpm/yum package manager vs apt/deb one – but there are quite a few other differences between these two major branches.
8 As I am <smile />

 

Configuring Network

Usually, if you can get to your Server box (except via Out-of-Band Management) – it means you have your networking configured. However, if you’re installing a new box from scratch (using some kind of “Virtual Media”) or are accessing your Server via Out-of-Band Management – you still may need to configure your network.

Assertive hare:as a Big Fat Rule of Thumb™, your Server boxes should have static IPsIn this latter case – keep in mind that as a Big Fat Rule of Thumb™, your Server boxes should have static IPs (while running DHCP in server environment is possible, it is rarely a good idea) – those from “private IP ranges” discussed above.

[[TODO: consistent IPv6 policy (configure vs disable = more-work-for-greater-good vs simplicity)]]

Configuring RAID

For most (if not all) your Game Servers – you’ll be using your drive only to boot your OS, launch your app and write your log files. As a result – requirements for RAID for these boxes are pretty trivial. Strictly speaking, most of the time you can even run your Game Servers9 without any RAID at all: as discussed in Vol III’s chapter on Fault Tolerance, most likely, you’ll need to handle Game Server failures anyway (effectively relying on having lots of Game Servers and rebalancing the load among remaining ones10), and in this case failure due to HDD/SSD will be just one of those failures-to-take-into-account.

On the other hand – if RAID is cheap (as it usually is), it still might make sense for Game Servers; if going this way – software RAID-1 (mirror) of two identical drives is usually good enough for Game Servers. To improve over simple 2-drive RAID-1, you usually can either have a third drive as a hot spare, or have a 3-drive (3-way) RAID-1 – but honestly, for most of Game Servers the 3rd drive is an overkill.

One very important note though:

All RAIDs are useless unless you know about drive failures in timely manner.

The whole point of RAID is that if one of your mirrored drives fails – RAID detects the failure, and continues to work off the remaining drive. So far so good11 – but you DO need to replace that failed drive before the second HDD/SSD fails too.12 BTW – dual failures can easily happen much sooner than you’d expect from assuming that the failures are random – especially if the failure is caused by a manufacturing defect affecting the whole batch where your drives came from. I heard even of same-day failure of two drives, but fortunately, this is extremely rare (BTW, chances of same-day failures are higher for mirrored SSDs because of some SSD failures having common causes; this, however, can be usually mitigated by using S.M.A.R.T.); still – detecting and fixing drive on a “next business day” is a good practice.

As a very practical result of the observation above –

Make sure that you DO have monitoring app which reports RAID drive failures.

How to do it – is a different story, and we’ll discuss the whole topic of monitoring your system in [[TODO]] chapter.


9 though NOT DB or storage servers – more on them below
10 even if with certain loss of in-memory state
11 Well, except in certain cases of partial failures which are not properly detected by RAID; fortunately – these are very rare.
12 In case of hot spare or 3-way mirror – you have more time, but still postponing replacement forever is not a good idea.

 

Disabling Swap

Most of the time – we won’t want our latency-sensitive Game World Servers to swap anything out. Hence – it is often (though not always) a good idea to disable swap file on your Game Servers, just to prevent OS from being a smart guy and swapping stuff-which-it-thinks-is-unnecessary out (in a typical gaming environment, we usually DO know that all the stuff we run, IS necessary – and IS latency-sensitive too; a situation, which is vastly different from typical desktop installs, where disabling swapping is rarely a good thing).

Surprised hare:Beware, however, of inherent dangers of disabling swap file.Beware, however, of inherent dangers of disabling swap file. If there is any way to force too many memory allocations from the outside (for example, if you allow an unlimited number of connections to your server, with each connection consuming resources) – then without a swap file you can easily end up with allocation failures (which will usually crash your whole system13); on the other hand – with a swap file, there is a (relatively small) chance of surviving such an occurrence. Still, I’d say that you SHOULD aim to eliminate such attack-vectors-consuming-all-the-RAM completely, so in the long run you should arrive to a situation when disabling swap becomes perfectly feasible (and usually becomes a good idea too).


13 at least unless you tested it for such scenarios – and really thorough testing against all possible scenarios of memory allocation failures is usually not realistic for games

 

Disabling Unnecessary Stuff

After you got your new shiny install – you need to make sure that you’re not running anything which you don’t really need. There are two Really Good™ reasons for it:

  • Having unnecessary stuff listening for packets is a recipe for security disaster
  • From time to time, this unnecessary stuff can easily cause strange activity, generating additional load/causing additional latencies/… (and Murphy’s law will ensure that this slow down will happen at the worst possible moment for your game; ideally – it aims to cause your 20-byte allocation during-your-Tournament-of-the-year to fail).

Usually, you need very few things to run on your production server; for example, on my own Debian box (which I didn’t spend much time on minimizing) – I am getting away with only the following services running (BTW, the list of services is obtainable at least on Debian via “service –status-all”: acpid (*), atd (*), console-setup, cron, dbus, kbd, keyboard-setup, kmod, mdadm-raid (*), networking, procps, rc.local, rsyslog, ssh, udev, udev-finish, and urandom; that’s it(!). NB: I marked with (*) those services which are IIRC usually avoidable for servers (but which I’m too lazy to disable on my own not-so-critical box <wink />).

As you can see – it is already pretty short list (by today’s standards, that is) – and probably can be shortened even further. Even more importantly – this configuration box doesn’t have any ports open, except for ssh/22:

#Holy Grail of netstat
netstat -tunap | grep LISTEN
tcp&nbsp; &nbsp; &nbsp; &nbsp; 0&nbsp; &nbsp; &nbsp; 0 0.0.0.0:22&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0.0.0.0:* &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; LISTEN
567/sshd
tcp6 &nbsp; &nbsp; &nbsp; 0&nbsp; &nbsp; &nbsp; 0 :::22 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; :::*&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; LISTEN
567/sshd

As we can see – the only port open on the server, is port 22 (SSH)14, listened by sshd (of course, when we run our game app – there will be our own port(s) open too – but for the sake of our players, let’s hope that these don’t count as “unnecessary stuff” <wink />).

Some further considerations:

  • List of things to disable for sure:
    • Any kind of UI stuff – however you may like UI, it doesn’t belong on production Linux servers, plain and simple. To manage production servers – the best is to learn the command line (and also learn how to enjoy it – or at least Midnight Commander).
      • If you absolutely cannot live without UI – you can use that out-of-band management we discussed above (which is pretty much a VNC in a disguise); I’d still stay away from VNC-running-within-your-OS and equivalents (such as X11RDP) as long as possible.
      • BTW, if you need to administer 100 servers or so – server-side UI is likely to be of a very little help anyway.
    • NFS and closely-related portmap and rpc* services. Way too much of a security hole for never-used-anyway kind of stuff
    • Printer daemons (lpd anyone on server?)
    • Hare with hopeless face:if your distro enables telnet or ftpd by default in 2017 – it is a good reason to think if you chose the right distrotelnet and ftpd (actually, if your distro enables telnet or ftpd by default in 2017 – it is a good reason to think if you chose the right distro)
    • named, bind, inetd/xinetd, httpd, and any kind of pop3/smtp/imap server
    • anything which listens on network ports (except for absolutely-necessary stuff)
  • After disabling unnecessary services (using whatever-method-you-Googled-for-your-distro) – make sure to reboot and double-check that both your list of services and list of ports are still as-you-want-it (it is easy to disable a service only temporarily, so it will come back after reboot).

14 Both for IPv4 and IPv6

 

[[TODO: time sync and ntpd (with “-x” option); time zone as UTC]]

Restricting Permissions for your App

As a Big Fat Rule of Thumb™ – your Server-Side app (such as Game World Server App, Matchmaking Server App, etc.) should run as a so-called daemon (an unattended program which starts as soon as server box is started).

Judging hare:your daemon should run under special restricted user account (not under root, for Linus’ sake)Moreover – your daemon should run under special restricted user account (not under root, for Linus’ sake). How to do it – depends on the flavor of your Linux distro (=”it’s different for Debian-based ones and RedHat-based ones”), but in any case you’ll likely need to mess with some script in /etc/init.d/ (while you can start your daemon using start-stop-daemon or daemon scripts ONCE – you want it to start in the same manner after each reboot, and this is done via scripts in /etc/init.d/).

The next question is – which permissions do you want to grant to this account-running-your-game-app? The answer is simple – “the absolute minimum you need”; in practice – Server-Side Apps tend to need only a bare minimum (interact with network and write your own log files, that’s pretty much it) – which can be done under usual user account (unless you’re using ports < 1024; BTW, if you DO need these “privileged” ports – it is still usually possible to avoid running your app as root – see, for example, discussion in [StackOverflow]).

 

Further Hardening

In general, it is a good idea to harden your Internet-facing Server against attackers further; two most common mechanisms for further hardening are chroot and SELinux. Most likely – if you’re reading this section (NOT intended for serious admins), both these tools are an overkill for your first deployment, but I still will provide a very brief overview of them:

  • chroot (a.k.a. chroot jail) is a tool which allows to create a “private” file system for your app, so it doesn’t really see anything else but its own directory. This limits options for the attacker to get to the rest of your system when he already got your app.15
    • Overall, chroot (both for your own apps and for 3rd-party apps such as web server) provides limited security benefit – at the price of a modest headache
  • Hare wondering if you are crazy:chroot and SELinux can be used together to supposedly achieve even-better protection than any of them separatelySELinux. SELinux, if used properly, provides significantly better protection than chroot – but it comes at the cost of Really Significant Headaches™. For stock apps (such as web server etc.) there are often reasonable SELinux policies; however – setting SELinux for your own app, while certainly possible, is guaranteed to take lots of time. As a result – if you’re still reading this section – chances are that you’re not ready to bear the pain of configuring SELinux for your own app (at the least I know that I myself am not ready <wink /> – at least unless it is stock exchange or casino I am going to run).
  • BTW, chroot and SELinux can be used together to supposedly achieve even-better protection than any of them separately; I don’t feel qualified to comment whether it is really the case – but I know that at least [Vermeulen] argues for it.
  • [[TODO: LXC, maybe LXD]]

15 usually, if the only meaningful app you have on the box is broken, preventing takeover of the whole box provides only limited practical relief – but usually there are still reasons to prevent further propagation of the attack.

 

[[TODO: OS update policy]]

Windows

Amicus Plato, sed magis amica veritas

(Plato is my friend, but truth is a better friend)

— attributed to Aristotle —

Sometimes, it just so happens that you need to run Windows in production. In addition to expressing my sincere condolences, I can give the following advice:

  • Unlike Linux, IMNSHO Windows is still not mature enough security-wise to face the the Internet (translation: you REALLY need a NAT-ing firewall between your Windows boxes and the Internet).
    • For Linux, firewall is also desirable, but in some cases it can be avoided; for Windows – it is still pretty much THE MUST <sigh />. If you have any doubts – history of Windows worms-which-require-just-one-packet to get you infected (Nimda and Code Red in 2001, Blaster in 2003, and WannaCry just now, in 2017) doesn’t really allow to say that Windows-facing-Internet can be considered as a reasonably good practice
    • As for using Windows software firewall for production Servers instead of the external one – I suggest to avoid it for two reasons: (a) software firewall is way too easy to misconfigure inadvertently (which instantly defeats protection), and (b) it does slow things down, which is a big problem at least for latency-critical games
  • On the positive side (yes, there are positives for Windows too) – you do not need to think much about choosing version of the software you want to run; usually, you won’t be able to get anything but the latest-greatest version anyway (and the choice between Editions is usually limited to Standard Edition because anything less won’t satisfy licensing requirements, and Datacenter Edition is outrageously expensive).
    • As for licensing – my current understanding (absolutely no warranties of any kind), is that for Windows Server 2016 to run your MOGs, you need to pay for (a) per-core license (~$900 total / 16-core server for Standard Edition), (b) External Connector license (~$2000 / server for Standard Edition), (c) CALs (or MLs) for your admins (some such licenses might be included into per-core pack – but make sure to check)
  • Hare thumb up:every time I disable Active Directory Service from a production server – I feel as a doctor who successfully removed a cancer tumor from a patientNetworking configuration: static IP addresses mentioned above, still apply to Windows. In addition (and I know I will be beaten really hard for saying so) – I strongly advise to avoid placing your Windows Servers into a Windows Domain. Keep your Servers standalone – and your system will be more reliable and more resilient to attackers too (admittedly, at the expense of certain headaches). This is also closely related to disabling unnecessary stuff (every time I disable Active Directory Service from a production server – I feel as a doctor who successfully removed a cancer tumor from a patient).
  • Just as with Linux, software RAID-1 (known as “drive mirroring” in Windows-land) is a good idea (and yes, it seems to work for boot drive too).
  • Disabling swap under Windows, while possible and sometimes desirable, is somewhat more risky than for Linux; especially so if you haven’t disabled really-resource-hungry services such as Search/Indexing (and under Windows, it is really difficult to be 100% sure that you did disable ALL of such services).
  • Principles of disabling unnecessary stuff (discussed in [[TODO]] section above) still apply to Windows-based servers. However – the process of disabling the services is usually MUCH more involved on Windows. The cycle usually goes like “disable some service which you have no idea about” – “reboot” – “see if something important failed to start” – “try your app” – “rinse and repeat”. Still, while time-consuming, it is very important especially for production Windows Servers. While there are literally dozens of services to disable – some of the most annoying ones are listed below:
    • Search Service, formerly Indexing Service (you don’t want it to wake up and slow your server to the crawl while your Tournament of the Week is playing, right?)
    • Everything you can disable and which shows in “netstat –na” as LISTEN
      • Most likely – unlike for Linux, even after disabling everything-you-can-disable, you’ll still have quite a few unnecessary ports open. It is actually one big reason why Windows is still not ready to face the Internet without a firewall.
    • Even if you DO need to run your web server on Windows (and even if you’re running .NET) – make sure your web server is not IIS.16 Yes, I know I will be pummeled really hard for saying this – but as they say, “Plato is my friend, but truth is a better friend”
      • BTW, internal boxes running reporting etc. is a different story – and it is ok to use IIS there.
      • If you absolutely cannot get out of running Internet-facing IIS in production – at least run nginx, Apache, or HAProxy as a “reverse proxy” between Internet and IIS (and while you’re at it – also make your “reverse proxy” serve all the static content).
    • Unnecessary client stuff such as DHCP Client (as noted above – you won’t run DHCP on your Servers). While it is not that big deal as having ports open – regular polling for DHCP is still a potential vulnerability.
    • NetBIOS support and (Lanman)Workstation / (Lanman)Server. If possible at all (and it admittedly changes between different Windows versions) – I strongly suggest to disable these for security purposes (and use only RDP for all the management purposes, including file transfers).
      • Admittedly, automating deployments won’t be easy without shares – but I’ve heard good things about PowerShell and PModem. And while PowerShell itself can be a security hole – it is still unlikely to be worse than bug-ridden-for-decades SMB stuff (if you have any doubts – keep in mind that most recent WannaCry worm exploited yet another bug in SMB).
    • To run your app after server reboot (and to make it similar to *nix daemons) – you need to teach your app to run as a “Windows Service”. When this is done – restricting permissions for your app become quite simple (and as a good starting point – try to configure that service-running-your-app under a predefined NetworkService account).

16 Yes, I know IIS is gaining in popularity (OTOH, I tend to be quite immune to “millions of flies can’t be wrong” line of argument). In any case – don’t ever come to me asking to solve your IIS-related problems for Internet-facing servers; the only answer you’ll get, will be “throw it away and replace with something decent”.

 

[[To Be Continued…

Tired hare:This concludes beta Chapter 23(d) from the upcoming book “Development and Deployment of Multiplayer Online Games (from social games to MMOFPS, with social games in between)”.

Stay tuned for beta Chapter 23(e), where we’ll discuss certain issues related to configuration of your DB Servers (and physical layout of databases)]]

Don't like this post? Comment↯ below. You do?! Please share: ...on LinkedIn...on Reddit...on Twitter...on Facebook

Acknowledgement

Cartoons by Sergey GordeevIRL from Gordeev Animation Graphics, Prague.

Join our mailing list:

Leave a Reply

Your email address will not be published. Required fields are marked *