TCP and Websockets for Games

	Author:	“No Bugs” Hare  Follow:
	Job Title:	Sarcastic Architect
	Hobbies:	Thinking Aloud, Arguing with Managers, Annoying HRs, Calling a Spade a Spade, Keeping Tongue in Cheek

TCP Buffers and Priorities

Another Pretty Bad Thing happens if you’re trying to send both large-but-slow data (like a new theme) and small-and-fast data (like those coordinate updates coming 20 times per second), simultaneously.

First of all, if you’re sending a 10Mbyte-file as a single TCP send(), it will often be accepted (even in non-blocking mode). However, with “stream and only stream” TCP ideology, it means that by sending such a large chunk of data, you have just occupied your Client’s incoming channel for quite a while (even for 100Mbit/s client-side connection it would take around a second – even more if something goes wrong so TCP congestion control kicks in). If you try to send something (like coordinate update) during this time, your send() call is likely to be blocked (or return EWOULDBLOCK or equivalent for non-blocking sockets), and you won’t be able to send your urgent data within this second-or-so, ouch.

“this problem can be mitigated by splitting your file in smaller chunks (say, 4K each)To a certain (and IMHO quite large) extent, this problem can be mitigated by splitting your file in smaller chunks (say, 4K each); these chunks would sit in some kind of priority queue on the sender’s side (before send() is called), and would be pushed to the TCP stack (via calling send()) only when socket indicates that it is ready to accept more data for writing (for example, such a socket MAY be observed via select() writefds or equivalent function¹). Then, if new more-urgent data (such as coordinate update) comes in, it goes to the same priority queue, but with higher priority, so that when socket becomes available for writing the next time, it is more-urgent data which will be fed to send().

However, it is not the end of the story (yet). One further thing here is that all the data which is fed to send() function, doesn’t necessarily result in the packet being immediately sent 🙁 (yes, it applies even if TCP_NODELAY is set). Specifically, in the scenario when you’re in the middle of sending a large file in 4K chunks, when the socket is reported to be available for writing, it only means that there is some space in sending buffer. As modern TCP sending buffers are typically in the range of 8K-32K, it means that you’re likely to wait for additional 8-32K to be sent as packets before your higher-priority update gets a chance to kick in.

In other words, if you’re sending high-priority data together with low-priority one over the same TCP channel, and splitting your data into maximum-N-Kilobytes-chunks, and your TCP buffers are M kilobytes in size, then you can easily experience an additional delay of up to (N+M)*client-side-channel-bandwidth. In practice, if N=4K and M=32K, and your player has a 10Mbit/s connection (which is not used for some heavy download at the same time), we’re looking at additional delays of about 30ms – which is not exactly fatal, but does add to not-so-perfect player experience.

“if you reduce the TCP-level buffer to, say, 4K (using setsockopt()’s SO_SNDBUF) – then in the example above you’ll be able to reduce additional latency from 30ms to around 6msOn the positive side, it is possible to reduce the impact of sender-side buffers: if you reduce the TCP-level buffer to, say, 4K (using setsockopt()’s SO_SNDBUF) – then in the example above you’ll be able to reduce additional latency from 30ms to around 6ms.²

On Multiple TCP Connections for Different Priority Data

NB: this is different from “Dual TCP” described above

It might seem that with all the problems with different priority messages going over the same TCP connection, it is obviously better to use different TCP connections for different-priority data. While sometimes it MIGHT be the case, in practice it is not always that obvious.

The main argument against using multiple TCP connections for different priority-data is related to two observations:

and

For example, if you’ll be transferring that large 10M file over one TCP connection, and try to send your high-priority data over another TCP connection, these two TCP connections may still interact (!). In particular, if your file is large enough to saturate Client’s downlink, then your high-priority packets will compete with this file transfer over Client’s downlink (at the ISP-side router facing Client’s “last mile” deciding which of the packets to transfer and which ones to drop).

With multiple TCP connections, you won’t have any control over this choice-made-by-router, which in turn often leads to rather poor player experiences. Using the same TCP connection, you at least can test behaviour of your system within your lab, and to figure out how it will behave in the field. In practice, I’ve seen single-TCP-connection-with-priority-queues working very well for a not-so-time-critical-game (with acceptable latency of 1+ second).

Bottom line about Single-vs-Multiple-TCP-connections for data with different priority

Let’s summarize my personal feelings about single and multiple TCP connections for the purposes of transferring data with different priority (YMMV, batteries not included):

• For not-so-fast games (those with acceptable latencies above 1 second or so), single TCP link with priorities tends to work pretty well
• “For fast time-critical games, it is better to avoid sending different-priority data at the same time completelyFor fast time-critical games, it is better to avoid sending different-priority data at the same time completely (and BTW, it is better to use TCP as a backup only)
  • If this is not possible – I suggest to try both single-with-priority and multiple-TCP-connections-for-different-priorities over real-world links (BTW, simulation won’t give you enough information for multiple TCP connections, as algorithms which routers use to choose a packet to drop, vary significantly and are poorly documented)

On OOB

Remember I’ve told above that TCP “is nothing but byte stream”? Well, I’ve lied (that is, if you didn’t read a footnote). There is one thing within TCP which goes against this principle, and it is known as OOB (Out Of Band data).

Unfortunately, use of OOB (indicated by MSG_OOB flag when calling send()) is extremely limited. In particular, only one measly byte can be sent via OOB at a time [Stevens]: if you specify MSG_OOB flag for a block-larger-than-1-byte, only the last byte will be considered out-of-band data.

Up to this point, I didn’t see any use for OOB in games (and know of only one instance of it being used in any app-level protocol at all). However, keep it in mind – it is quite an interesting capability which MIGHT come handy some day.

Detecting Link Saturation. Conflation

One good thing about the model described above (the one with a priority queue storing incoming data) is that it can be generalized to deal with scenarios where there is only one priority, but your Client’s link still gets saturated one way or another (reasons for saturation can be numerous – from you sending too much to a player’s roommate starting a Huge Download).

“if you’re about to push another coordinate update into the pre-TCP queue, BUT the queue already has the coordinate update – you MAY (and SHOULD) drop previous coordinate update from the queue and replace it with a new oneThis feature may work the following way: if you’re about to push another coordinate update into the pre-TCP queue, BUT the queue already has the coordinate update – you MAY (and SHOULD) drop previous coordinate update from the queue and replace it with a new one (you still MAY use the-last-coordinate-message-already-pushed-to-send()-function as a baseline for compression). This kinda-merging of updates is known as “conflation” and first time I’ve heard about it in the context of TCP updates, was from Alessandro Alinone from [Lightstreamer] (though Lightstreamer implements it in a somewhat different manner).

In a sense, such “conflation” acts as an additional way of flow control, which allows to avoid congesting the client link with the data which don’t really fit there (at the cost of dropping some of the updates, but this is still MUCH better than falling further and further behind).

Terminating connection and SO_LINGER

One setsockopt() option which is often beneficial when using TCP in game context, is SO_LINGER with l_onoff = true, and l_linger = 0.

Normally, when you close TCP socket, it still delivers everything-you’ve-already-called-send()-for to the other side of connection. While such behavior is exactly what’s necessary for serving file transfers and web requests, it is not necessarily good for interactive games. In short – using SO_LINGER with options described abovewill cause TCP stack to terminate connection immediately rather than wait for graceful termination.

In practice, using such non-graceful connection termination for routine disconnects has both positive and negative aspects. On the positive side, it allows to avoid TCP connections in FIN_WAIT2 state from eating too much of server resources (and it MAY become a Big Problem).

On the negative side, non-graceful termination causes a different TCP packet (RST instead of normal FIN) to be used for connection termination; while this is perfectly legal per all the RFCs, some monitoring tools (and quite a few admins) will make a lot of noise about it. On the other hand, except for misreports by some tools, I didn’t see any real negative side for such a non-graceful termination; in particular, I know of a X00’000-simultaneous-player game which has been doing exactly this for 10+ years without any problems.

To summarize my opinion on SO_LINGER with l_onoff = true, and l_linger = 0:

• There isn’t that much difference, and usually it is not that big deal to change later, so it is not that important to decide on it right away
• However, if you ever run into “too many FIN_WAIT2 connections eating too much of your server resources” – it MAY save your bacon
• As for the noise made about too many RST packets by tools and admins – don’t worry much about it (see reasoning above).

TCP and Compression

“One thing which is obviously simpler with TCP than with UDP, is compression.One thing which is obviously simpler with TCP than with UDP, is compression. Basically (unlike with UDP), TCP allows for your usual streaming compression. However (as with pretty much anything else), games have their own specifics even for TCP-based compression.

In short – in game environments, your typical messages are small, and quite a few algorithms out there, while technically working correctly under such conditions, become suboptimal.

One example of such an algorithm which works well for file transfers but starts to behave less-than-optimal for games, is well-known deflate. For deflate, the cost of preparing the block of data for sending (known as flush) is quite large, which causes significant losses in terms of traffic [SI98]. One alternative to deflate, based on the same LZ77+Huffman ideas but optimized for small updates, is LZHL algorithm [SI98].

Another thing to keep in mind with regards to TCP-based compression is that for potentially-conflated fast coordinate-like updates (see discussion in [[TODO!!]] section above), you will actually need two different compression algorithms. The first one is a coordinate-based stuff (using delta compression and dead reckoning), and this one will also participate in conflation. The second one is your usual TCP-layer compression (such as LZHL).

TCP Checksums and Encryption

“TCP checksums (just as UDP checksums) are 16-bit and are not exactly reliable as a result.TCP checksums (just as UDP checksums) are 16-bit and are not exactly reliable as a result. In other words, if you recv() something over TCP, it is not necessarily exactly the same as sender has provided to send() function.³ In general, if using unencrypted TCP, you would need to add your own larger-then-16-bit checksum (at the very least something along the lines of CRC-32 or Fletcher32).

On the other hand, in game environments most of the time there are Big Fat Reasons to keep your traffic encrypted⁴ (see “Why Encrypt??” subsection above for discussion of these reasons). When using any reasonably good encryption, which aims to protect traffic against malicious changes, any accidental changes along the road can be can considered as it is not going to happen, ever category. Note, however, that encryption does NOT eliminate the need for data sanitizing at least on the Server-Side (as malicious Client can easily push the maliciously malformed data to one of the ends of a perfectly encrypted channel).

Encryption for TCP

When speaking about encryption protocols and libraries for TCP, we’ll see that there are MUCH more options available in this department. These include:

• TLS. This is default option for security over TCP and it tends to work pretty well. It also have advantages of being more firewall-friendly (see also section “Magic port 443” below).
  • TLS libraries⁵ include: OpenSSL, LibreSSL, GnuTLS, S2N, mbed TLS, Botan, and probably some others which I forgot about (though for others make sure to double-check for their license – if it is “only GPL or commercial”, usually it won’t be good for your game unless you want to pay).
• SSH. SSH protocol is widely used for secure shells around the world. It CAN be used for applications, but in practice app-level SSH is rather rare.
  • There is at least one decent and currently-supported library [libssh],⁶ and you MAY implement your own messages over SSH. However, using it for application-level is quite uncommon, so you’re likely to have more problems with SSH than with TLS. As a result, I suggest to use SSH only if you have strong feelings against TLS security (and honestly, you shouldn’t – at least not in most of traditional game environment where cost of in-transit compromise is usually fairly low compared to attack costs).
• CurveCP. CurveCP protocol is a very interesting development, and would be very interesting for games; in particular, CurveCP is slim and fast (and the fact that it isn’t as mature as TLS or SSH has little bearing on most of the games out there).
  • However, as of beginning of 2016, there seems to be no supported implementations of CurveCP (CurveCP was separated from supported [libsodium] into [libchloride], and the latter looks pretty much abandoned now), and it is a pity, as implementing a secure protocol yourself is the last thing you want to do when developing your game.

Overall, a brief summary of my take for TCP encryption:

• if your game is not not-so-security-critical – use any of the implementations above, it won’t matter too much
• “if your game IS security-crticial (stock exchange or so) - consider double-encryptionif your game IS security-crticial (stock exchange or so) – consider double-encryption (along the lines described in “Common Encryption-Related Notes” section above)
• DO make sure to read “Common Encryption-Related Notes” section above and to follow all the advice there. It still applies.

On writing ‘better TCP’ (on top of UDP or via RAW sockets)

One thing which pops up in discussions about “TCP better suitable for games” on a regular basis, is writing “better TCP” (either on top of UDP, or on top of RAW sockets).

While it IS possible, I would rather stay away from doing it. TCP stack is quite a complicated beast, and re-implementing it is not easy (even very basic [RFC793] is 80 pages long, and in practice you’ll need to implement much more than just this one RFC).

As a result, for protocols-over-UDP I would certainly prefer a ready-to-use library such as [libquic].

On the other hand, using hacked-TCP over RAW sockets MIGHT have some merit because of being more firewall-friendly; however, I would still prefer if somebody implemented (and tested in the wild!) such a library for me instead of doing it myself. One of the things I would look for in such hacked-for-gaming-TCP, would be working without exponential backoff (but for a very limited time, to avoid congesting the Internet beyond what-we-really-need, see discussion about “time-critical mode” and “connection-seeking mode” in “Retransmission Policies” section above).

“However, even if only 'hacking' the Server-Side of your TCP connections, it MIGHT provide some useful resultsOne further note about hacked-TCP over RAW sockets: you certainly SHOULD NOT count on RAW sockets being available “everywhere” (and even less on them allowing you to create packets-pretending-to-be-TCP). In practice, this means that your “hacked” version will most likely be limited to the Server-Side (most likely running Linux).⁷ However, even if only “hacking” the Server-Side of your TCP connections, it MIGHT provide some useful results (such as limitations on exponential backoff when sending data, and/or allowing to get out-of-order Client-Side data before retransmission-of-missing-packet happens).

Dealing with Firewalls

It is usually a not-so-bad idea to allow your players to play from behind the firewalls. In modern world, you never know where your player will be located next time she plays (and with mobile/hotel/public-WiFi networks, running into a firewall is rather likely).

Magic Port 443

“to allow playing from behind firewalls, one very simple thing tends to help significantly: it is mere listening on TCP port 443Surprisingly, to allow playing from behind firewalls, one very simple thing tends to help significantly: it is mere listening on TCP port 443⁸ 🙂 . Especially if you’re using standard TLS, ISP cannot possibly tell what’s going on inside encrypted portion of TLS, and port 443 is a standard port for HTTPS (which is usually enabled even in hotels), so – the traffic goes through (as a rule of thumb⁹).

Web sockets

One other thing you may want to try to be more firewall-friendly, is to use Websockets (WSS variation a.k.a. WebSockets Secure) as one of fallbacks; I didn’t do it myself, but there were reports that some of the firewalls are less annoying timeout-wise when you explicitly declare that you’re using Websockets (while some other firewalls were reported to disallow Websockets completely). On the other hand, I do NOT expect the difference with plain-TCP-over-port-443 to be anywhere significant.

Note though that quite a few of HTTP proxies tend to have problems with non-HTTPS (WS opposed to WSS) variation of Websockets; as a result, using non-HTTPS Websockets is NOT recommended.¹⁰

Be ready to Frequent Reconnects

Some of the more annoying firewalls out there happen to limit time of your HTTPS session to single-digit minutes. While it is not too big of a problem per se, you need to keep it in mind (and quite often methods described in “Dealing with Hanged Connections” section above, seems to help too).

Websockets: TCP in Disguise

“For most of intents and purposes, you can think of a Websocket connection as of a TCP one (and for WSS connections - as of 'TLS-over-TCP').In a nutshell, Websocket connection starts over TCP as your usual HTTP/HTTPS conversation, and then switches from HTTP Request/Response into good ol’ TCP-as-bidirectional-stream. For most of intents and purposes, you can think of a Websocket connection as of a TCP one (and for WSS connections – as of “TLS-over-TCP”). Most of the differences between Websockets and TCP are related to their support in browsers, but other than that the differences are usually pretty much negligible (especially if you’re using TLS-over-port-443).

Bottom line:

• if your game is browser-based – use Websockets over HTTPS (i.e. WSS)
  • on the Server-Side – make sure to use those TCP options described above (as long as they’re relevant to your game)
  • on the Client-Side, you MAY have quite a bit of problems, which have their roots in the TCP stuff discussed above, but which MAY be more difficult to resolve within JS and without direct access to options such as TCP_NODELAY. See, for example, [StackOverflow.Websockets].
• other than that – feel free to experiment, but most likely, you won’t see much difference
  • note that websockets normally have a bit longer initial connection times (of the order of one extra RTT)

[[TODO: passing embedded Web browser over Client channel]]

[[To Be Continued…

This concludes beta Chapter 13(d) from the upcoming book “Development and Deployment of Multiplayer Online Games (from social games to MMOFPS, with social games in between)”. Stay tuned for beta Chapter 13(e), describing different APIs for handling sockets on different platforms, as well as ways to test network-related stuff.]]

Don't like this post? Comment↯ below. You do?! Please share:

Acknowledgement

Cartoons by Sergey Gordeev from Gordeev Animation Graphics, Prague.