Client: Installer & Auto-Updates

 
Author:  Follow: TwitterFacebook
Job Title:Sarcastic Architect
Hobbies:Thinking Aloud, Arguing with Managers, Annoying HRs,
Calling a Spade a Spade, Keeping Tongue in Cheek
 
 
Update of Updater

Update of Updater

#DDMoG, Vol. V

[[This is Chapter 18 from “beta” Volume V of the upcoming book "Development&Deployment of Multiplayer Online Games", which is currently being beta-tested. Beta-testing is intended to improve the quality of the book, and provides free e-copy of the "release" book to those who help with improving; for further details see "Book Beta Testing". All the content published during Beta Testing, is subject to change before the book is published.

To navigate through the "1st beta" of the book, you may want to use Development&Deployment of MOG: Table of Contents.]]

Installing

For want of a nail the shoe was lost.
For want of a shoe the horse was lost.
For want of a horse the rider was lost.
For want of a rider the battle was lost.
For want of a battle the kingdom was lost.
And all for the want of a horseshoe nail.

— Nursery Rhyme —

Actually, the most important thing when speaking about installing your Client, is not the installer as such, but the way how your Client is organized. I’ve seen way too many Clients which were re-using indiscriminately – and, as a result, the whole Client was a mess of 3rd-party DLLs, and 3rd-party components – and these were causing endless troubles to installer guys, to players and to CSRs; all because a (rather natural) desire of developers to re-use wasn’t kept in check.

Making your Client Installer- and Player-Friendly

Once upon a time, I was working for a company which delivered such a program-with-tons-of-dependencies. A friend of mine, who was responsible for the installer, told quite expressively on more than one occasion: “You see, our program is just a lot of **it. And to install it, I need to take this **it into my hands, and carry it to the customer’s PC, and without spilling a drop”. It was long ago – but I still remember that expression of the real pain which was present on his face while he spoke. It was at that point when I realized how important it is to make sure that programs we write, DON’T need to be deployed like **it.

Hare thumb down:From the point of view of organizing your Client, the worst possible thing is 3rd-party dependencies shared with other programsFrom the point of view of organizing your Client, the worst possible thing is 3rd-party dependencies shared with other programs. If your Client uses a 3rd-party DLL (or COM component) – and this DLL/COM is in a common folder shared with the other programs – you can count on the following:

  • Some of your players will have different versions of the DLL/COM. It means that you MUST test your Client with all the possible DLL/COMs (ideally – including betas etc.) – which is not realistic.
    • BTW, DON’T think that player-using-beta-DLL is not your problem; from the player’s perspective, if it is your Client which crashes – it is you who’re guilty, period.
      • Once upon a time, I ran into a problem where like 0.01% of players (like 100 people) were having BSOD – and BSODs were reported to happen only while playing our Client.1 As we weren’t doing any driver/ring0 stuff – BSODs couldn’t possibly be our problem, but explaining it to the players was outright impossible (that is, until hardware manufacturer has updated the driver for the laptop video card – and then the issue became moot).
    • In addition, you can count that at least on some of your player PCs, there will be a 4th-party program to replace exactly that shared DLL/COM you’re using – with their own “improved” version (which will break your Client); this kind of trouble is known as “DLL Stomping”
      • It is more common as you might think (there is a reason why it got a special name) – and no library is safe from such replacement. Once, I saw MSVCRT.DLL replaced by an “improved” version(!)
      • While Vista+ (and WFP) does make DLL stomping more difficult – there are still ways for enterprising developers to do it.
      • DON’T think that it is not your problem – it is. See discussion above, why.
    • Just for reference – it is these shared DLLs which are still causing a phenomenon aptly named “DLL Hell”.
      • BTW, DLL versioning – being “sold” to us as a solution to “DLL Hell” problem – is just an attempt to kinda-fix the problem which should have never existed in the first place. I contend that private DLLs are MUCH simpler to deal with, and are MUCH less error-prone than “versioned” DLLs.

As a result – rule #1:

At the very least, install your DLLs privately.

Arguing hare:while waste-for-no-reason is always bad, here we have a Very Good Reason(tm) – to keep players happyYes, I know that it goes against every other book (and yes, this additional DLL won’t be able to be shared with other running programs, inevitably causing extra memory usage). However – with 1G of RAM being a realistic minimum for modern PC, what are the chances that mere code of three DLLs (with 200K size each) will cause any realistic trouble? We’re speaking about wasting 600K/1G ~= 0.06% of RAM; while waste-for-no-reason is always bad, here we have a Very Good Reason(tm) – to keep players happy (and if your Client will crash less – you will make your players happier). This noble goal IMNSHO overrides all the concerns about saving those few kilos of RAM.2

BTW, the rule #1 above does NOT apply to system DLLs (you won’t be able to make kernel.dll private anyway); on the other hand – even MSVCRT.DLL should be made private (or even better – compiled statically, see below).

[[TODO: security implications; in short – keeping DLLs up-to-date becomes a necessity, but with typical update cycle being like once-a-week – it is not really a problem]]

Rule #2 says that

Even better – compile your 3rd-party libraries statically.

Replacing DLLs with statically compiled libraries has several benefits, but the most practical one is that static libraries are significantly less vulnerable to reverse engineering. We’ll discuss more of it in Vol. VIII (chapter on Bot Fighting) – but for now let’s just note that DLLs introduce very-easily-seen points into your program, which helps dissecting your Client significantly. One ultimate case of carelessness in this regard – is using OpenSSL.dll (it provides a very direct attack vector on your protocols, which is one thing that you really really want to protect if you care about the MOG bots).

Static libraries are clearly better than DLLs, but there is one further step we could do. Rule #3:

Better still, remove the dependency entirely.

BTW, I am deadly serious here. From what I’ve seen – at least 50% of the dependencies used in downloadable Clients, shouldn’t been used in the first place. Overall – the fewer dependencies your program has – the fewer bugs it will have too; this is especially true if you’re trying to re-use a huge obscure component just to add some very minor bell and whistle to your Client (and it happens all the time; once I saw a game Client which used IE HTML control merely to show an animated palm tree while Client was being started – and it apparently happened to cause a crash of the Client if player’s PC had a very specific build of IE).

Hare wondering if you are crazy:honestly, if you’re using a dozen or so 3rd-party DLLs – you SHOULD make sure to go through the list and see whether you really want to use each of them.However, let’s note that I am not exactly arguing that you should aim for an executable which depends only on kernel32.dll (while I did manage to produce such executables myself – it wasn’t a full Client, but just a part of it3) – but honestly, if you’re using a dozen or so 3rd-party DLLs – you SHOULD make sure to go through the list and see whether you really want to use each of them.


1 It also happened that BSOD was happening only on a very specific model of laptop
2 I remember one time when the argument about saving RAM by sharing DLLs was really important – but it was a time of Windows 3.1 and computers having 1M of RAM; with 1G of RAM becoming minimum (while size of code didn’t change much(!)) – well, this argument doesn’t really fly anymore.
3 Last time I’ve checked, sockets still weren’t the part of kernel32.dll, so networked-program-without-dependency-on-ws2_32.dll was pretty much impossible (at least as long as we’re not going to write our own network card drivers)

 

Just Bunch of Files

As soon as you removed/compiled-as-static/made-private all your dependencies, you’ll realize that at least for a MOG Client, you won’t have much local information to be stored. Sure, there is local configuration – but besides login (and saved password) – pretty much everything else can (and SHOULD) reside on your Servers. And – this is a Good Thing™ from the installer/updater perspective.

As soon as we can say that any of our Clients can work with local data produced by any other Client – we can say that

All we need to do to install/upgrade Client – is to deliver a bunch of current Client files4to the player’s PC, and then to launch it.

Let’s write this observation down – as it will have very important implications for auto-updater.


4 And an occasional registry key

 

Signing Installer

For quite a few platforms (such as mobile ones) signing your Client and/or installer is a firm requirement; however – I’m advocating to sign your installer executable even in those environments where code signing is not strictly necessary (such as on desktops).

It is normally done with a special “Code Signing” certificate5issued by a 3rd-party authority such as VeriSign etc. (BTW, make sure to allocate at least a month for obtaining one) – and trusted by most of the browsers out there.

The point of signing your installer is two-fold:

  • To have browser complain less when player tries to launch your executable (that is, if the browser trust that-company-which-issued-your-Code-Signing-certificate).
    • This, in turn, reduces the player drop-out rate at this point.
  • Surprised hare:signature provides a chance for an unusually-aware player to notice the attackIf your executable was modified by somebody (for example, to include adware, or a virus, or a backdoor-targeting-your-game), signature provides a chance for an unusually-aware player to notice that “hey, it was normally signed – but it isn’t signed now, something fishy may be going on”.6

5 strictly speaking – signing is done with a private key which corresponds to this certificate
6 granted, less than 1% of your players will realize it, but at least we can be sure that we’ve done everything we could do to prevent such attacks.

 

Auto-Updates

After installing our Client – we’ll need to keep it up to date. And one thing which our MOG simply MUST have – is automated updates for our Client. If for any other app (or a single-player game) it might be ok to say “hey, whenever user/player wants – he can just go to the app store and update” – for MOG having an as-seamless-as-possible update is a MUST-have.

Even in the most extreme case of seamless updates being utterly unsupported by our platform – the very least we should do is the following:

  • Client MUST detect that Server doesn’t support protocol anymore, and
  • It MUST show something along the lines “Sorry, currently installed Client is no longer compatible with the Server. Please go to <such-and-such-place> and upgrade.”

The approach above reflects the absolute minimum (which is, as far as I know, is always possible); however – for quite a few platforms (at least desktop PCs), much more simpler-for-the-player approach of:

  • It is not Client which is launched, but an updater
  • Updater checks whether newer version is available, if yes – it updates Client
  • Updater launches the Client

can be used. If your platform allows it – this approach is strongly preferred to the previous one.

[[TODO: post-play update]]

 

Compatibility Issues

Protocol Compatibility

There are two Big Fat Reasons™ why automated updates are so important for MOGs:

  • We DO want to provide consistent experience for all the players.
    • Having different Clients for different players for a considerable time (a) is unfair, and (b) causes lots of pressure on support
  • Protocols between Client and Server DO Change.

And whenever we deploy new-Client-with-a-protocol-change, as a rule of thumb (though YMMV):

  • Hare pointing out:Usually, we DON’T want to stop the play at the point of deployment – so we probably DO want to have a Server which is compatible both with Client-version-X and Client-version-X-plus-one.Usually, we DON’T want to stop the play at the point of deployment – so we probably DO want to have a Server which is compatible both with Client-version-X and Client-version-X-plus-one.
    • More generally – we can speak about “compatibility window” where our current Server supports more than one Client.
      • Reasons to support more-than-two-Clients, are different, and range from the need-to-deploy-hotfixes – to long waiting lines in AppStores.
      • However, supporting all the previous Clients is unrealistic. Usually, the most we can realistically hope for – is supporting around a few weeks worth of Client releases.
      • Also – let’s note that those Clients which fall outside of the “compatibility window”, SHOULD be notified that they cannot play. This is necessary even if you’re using an automated updater (because whatever-you’re-doing-about-automated-updates – there will be players bypassing it; more on Stale Clients a bit below).

One all-important result of these observations is that compatibility of different Clients against the same Server needs to be tested (and this is going to be quite time consuming). Another practical observation is that IDL-with-compatibility-checking (along the lines discussed in Vol. I’s chapter on IDL and Vol. IV’s chapter on Marshalling and Encodings) can help quite a bit in eliminating at least the most glaring incompatibilities.

Compatibility of Local Data

On the other hand, while with MOGs protocol compatibility is a big issue – compatibility of local data is rarely a problem. As noted above – most of the time, we can see our Client as “just a bunch of files”, so the only thing the updater needs to do – is to make sure that current files correspond to a consistent-version-of-the-Client – and then to launch the Client.

This, in turn, implies that

The previous version of the Client is immaterial; what we want – is just to update our Client files “to the same files as currently-published-on-our-web-Server”.

In other words – for MOGs, we can (and IMNSHO SHOULD) be free from the mess of “upgrade paths”; the job of the updater is to bring a self-consistent version of the Client to the player’s device – and start Client there; that’s it. As long as our Client is just a bunch of files (with no local data depending on the version of the Client) – no scripts to upgrade data are ever necessary; and at the very least, we always can write our Client in a way to make sure this rule-of-thumb stands.

Stale Clients

Stale Clients on Client machines

Assertive hare:Pretty much whatever-we-do, there will be a certain percentage of players which are trying to run an obsolete version of the Client forever-and-everPretty much whatever-we-do, there will be a certain percentage of players which are trying to run an obsolete version of the Client forever-and-ever. From what I’ve seen (YMMV) – such players tend to come from three distinct categories:

  • Players who’re doing it unintentionally. If your auto-update is just launching a Client executable – well, there will be players who deleted the shortcut-to-update (created by your installer) from their desktop – and then will re-create the shortcut manually, pointing to Client instead of the upgrader.
    • This is easily alleviated by creating a stronger bond between updater and Client (for example, by passing a pre-defined randomly-looking parameter in Client’s command line).
  • Players who’re running mods/themes. If mods are allowed (which does happen even for some MOGs – more on it in Vol. IX’s chapter on themes/mods) – then your best bet to make such players use updater will be to block Clients-with-obsolete-version-from-playing. However – if you prefer to love your players – it is better to provide an API to support those-allowed-mods-and-themes;7 in this case – your modding players will get best of both worlds: updates and ability to do their modding (up to the extent allowed)
  • Players who’re reverse-engineering / cheating / etc. There is not much reason to care for them (and BTW, trying-to-use-an-ancient-version can serve as one of the indication of the modder – or a potential cheater).

7 One side benefit of this approach is that you’ll be able to draw a very clear line, saying “hey, whatever is done via this API – is good and allowed, and everything else is bad and prohibited”.

 

Stale Clients on download Servers

Last but not least about stale Clients:

Make sure that Client-which-is-downloadable-on-your-Web-Site, is consistent with the-Client-which-your-updater-refers-to.

There are only a few things which are more annoying for your potential player, then downloading-500M-of-Client – and then running an update which downloads-another-500M.

Sure, for some of the players (those who came around the time of the Client changing) – this effect of updating-right-after-downloading would be still present, but:

  • It will be only a small portion of your players.
  • If implementing differential/delta updates (see below) – the update will be much smaller too.

As a result, I strongly suggest to think of your download Server as a part of your deployment system – and to push Clients there about the same time as you’re pushing it to your Client-upgrade Servers. You’ll be surprised how many of the companies are ignoring this IMO-very-obvious advice.

 

Differential (Delta) Updates

if your environment supports them – or if you can implement them yourself – DO use delta updatesThe situation with differential/delta updates is simple: if your environment supports them – or if you can implement them yourself – DO use them. It will save your players quite a bit of completely-unnecessary-frustration.

One thing to note about differential updates: strictly speaking – there are two different types of them:

  • file-level differential updates (that is, whenever a file is different – the whole file gets uploaded and updated), and
  • intra-file differential updates. With them – you effectively ship a kind of “patches” for the file. These intra-file differential updates are well-known in theory, but are still quite rarely used in practice (though situation around them does change – with Google being one of the major pushers of this technology). We’ll discuss “how-to-implement-them-yourself” in detail in [[TODO]] section below.

Implementation

By now, we’ve (give or take) defined what we want from our updater. Now let’s see how we can implement it. In practice – it all depends on the environment where we’re operating (even more than usually, that is).

Different Environments

Different environments gamedevs usually need to deal with, can be roughly divided into the following:

  • AppStores and consoles. For AppStores – well, updates are pretty much defined by the platform. Whatever kind of updating/patching on a respective AppStore/Console is the most convenient for player in our scenarios – we should use.
  • For desktops (both Win and Mac) – most of the time, it is feasible to write our own updater – or use a 3rd-party one. We’ll discuss a DIY updater below; as for the 3rd-party ones – I usually hate both user experience and developer experience when using 3rd-party auto-updater (the only possible exception is NSIS – but I haven’t try its fairly new auto-update feature yet). That’s why I usually prefer to have my-own-updater for desktops – and mechanics of DIY installer is described in detail below.
  • In-Browser Clients. With respect to In-Browser Clients – we cannot do much with regards to updater; however, the following is perfectly possible:
    • Limit HTTP caching time for Client files (and if you’re using emscripten as discussed in Vol. II’s chapter on Client-Side Architecture – you’ll have only one self-consistent file, phew) to, say, 1 hour
    • Detect protocol version at Client level – and terminate Client if Server doesn’t support this Client (with a message like “Sorry, current version of the Client is incompatible with our current Server. Press Ctrl+R/F5/Ctrl+F5 to reload your Client”)

MOG-Oriented DIY Auto-Updater

First of all, let’s note that the principles and protocols which we’ll discuss below, will apply not only if you’re trying to write your own updater in C++ (or whatever-other-generic-programming-language). Normally, they will also apply if you’re using a 3rd-party scriptable installer (such as NSIS installer8) to write your own auto-updater in a whatever-programming-language-scriptable-installer-provides.

Now, let’s re-iterate some of the principles we mentioned above – and which are very typical for MOG updaters:

  • We’re updating only files (an occasional registry entry etc. is ok too)
  • There is no special migration process between different versions; in other words – as soon as we get our files right – we’re done, and nobody really cares how we arrived there.

With this in mind, we can now proceed to implementing our own DIY auto-updater.


8 IMO the best 3rd-party installer I’ve seen, but I haven’t seen all of them, and as always, YMMV

 

Version Numbers are Evil

At this point, I am going to say a thing which goes against all the practices in the installing world:

Using version numbers for updates is evil.

Assertive hare:I do have very significant reasons to dislike version numbersAdmittedly, this is a very bold statement (hey, everybody and their dog is using version numbers for upgrades) – but I do have very significant reasons to dislike version numbers. In particular:

  • If you try to manage version numbers for your files manually (especially in a dynamic environment such as MOGs) – they will inevitably get messy, period. There are too many of them, and you will get new files with the same version, and sometimes they will accidentally go back, etc. etc.
    • One drawback of version numbers is that they’re inherently fragile. Unless you’re assigning version numbers automatically (as discussed below) – you can easily forget to update the version number, or release two different versions with the same number – and cost of such mistakes can be very high (in the extreme case – leading to all your players having to re-download the whole Client, pretty much stalling your games for half a day, and losing who-knows-how-many players).
    • With deployments being frequent – thinking in terms of “major updates”/”minor updates” often becomes extremely confusing and error-prone. However – as discussed above, for MOG Clients we don’t really need all this stuff – we just need to get current bunch of files from our web server to our Client, that’s it; this makes all this complicated and error-prone analysis completely unnecessary too.
      • From a certain perspective – file version numbers are artifacts coming from the dark ages of waterfall development. Even Agile development started to dilute them, and in these days of Continuous Integration/Continuous Deployment – it becomes more and more difficult to answer the-absolute-prerequisite question of “what is the version number we’re speaking about?”.
    • Most of the time – we’re able to make sure that “just get up-to-date bunch of files and launch” rule stands for our MOG Client (see [[TODO]] section above). And as soon as it stands – we can start using much-better-then-version-numbers-identifiers-to-our-files – we can start using hashes.

[[TODO: time as file identifier is evil too! See below about caches]]

Hash-based Updates to the Rescue!

As a result of those deficiencies of version numbers – I strongly advocate for using hashes (such as SHA256), rather than version numbers, for updates. With “just get up-to-date bunch of files” logic – using hashes for updates is extremely straightforward:

  • Get new (and tested) Client version in a folder
  • Create a separate file with a list of files – and their respective hashes
    • to make everything bulletproof – let’s sign this list of files too (with a private key of our special “update CA”).
  • Publish new Client on the web server – including the signed list of files.
  • Auto-updater, when run, does the following:
    • Gets signed list of files via HTTP.
      • Checks its signature (using root certificate of CA, embedded into our Auto-Updater).
    • Calculates hashes for all the local Client files.
      • For each of the files on the list – checks whether the file has the same hash (and if not – downloads it from the web server and replaces).
      • NB: to speed this process up, some trickery (with caching hashes and checking if the file is the same using only stat() – or any of reasonable facsimiles) is possible on the updater side (and I’ve done it myself too).[[TODO: more detailed description of hash caching here]]
    • That’s it. As soon as all the hashes are the same as in the published file list – we have a consistent Client, and can launch it.

From what I’ve seen –

Such hash-based updaters tend to cause much less problems than traditional version-number-based ones.

In particular:

  • Hare thumb up:Hash-based updates do NOT require mundane-and-error-prone work to assign version numbersHash-based updates do NOT require mundane-and-error-prone work to assign version numbers. In a sense – hash is a perfect version ID of any file (that’s by definition of crypto hash), so using it makes the whole thing much simpler and provides much less opportunities to make a very costly mistake.
  • Hash-based updates are self-healing: even if an update was messed up for any reason – the next update (done under exactly the same rules) – will recover the Client into a correct state (and without a need to download-the-whole-thing too).
  • Hash-based updates guarantee that only-those-files-which-have-changed, are updated.

As a result – I strongly suggest, if you have any choice – to make your updater hash-based. If you still need to be convinced about hash-based updaters, I can tell that I’ve seen a hash-based updater working for over 10 years for a game with millions of weekly players – and it worked like a charm (with update failure rate being IIRC around 0.01% or so – this is by far the best result for home-PC-updates I’ve ever seen).

Auto-Assigning Version Numbers (based on Hashes)

We discussed implementing hash-based updates (the ones which I like a lot). On the other hand, what to do if for your environment, the update system is required to use version numbers (which happens all the time on cell phones)? Once again – I strongly suggest to avoid manual assigning of version numbers at all costs (as noted above – it is extremely error-prone). Instead – I suggest to:

  • At developer level – work along the lines of hash-based system above. It is still the one that is very obvious – and very robust too.
  • Then – you can have a script, which will take the history of updates – and will generate a new update, and with a new version too. Once again, detecting of what-has-changed, can and SHOULD be implemented based on hashes – they’re still very good for this purpose.
    • New version numbers MUST be assigned by the script itself (and NOT manually). Otherwise – you’re opening doors for all kinds of trouble (in particular, releasing two different versions with the same number is not just a problem – it can easily end up with a disaster, with re-install required for all your players).

Differential/Delta Hash-Based Updates

As noted above – differential updates are quite important for MOG Clients. However – how can we make a hash-based differential update (more specifically – intra-file deltas, as defined above)? I’ve seen the following schema working extremely well:

  • When generating that current-file-list for hash-based updates – we also generate a list of differences for certain files (with differences calculated against different prior versions).
    • How to generate updates – depends (I was using my own delta calculation algorithm,9but now you can use bsdiff or courgette – or actually even format-dependent differential compressors).
    • However, regardless of the way the update was calculated – you SHOULD include the following into current-file-list for each of such differential/delta updates:
      • Hash of the original file; hash of the final file.10
    • When auto-updater runs into such a file-with-one-or-more-differential-updates-listed in the current-file-list, it:
      • Checks hash of the current version; if the hash matches “original file” field of any of the available differential updates – auto-updater gets the differential update, and tries to apply it (and if after applying it, the hash of the resulting file matches the “final file” field – well, it means that we’ve got exactly-the-file-we-want, plain and simple).
      • If there is no matching differential update in the list – auto-updater defaults to a full download of this file (this option is always available regardless of the differential updates, and of current version available on the Client-Side).

That’s it. We can have our hash-based updates – and to have them intra-file differential too.


9 it was in the days when none of the modern diff algos existed
10 strictly speaking, the “final” hash is redundant – but it never hurts to double-check

 

Implementation: Transactional Updates

Hare with an idea:It is very important to keep updater transaction-oriented.One implementation detail which is very important when implementing updates – is trying to keep all the actions transaction-oriented. I don’t mean to use transactional file system facilities provided by OS11 – but rather just splitting your update into several stages, while trying to provide all-or-nothing handling for each of the stages:

  • Stage 1. Checking for the update.
    • If this stage fails – there is nothing to roll-forward, we can just terminate safely (phew)
  • Stage 2. If update is necessary – download all the needed Client files.
    • This stage is risky (i.e. all kinds of trouble can happen while we’re doing it, from loss of connectivity to running out of disk space) – and I strongly suggest to download all the Client files to a separate folder (instead of replacing them in place).
    • If this stage was downloading to a separate folder – then rollback is trivial (just remove all the contents from that separate folder, that’s it); even if rollback itself fails – it is not a problem from consistency point of view.
  • Stage 3. After all the files are downloaded – make the update itself.
    • I strongly suggest moving files-before-the-update into a “backup” folder
    • Then, if update fails for any reason – we can still try to perform a roll-back using “backup” folder.
      • NB: it is not a strict rollback (i.e. if roll-back itself fails – we can end up in trouble). As a result – this is a place where OS-provided transactions over file system may come handy (though I have to admit that I never used them; OTOH – that 0.01% failure rate I mentioned, was achieved without OS-level transactions, just with transaction-like handling described here)

BTW, while we’re speaking of transactions: they are complemented very well by allowing the player to play after the update failure. As noted above – most of the time a-tiny-bit-older-Client will still work, so if the update failure is a temporary hiccup (and we managed to restore previously-working Client because of kinda-transactional logic above) – it is only fair to allow our player to play. From what I’ve seen – kinda-transactions (as described above) DO allow to keep Client in a consistent state (the only exception being a failure of rollback at Stage 3, but these are extremely rare).


11 if you try using them for installing/updating – please let me know about your experiences

 

Windows: Vista-and-later UAC Craziness

UAC User Account Control (UAC) is a technology and security infrastructure... It aims to improve the security of Microsoft Windows by limiting application software to standard user privileges until an administrator authorizes an increase or elevation— Wikipedia —Vista-and-later Windows is quite a special beast when it comes to updatable software such as MOG Clients. In particular – to install/update an executable, we need to jump through quite a few hoops, one of them being UAC (a.k.a. “User Account Control”). Overall, I know of two ways of dealing with Vista+ UAC-related peculiarities:

  • Install your Client into AppData\Roaming\
    • This is probably the simplest option. However, if you ever run into permission issues with it – make sure to read about UAC below (permissions are changing all the time, so what-worked-just-a-month-ago – may stop working tomorrow)
    • NB: you may still want to have your updater installed in Program Files (it is more security-sensitive, so it may make sense to have it better protected) – but in this case, you’ll still need to jump through UAC-related hoops described below, when performing update-of-updater.
  • Install your Client into Program Files.
    • Then, under Vista or later, one of two following things will happen:
      • Either your updater will cause a dreaded UAC dialog each time it is launched (and even if there is no Client update) – making your players crazy (this will happen if your updater has a manifest specifying “requireAdministrator” priviliges – or in some cases without any manifest)
      • Or your updater won’t ask for an UAC – but then it won’t be able to perform an update when it is necessary (this will happen if your updater has a manifest specifying “asInvoker” priviliges – or in some cases without any manifest)
      • Neither of these options is particularly good, but there is a way to bypass it:
        • Have your updater executable to specify “asInvoker” in the manifest – which means no UAC dialog, but no ability to update the Client.
        • When updater finds that the update is necessary – instead of updating, it asks player whether she wants to update – and then launches a “stub” executable and terminates; this “stub” has a manifest specifying “requireAdministrator” privileges – and simply re-launches updater once again with a flag specifying that it is ok to update.
          • As “stub” has “requireAdministrator” set – launching it will cause UAC dialog, but this is exactly what Windows wants us to do, and it is a normal behavior from player’s perspective too (in particular, it happens only when update is necessary).
        • After being re-launched, updater (having “asInvoker” privileges) inherits permissions from the “stub” – and can proceed with updating the Client.
        • We’ve got our Client updated – and without calling UAC dialog too much too.
      • With all the things happening and changing out there – there can be other ways of skinning this cat; what’s most important though – is to make sure to test any of them on computers running different versions of Windows; this MUST include tests over a Cartesian product of default configurations of all of the following:
        • All versions which you want to support (7, 8, 10, whatever-else)
        • All editions of these versions (such as Home/Pro/Ultimate editions)
        • Both Admin and non-admin accounts(!)
        • Otherwise – you’re risking to find yourself in a really hot water (these things change greatly from edition-to-edition and from admin-to-non-admin).

 

Update-of-Updater

Normally, when we’re doing update – it goes as follows:

  • Updater starts
  • Checks for updates
    • Updates the Client if necessary
  • Starts Client

However, when we need to update the updater itself – the whole thing becomes much more complicated (see also illustration at the beginning of this Chapter):

  • Updater starts
  • Checks for updates
    • Realizes that it itself needs to be updated
    • Downloads and prepares the update in a separate temp folder (NB: at least on Windows – it needs to be separate from the updater, as we cannot update running executable, but see also above re. kinda-transactions)
    • Launches special “stub” – and terminates
      • “stub” waits for the updater process to complete (make sure to use process handles, and NOT timeouts here)
      • “stub” copies files-prepared-by-updater (from temp folder), overwriting updater
      • “stub” launches updater and terminates
    • phew, updater can proceed with updating Client
  • … updater proceeds along the lines above

Overall, update-of-updater tends to cause significantly more problems than simple update-of-the-Client. Fortunately enough – updates-of-updater can be made extremely rare (for a large game which was updated once per 2-3 weeks, I know of only 4 or 5 updates-of-updater during 10 years of operation).

[[TODO: update via renaming on Windows]]

[[TODO: updates-while-playing: from theme downloads to streaming updates; prioritization with main game traffic!]]

[[TODO: updates while playing; prioritization compared to game traffic; integration with network library]]

[[TODO: protocol updates]]

[[To Be Continued…

Tired hare:This concludes beta Chapter 18 from the upcoming book “Development and Deployment of Multiplayer Online Games (from social games to MMOFPS, with social games in between)”. Stay tuned!]]

Don't like this post? Comment↯ below. You do?! Please share: ...on LinkedIn...on Reddit...on Twitter...on Facebook

Acknowledgement

Cartoons by Sergey GordeevIRL from Gordeev Animation Graphics, Prague.

Join our mailing list:

Comments

  1. Peter Feichtinger says

    Since this is a beta test I feel compelled to give a little bit of (hopefully constructive) criticism.
    There is one major thing which always annoys me when I read one of your book beta test articles: you should really stop connecting-words-with-dashes when it’s not necessary, it makes those passages really hard to read.
    What is also somewhat annoying, but not as hard to read, is when you unnecessarily capitalize words, something like Good Thing™ is OK IMO, but why do you capitalize Client for example?
    And since this is going to be a book, please drop abbreviations like IMNSHO, just write “in my opinion” (you’re allowed a not-humble opinion in your own book I’d say ;-).

    • "No Bugs" Hare says

      Thanks for the criticism :-). We discussed exactly these issues with professional editor – and she has already removed quite a few of them (in “rc1” of Chapters I-II, Chapter III in progress); however – in quite a few cases they still stay. In particular, both stuff-which-is-connected-with-dashes and Capitalized Nouns (as if they’re proper nouns) are intended to make it easier to parse longer sentences (otherwise it is quite difficult to locate what is the noun within). Just one example: while single capitalized Server may indeed be somewhat confusing – sentences including “Game Server” are generally easier to parse than wall-of-text-ones (without capitalization); also capitalization makes it clear that it is the same kinda-proper-noun term which is used elsewhere in the book.

      If nothing else – think of it as of “author’s preference” ;-).

  2. says

    Just one note about UX of updating – if at all possible (say the new and old versions *can* use the current protocol) have the download/update happen in the background and *after* the player has finished playing.

    The benefit for your player? They don’t need to wait for a (possibly) lengthy update when all they want to do is play the game. If you can defer updating to when they finish playing, they get the best of both worlds.

    You will need to have a flag of some sort of compatibility flag to tell your updater when it can go ahead with such a scheme and when it cannot and needs to update there an then.

    • "No Bugs" Hare says

      Well, I done it myself for a what-became-a-N’000’000-player game – IIRC, it took like 2 weeks total (that’s without diff updates, and it was pre-Vista, so UAC problem didn’t exist yet). Not too much if you ask me. Sure, it would be MUCH better if somebody else does it for us – but none of the-systems-I-know comes even close. Do you have any specific suggestions for a patcher which provides like 0.01% failure rate in the wild?

      P.S. took a look at your link – yes, it all looks very familiar (been there, done that) :-). Just one note as for file times – they’re not TOO bad as long as we can limit their use to the LOCAL computer. It _is_ possible – and most of the time-related problems go away then (in particular, no conversions are ever necessary, we can just store whatever-local-representation of the file time, that’s it). OTOH, using time to _identify_ the file (i.e. relying on time-within-manifest to decide whether to do the update) – is indeed very VERY error-prone (but it can and SHOULD be avoided, using hashes to identify files instead). I’ll add more detailed description of these techniques, THANKS!

      P.P.S. Bottom line: yes, I agree that writing a good patcher IS not too easy. No, I don’t know of any readily-available and decent game patcher.

    • "No Bugs" Hare says

      Indeed, fixed now, thanks! Just want to tell that first Chapters of the book are currently being edited by a professional editor – so these things should be fixed in the book anyway. In other words: don’t be afraid to buy the book – English in the book is going to be MUCH better than here :-).

  3. says

    Another awesome piece! Something you might want to give a few paragraphs to, many updaters, such as StarCraft, allow you to start playing before the update is finished. Might want to explore that.

  4. Dahrkael says

    About updating the updater, if I’m not mistaken, Windows allows you to rename a running executable, simplifying all the process as you can skip the stub program. it would go like this:
    – updater finds its outdated
    – updater downloads new updater with a temp name
    – updater rename itself to .bak or something
    – updater renames new version with the correct name
    – updater relaunches itself

    That should work (in Windows at least)

    • "No Bugs" Hare says

      Yeah – I even knew about this trick at some point, but forgot about it completely. Will add it, THANKS!

Leave a Reply

Your email address will not be published. Required fields are marked *