Bot Fighting 104. Known Bots and Environment Scans.

 
Author:  Follow: TwitterFacebook
Job Title:Sarcastic Architect
Hobbies:Thinking Aloud, Arguing with Managers, Annoying HRs,
Calling a Spade a Spade, Keeping Tongue in Cheek
 
 
MOGdev facing a choice of 'Fair Games' vs 'Privacy'
#DDMoG, Vol. VIII

[[This is Chapter 29(l) from “beta” Volume VIII of the upcoming book "Development&Deployment of Multiplayer Online Games", which is currently being beta-tested. Beta-testing is intended to improve the quality of the book, and provides free e-copy of the "release" book to those who help with improving; for further details see "Book Beta Testing". All the content published during Beta Testing, is subject to change before the book is published.

To navigate through the "1st beta" of the book, you may want to use Development&Deployment of MOG: Table of Contents.]]

One technique which is used by quite a few gamedev teams out there, aims to identify known bots, and to get respective players banned. Let’s take a look at this class of techniques in more detail.

Motivation: Publicly Available Bots are The Biggest Threat

One reason

why it is so important to deal with known publicly-available bots, is that they represent the biggest threat to the game ecosystem.

Indeed, one single bot writer writing a single bot will get his unfair advantage and will spoil game experience of a few other players; however, if the same bot writer publishes the very same bot (or starts selling it for a few bucks) – it will be used by thousands of wannabe-cheaters, increasing the negative effects on the honest players by orders of magnitude. In extreme cases, it is these publicly-available (or commercially-available) bots which will create a feeling that “cheating is rampant in such-and-such game”, which, in turn, can easily lead to the players starting to leave your game <very-sad-face />.

Hare pointing out:That’s why most of multi-million-dollar games have a special team dealing specifically with public and semi-private botsThat’s why most of multi-million-dollar games have a special team dealing specifically with public and semi-private bots (more on the latter in [[TODO]] section below).

Arms Race

Fortunately enough, as soon as we’re dealing with a known bot – we’re playing on our home turf. For a detailed discussion of home-vs-road-games with regards to bots – see Vol. I’s chapter on Cheating, but very briefly – as soon as we have a known bot, the tables are turned, and basically we can use the-same-reverse-engineering-techniques which were used against our game Client, to dissect the bot and to disable its attack vectors.

On the other hand, as soon as we release our updated Client which disables bot’s attack vectors – the fight gets back to the attacker’s home turf, so they can dissect our defences, and release a new version of their bot.

This represents a typical arms race, with each of the parties leap-frogging ahead of the other one for an inherently-limited time. This is the best we can hope for – but still, this is MUCH better than leaving publicly-available bots unchecked.

Environment Scans

One popular (and surprisingly-still-working) way of detecting publicly available bots is based on scanning the environment-where-our-Client-is-running, looking for the artifacts typical for known publicly-available-bots. There are apparently lots of ways how the bot can reveal itself to such scans; this includes very naïve (but still working) things such as:

  • Looking for window with a specific title (yes, there are bots out there for which it still does work)
  • Looking for a specific registry key
  • Looking for a file with a specific name
  • etc. etc.

Signature Scanning

Going a bit further than naïve environment scanning mentioned above – it is possible to scan RAM-or-modules of the processes-running-on-the-same-box, looking for well-known signatures (which is BTW more-or-less what antiviruses do). Technically, this consists of two parts:

  • Choosing which API to use to get access to the other process. On Windows, it is at least possible to use ReadProcessMemory() – or to use Toolhelp32 (either to heapwalk the snapshot, or to enumerate modules of the running processes, and to scan .exe/.dll files of those modules for the signatures). As for what-to-choose-for-this-purpose – doesn’t matter too much except for potential AV false positive, more on it in [[TODO]] section below).
  • Rabin–Karp algorithm Rabin–Karp algorithm or Karp–Rabin algorithm is a string searching algorithm... that uses hashing to find any one of a set of pattern strings in a text.— Wikipedia —The second part of the puzzle is choosing an algorithm which scans large volumes of RAM-or-executables efficiently. This is a well-known field of academic research (usually referred to as “multi-pattern matching”), and I don’t want to go too deep into it, just mentioning that from what I know, for our not-really-AV-purposes, Rabin-Karp algorithm tends to work good enough.

Legality Issues

IANAL IANAL is a Usenet and chat abbreviation (initialism) for 'I am not a lawyer'. It may be used by non-lawyers who are seeking to avoid accusations of unauthorized practice of law and are not making any recommendation to the particular addressee of their remarks.— Wikipedia —There are claims out there that environment scanning (especially memory scanning) is illegal (see, for example, [Honorbuddy]). I have to emphasize that IANAL,

so make sure to ask your legal team before implementing any kind of scanning beyond your own process;

however, from whatever-accidental-knowledge-I-have, the legality of scans is rarely a problem in the real world: even if such scanning requires explicit consent from the player, such consent is usually requested in ToC-which-player-agrees-with, and as long as information-obtained-from-scanning is used only for the relevant purposes (such as ensuring game integrity) – it tends to be ok (however, once again – do NOT trust me, and make sure to ask your legal team about it).

The most widely-known case for such scanning is Blizzard Warden, which was claimed by EFF to be a spyware1  – but caused no real trouble to Blizzard business whatsoever. Moreover, most of the non-cheating players did support Blizzard on it – see, for example, [Ward] (and there were reports that the-guy-who-complained-the-loudest, was a bot writer himself). BTW, I am not trying to judge whether doing scans is “right” or “wrong” in a theoretical sense2 – but from my purely-practical perspective I can tell that by now, it is a standard accepted practice for MOGs – and is the one which, if implemented properly, tends to cause much less trouble than if not implemented at all.

Inquisitive hare:it might help to ensure that whatever-information-you-collect, DOES NOT leave the player’s box (except that tiny-bit-of-absolutely-relevant-information that “this player is running such-and-such-bot”)Also, it might help to ensure that whatever-information-you-collect, DOES NOT leave the player’s box (except that tiny-bit-of-absolutely-relevant-information that “this player is running such-and-such-bot”); BTW, enforcing such limitation is a really good idea even regardless of the potential legal issues (we do NOT want to get any sensitive information even accidentally; as we’re not Google – we might not be able to get away with a StreetView-like scandal). While it can be difficult to prove it to the community – it is still a Really Good Thing™ to ensure. Oh, BTW, as the discussion in [SchneierOnSecurity] indicates – there is a significant perceived concern about even sending hashes of the scanned data back to the Server-Side (opposed to sending just-result-of-the-comparison).


1 though even EFF, claiming that Warden is “spyware”, has tacitly admitted that it is legally ok [EFF]
2 though, as of now – in a drastic change from 2005 where all such discussions have occurred – scanning has became a standard practice for MOGs (=“it is to be expected from any MOG”), which changes landscape a lot (in particular, about player expectations) – which we as MOGdevs should thank Blizzard for.

 

On Permissions and Services/Drivers

One thing which can easily limit our ability to scan things – is OS-level permissions; and even if currently your scanning does work from a non-admin account, as the time goes – I am sure that restrictions for scan-like operations are going to become more and more severe. One surefire way to deal with it is to install your own service (or even driver); however,

Unless you’re already Really Big™ and Really Suffering from Cheaters™ – I would advise not to deploy driver-based detection methods.

The reason for it is simple – at early stages of your game, asking to install a service (or even worse – driver3) will look suspicious to the player, reducing their desire to play (which can lead to catastrophic results early in the game life cycle); moreover – at early stages of your game you’re unlikely to need such drastic methods of dealing with cheats. It is when your game becomes a household word, you (a) may have a legitimate reason to use driver-based scanning, and (b) may be able to get away with it.


3 For some reason, perception-wise you might be able to get away with a service, in spite of it being about-as-bad-as-driver both security- and privacy-wise.

 

On AV False Positives

One danger with scanning-like techniques for our Client app is to be misidentified by some antivirus product as malware. The only way to learn about it – is to have a procedure of testing your Clients on boxes-with-different-AVs (and you should certainly perform such testing). As for ways to avoid such false positives – all the evidence in this field is very anecdotal, but there are a few potential hints (with absolutely no warranties of any kind):

  • Looking-for-something-specific (such as FindWindow()) is generally safer than enumerating (such as EnumWindows()).
  • Reading executables and DLLs is generally safer than scanning memory directly (so it might be better to use Toolhelp32 to get a list of modules for running processes, and to scan EXEs-and-DLLs-from-the-list for known signatures).

On VMs

If the bot is running in a VM host, while our game runs in a guest VM – all the scanning is hopeless to start with. Moreover, for certain game types, VM-based bots have become a standard more than a decade ago.

Femida hare:Some games decide to prohibit running within VM outright. Personally – I doubt it is wise, preferring to report VM-being-used to the Server-Side, where a “red flag” is raised, which actually means such-players-using-VMs to be scrutinized more than the othersThe only thing which we could achieve in such case – is to detect the fact that our Client is  running within VM (more on it in [[TODO]]). Some games (such as LoL) decide to prohibit running within VM outright. Personally – I doubt it is wise (both because there are legitimate reasons for using VMs, and because giving an explicit feedback about protection is rarely a good thing), preferring to report VM-being-used to the Server-Side, where a “red flag” is raised, which actually means such-players-using-VMs to be scrutinized more than the others (and additional scrutiny may include using captchas, server-side stats analysis, comparing-game-patterns-when-playing-under-VM-and-without-it, human-reviews-when-such-players-reach-certain-level, and whatever-else-you-can-think-of).

On Obfuscation

One further thing which is going to get in a way of signature-based scanning, is publicly available obfuscation libraries such as ithare::obf4. While (as discussed in [[TODO]] section), such obfuscation techniques can be an invaluable tool to protect our Client executables from reverse engineering, it would be silly to assume that bot writers won’t use them too <sad-face />. In particular – nothing would prevent serious commercial cheating team from compiling randomly-obfuscated executable for each and every customer, which makes signature-based detection pretty much hopeless <sad-face />.

On the other hand, given the existence of VM-based bots, it is not expected to change the landscape too much; in addition – on the balance of pros and cons of obfuscation techniques, they will push the battle into more-professional field (with higher entrance barrier), where professional gamedev/protection teams should have an advantage in the long run.


4 WIP

 

Reverse Engineering Arms Race

As soon as the bot is smart enough to get out of sight during our environmental scan – we’re back to the good ol’ arms race between cheaters and bot fighters. Unfortunately, this reverse engineering arms race cannot be generalized (and moreover, whatever-is-written-about-it-in-public, loses its value almost immediately <sigh />5). Just a few very basic examples which fly surprisingly often (and which are already known well-enough so revealing them won’t be anything new for bot writers), range from simplistic stuff such as “hey, this guy clicks with a distribution of mouse clicks being too-evenly-distributed-around-certain-point and doesn’t change when we move this button by 5px (which can happen because we got a complaint about him cheating)”, and detecting grinding-bots playing 24/7 (you will be surprised of the numbers of bots doing this kind of stuff) all the way to things such as “hey, this bot tends to detect this kind of corner-case and reacts in exactly this manner – which is better-than-human-being-can-possibly-do, or is statistically impossible to have exactly the same reaction all the time”.

However, what is perfectly clear, is that all such fights are 100% down to the question

Who is more persistent – attacker or defender?

Most of the time, it is us (=”gamedevs”) who have more to lose in such a fight, so we tend to be more persistent. On the other hand, IF we slacked long enough to allow forming of a thriving-hacking-community-which-lives-from-selling-cheats – we have already increased the reasons for the cheaters to persist by many orders of magnitude (if selling-cheats is how-the-guy-pays-his-bills – he will be MUCH more persistent with breaking-our-additional-defenses than somebody who has just got his first $10 out of it). The point here is:

DON’T allow cheaters to make money out of their cheating. Otherwise it will be MUCH more difficult to get rid of them later.

5 one exception is randomized obfuscation techniques discussed in [[TODO]] section

 

Mechanics of Anti-Public-Cheats Team. Procuring Cheats. Semi-Private Cheats

As noted above, quite a few serious games have their anti-publicly-available-cheating-teams. Mechanics of these teams are rather similar across the board and tend to include most of the items from the following list:

  • Identifying a publicly available/commercial cheat. Usually done by monitoring relevant forums.
  • Procuring it. Usually trivial, just make sure not to buy from your headquarter’s IP address <wink /> (though anonymized IPs are usually fine).
  • Finding its signature and incorporating it into the Client’s signature detection.
  • If necessary – reverse engineering it to find out its attack vectors and developing specific antidote.

Surprised hare:the-more-widely-spread-the-cheat-is – the more harm it causes for the game – but at the same time the easier it is to obtain <phew />One special case of such cheats – is semi-public/semi-private cheats. As cheaters have realized that for public/commercial cheats, the advantage is on gamedev’s side – they started to cover their tracks, offering the cheat only in private forums. This, in turn, means that identification/procurement subteam has to infiltrate such forums (which can be tricky and can require lots of time); however – let’s keep in mind that the-more-widely-spread-the-cheat-is – the more harm it causes for the game – but at the same time the easier it is to obtain <phew />.

[[To Be Continued…

Tired hare:This concludes beta Chapter 29(l) from the upcoming book “Development and Deployment of Multiplayer Online Games (from social games to MMOFPS, with social games in between)”.

Stay tuned for Chapter 29(m), where we’ll start discussion on dealing with ‘Ultimate Bots’]]

Don't like this post? Comment↯ below. You do?! Please share: ...on LinkedIn...on Reddit...on Twitter...on Facebook

Acknowledgement

Cartoons by Sergey GordeevIRL from Gordeev Animation Graphics, Prague.

Join our mailing list:

Comments

  1. "No Bugs" Hare says

    > which ranks first in a lot if not most MMO rankings

    Being at #1 with mere 10M of _total_ players (which is widely understood as “registered accounts”) accumulated over 7+ years – gimme a break (however – yes, they’re doing well-enough financially to buy those ratings :-)). From what I heard, they have up to 100K simultaneous players – which according to yours truly 😉 is barely at the level for cheating to be guaranteed-to-be-present, but is still not guaranteed for cheating to become rampant. Compare to Dota/LoL/CS all of which have 500K-1M simultaneous players.

    > doesn’t have a single anticheating measure.

    They did claim to catch some of the cheaters as early as 2013 ( https://www.engadget.com/2013/10/01/over-6100-accounts-banned-for-cheating-in-final-fantasy-xiv/ ) – so they should have _some_ anti-cheating measures (I won’t believe that they merely observed the game play, and accidentally run into those 6100 cheating accounts).

    > Which makes you wonder if the actual game design is a way better anti cheating system than friendly rookits

    They already have a pretty big problem with cheating at their hands (look at their forums – for example, at http://forum.square-enix.com/ffxiv/threads/326434-Cheaters-in-the-Gold-Saucer-What-Can-We-Do , and quite a few of others) – and it will only become worse if they grow (which they apparently don’t at the moment). It is just that FF devteam chooses to turn a relatively-blind-eye towards this problem (which apparently didn’t hit them hard enough _yet_). I’ve seen quite a few companies with this kind of attitude – and it never worked good for them (they usually do survive, but emerging competitor-without-cheating starts to eat market share sooner rather than later).

    As for their game design – of course, the less competitive your game is, the less incentive cheaters have, but if their numbers _really_ grow – cheaters will start to hurt them _badly_ regardless of game design/mechanics (WoW is just one example of a game which was hurt badly enough to start dealing with cheaters). It is the very same painful path lots of companies already went through (on GDC16 there was even a talk by DayZ guys, who’re spending very good portion of their time dealing with cheaters; ok, for FF they grew about 10x larger before attracting cheaters badly enough, but around 17 cheaters already came in large numbers – and making money out of FF population too).

    In general, unless we’re speaking of a perfectly-cooperative game (something like Minecraft-w/o-PvP) – _and_ the one without _any_ grinding – I don’t see a way to design the game without potential to cheat. BTW , it goes pretty much the same way that it is not possible to run a large-enough society without police (while for a village with 100 people, it might be viable, for 1M-city there is no way and no chance). Think of anti-cheating as of police – which is a kind of necessary evil.

    So, I don’t think that there is a realistic way to get without _some_ anti-cheating measures for a 1M-simultaneous-players-game (hey, even FF itself did ban cheaters as early as in 2013). _Which_ measures to use, and whether there is a genuine need to run “friendly rootkits” – is less obvious. Personally, I prefer to stay without external scanning (relying on obfuscation, integrity scanning of my-own-process, server-side stats, and so on) – but from what I have seen so far, for highly-competitive environments (in particular, for eSports including those-huge-games-listed-above) – I won’t jump too high if they tell me that there is no way around policeman-in-each-player’s-PC 🙁 .

    BTW, as I mentioned, more often than not, it _is_ possible to detect known bot without resorting to environment scans (it is more complicated, but it _might_ fly). EDIT: I added a few sentences about it – will try to extend it in the paper book.

    Hope it makes my view of it more clear 🙂 (I am NOT an avid supporter of the scans – as long as they can be avoided, but they often can’t 🙁 ).

    • Dahrkael says

      > From what I heard, they have up to 100K simultaneous players
      They have 66 servers which I heard support up to 10k players each before starting queueing people, so maybe a bit more ( I’ve seen queues of 6000 people during expansion launch and peak hours still have queues of ~100).
      I’m talking about MMORPGs of course, which cannot compare in numbers to other genres on these years. (still 100k x 12€ each per month I would sign!)

      > so they should have _some_ anti-cheating measures
      they regularly ban a lot of “illegal activities and RTM accounts”, but theres no client or server side checks whatsoever, is more report based. The thing is the hardcore community relay on memory and network scanners hooked up into the game to get their performance logs, so SquareEnix won’t add any measure to avoid those.

      Effectively theres a rampant botting problem going on, which effectively mostly affects PvP matches (a totally anecdotical mode). The point here is the game economy is so superfluous they don’t really disrupt other players, as seen in the thread you linked the player claims its unfair, but thats it. Again not bad after 7 years as opposed to other MMORPGs (these guys explain it very well https://www.youtube.com/watch?v=hABj_mrP-no )

      On a side note, FF team suffers another issue, which is RMT guys DDoSing them when they close the high population servers, effectively cutting them from creating new bot characters to sell currency; which translates in a serious service disruption for everyone. Looking forward to the DDoS section.

      • "No Bugs" Hare says

        > which I heard support up to 10k players

        FWIW, 10k players/box would be _highly_ unusual (“industry standard” for simulation games is more like 100 players/core, 1k players/box). I myself did 10k players/box for front-end servers – and it was quite tough even without simulation there (for example, I don’t think that I would be able to reach these numbers w/o (Re)Actors)… And if it is more-industry-standard 1k-2k players/server for FF – then 66 servers will get them only 100k simultaneous players overall, which is consistent with the other numbers out there (NB: “simultaneous players” != “subscribers”, it can be more or it can be less).

        > The thing is the hardcore community relay on memory and network scanners hooked up into the game to get their performance logs, so SquareEnix won’t add any measure to avoid those.

        Statistics is a known issue in the industry, and is solved by providing an “official” API to get information (allowing 3rd-party vendor to register a DLL, and get whatever-gamedevs-deemed-acceptable-to-get-without-affecting-others-too-much); also, allowing to get _some_ stats (and even _some_ automation) doesn’t conflict with banning specific offending programs. I know at least one huge company which has a long list of banned and allowed helper programs (which is updated on regular basis, and the line between the two is very roughly drawn around the concept of unfair advantage).

        > PvP matches (a totally anecdotical mode)…. The point here is the game economy is so superfluous they don’t really disrupt other players

        Ok, then they _could_ be the case of nearly-perfectly-non-competitive Minecraft-like game; however – (a) they already got complaints about cheating (and there will be more of them if they grow), and (b) saying that all the games have to be specifically designed to be non-competitive – is IMNSHO a bit too much :-). ANY kind of competition (and it includes ANY kind of leaderboard) – and you’re out (even games such as Township or SimCity BuildIt are competitive and have to fight cheaters); I daresay that all the market for non-competitive games (such as Minecraft) is like 0.1-1% of the whole MOG market (see above re Township and SimCity; also note that leaderboards is a very common way of monetization); or from a different angle – those who don’t like competition, tend to play single-player games, not MOGs :-). Bottom line: niches like this one do exist – but they’re not common at any rate.

        > Looking forward to the DDoS section.

        Spoiler for state-of-the-art anti-DDoS :-): (a) get your own AS (=”Autonomous System”) number, (b) get your own BGP router(s) – and (c) get a contract with BGP-based anti-DDoS provider. In case of big attack – just reconfigure your BGP so that your IPs are routed to that anti-DDoS provider (which can handle terabits/second, good luck DDoS-ing them) – and within 120 seconds ~99% of your players will migrate to anti-DDoS-provider boxes; anti-DDoS provider will “clean” the traffic (to help them do it, you’ll need to cooperate with them), and they will tunnel your “cleaned” traffic back to you. The rest of DDoS protection is all about “how to make sure that you can handle smaller attacks without resorting to this kind of stuff, which works, but is expensive and somewhat-disrupting”.

  2. Dahrkael says

    Very interesting conversation for sure!

    > FWIW, 10k players/box would be _highly_ unusual (“industry standard” for simulation games is more like 100 players/core, 1k players/box)
    Directly from the mouth of the director&producer: http://forum.square-enix.com/ffxiv/threads/73680-Further-Details-on-Access-Restrictions
    The game works with area servers (maps are instanced), so the 66 are indeed not monolithic. With a maximum capacity of 5k on launch, it would be feasible to assume they can support up to 10k now, as theres easily twice the maps with the 2 expansions. (and they reached 300k+ on launch!)

    • "No Bugs" Hare says

      In your link, they wrote “each FINAL FANTASY XIV: A Realm Reborn World being made up of several of these machines.” Which I translate into “while ‘World’ can handle 5’000 players, it consists of several PC boxes” – then everything starts to make sense :-). An alternative would be if they had “100% dumb” non-authoritative servers (with all the logic going via authotitative clients, and servers merely bouncing information – in this case, it is possible to handle 10K players/2S-2U box) – but I don’t want to think _that_bad_ about them ;-).

      > and they reached 300k+ on launch!

      They’re writing about 218K (and 300K was “if-not-for-timezones”, which I translate into “DAU=300K on launch”, but which is not directly related to the simultaneous players number). BTW: (a) their 218K is kinda-in-between “your” 300K and “my” 100K ;-), and (b) by all available accounts they’re declining ever since (at the very least – otherwise you could be sure them to brag about it)…

Leave a Reply

Your email address will not be published. Required fields are marked *