Back End Tools for MOGs

 
Author:  Follow: TwitterFacebook
Job Title:Sarcastic Architect
Hobbies:Thinking Aloud, Arguing with Managers, Annoying HRs,
Calling a Spade a Spade, Keeping Tongue in Cheek
 
 
Front-End vs Back-End
#DDMoG, Vol. VII

[[This is Chapter 26(a) from “beta” Volume VII of the upcoming book "Development&Deployment of Multiplayer Online Games", which is currently being beta-tested. Beta-testing is intended to improve the quality of the book, and provides free e-copy of the "release" book to those who help with improving; for further details see "Book Beta Testing". All the content published during Beta Testing, is subject to change before the book is published.

To navigate through the "1st beta" of the book, you may want to use Development&Deployment of MOG: Table of Contents.]]

One all-important thing which is very often neglected in MOGs – is Back End Tools. On the other hand, Back End Tools is one thing which is extremely important for your game. After all – it is Back End which controls (and allows to see <smile />) all the money flowing into your game; even more importantly – it is the way your CSRs can access the system, and reply to support requests (ranging from “Help! I lost my password” to “I paid you my $1.99, but got nothing”).

Within this book, we will understand Back End as a set of tools, which provides the following capabilities:

  • Control. Whatever your game is – you’ll probably want to change its operational parameters (or more generally – control your game operation at business level). This includes such things as running scheduled tournaments, changing gameplay parameters, announcing server restart to all the relevant players, etc. etc. etc.
  • Monitoring. It is all-important that your support team knows the current health of the system at any given point in time (saying to the player complaining that the game is slow, to reboot his computer – while it is your servers which are lagging – is usually not the best idea).
  • Reporting. In addition to controlling your game – you will need to get all the data your non-programmers need, from the databases.1 This includes all kinds of data, ranging from operational data such as “Last sessions played by player X”, to analytical data such as “Give me per-country stats of all the players filtered according to such-and-such criteria”.
  • Help Desk. Most of the time, you will find that your Help Desk is just a fancy name for an elaborated-and-highly-customized Email Queue. While in theory, in addition to e-mails you can have tickets, live chat, and even phone call center – in practice, pushing your players into explicit tickets is unrealistic, and live chat / phone is just too expensive to run (with a few narrow exceptions discussed below).

While the spectrum of these tools is rather wide – they have one significant thing in common:

The vast majority of Back-End Tools are intended for NON-programmers and NON-admins.

Surprised hare:primary users of your Back-End Tools are your CSRs, your Security Team, your Marketing-&-Monetization Team, and last the Almighty ManagementInstead, primary users of your Back-End Tools are your CSRs, your Security Team, your Marketing&Monetization Team, and last but certainly not least – the Almighty Management. As a result – most often, Back-End Tools are implemented as an internal web site.

Sure, admins and programmers MAY use Back-End Tools too (though in larger companies there is often an administrative prohibition on doing it in production at least for programmers2) – but Back-End Tools should be easily usable by people who have no idea about command line, threads, CIDR addressing notation, and so on.

Now, let’s take a closer look at these capabilities – and see what they might entail; on the other hand – let’s keep in mind that each game has its own unique requirements, and I will be able to mention only a very small subset of things which your specific game will need in the long run.


1 BTW, for our purposes – we do not consider databases as such as a part of the Back End Tools
2 it may also be a regulatory requirement – for example, PCI DSS calls for a clean separation between development and production requirements, and for restricting access to card numbers on need-to-know basis.

 

Control

Controlling your game is a very game-specific task, and all I can do is to give a few examples to demonstrate the range of Control-related Back-End Tools I’ve seen or heard of:

  • Announcements (ranging from server restarts to saying “hey guys, our Big Tournament is about to start, you can watch it in real-time”).
  • Preparing for server restart (pretty often – stopping creating new games and waiting until current ones finish, a process somewhat similar to Docker’s DRAIN).
  • Scheduling tournaments.
  • Adjusting gameplay parameters for different lobbies (different countries, …).
  • Managing payment methods (percentages of transactions going via different payment methods, disabling a method, etc.)
  • Managing player accounts – ranging from player ban (or “ban wave”) to giving players some goodies (for example, extending their subscription as a compensation for server crash during all-important-for-them-tournament).
  • Approving uploaded avatar images (that is, if avatar images can be uploaded).
  • etc., etc., etc.

A question whether to include admin-level manipulations (such as “let’s add this server/cloud provider/…” to the mix of the servers we’re using”) into Back-End Tools – is an open one. Still, I’d say that if these operations have some impact on gameplay – all the servers (with their parameters) should be at least visible in your back-end; usually – it is also a good idea to allow control of certain gameplay parameters for already-added servers from Back-End Tools. For example, while adding a new cloud provider can be seen as a purely admin task (and can be made without help from Back-End Tools) – pretty often, the same admin will need to add this provider into Back-End Tools, so non-admin Gameplay Team can see it and say via Back-End Tools: “hey, we can see in Monitoring that this provider has frequent connectivity problems, so we won’t run our Important Tournaments there”.

Inquisitive hare:Implementation-wise, Back-End Control Tools are usually based on a web server, plus an app server behind it – which app server then communicates with your Game ServersImplementation-wise, Back-End Control Tools are usually based on a web server, plus an app server behind it – which app server then communicates with your Game Servers (such as Game World Servers, MatchMaking Servers, Cashier Servers, etc.; more on them in Vol. III’s chapter on Server-Side Architecture).

What to use as your web server – doesn’t matter too much (it is an internal web server, so even IIS will do3); however, you better think from the very beginning on “how we’re going to integrate our app server with our Game Servers”. In other words – if you’re going to have your Back-End app server in Java while your Game Servers are C++, you’d better agree from the very beginning to use protobufs (FlatBuffers, Thrift, …, your-own-infrastructure-supporting-both-Java-and-C++) for communications between your Back-End app server and your Game Servers.


3 Just make 200%-sure it is never exposed to the Internet.

 

Monitoring

As noted above, monitoring is an important part of the Back-End Tools. In particular, CSRs should:

  • Be aware of the current problems on your side to give reasonable advice to the players. For example, if you know that all the players from a large ISP are currently disconnected (which can be easily caused by the ISP’s router failing, and BGP being misconfigured) – it is better to describe the situation to the customer instead of anemic and unprofessional “please run Windows Update and update your drivers”4
  • Raise the hell if something goes wrong – calling admins/developers/whoever-else in case of alerts raised by your monitoring tools. While such notifications can be done automatically – a well-trained 24×7 support team was seen to be more reliable in this regard.5

Technically – most of the time, your CSRs will just use a custom view (“custom device report”, etc.), which you create specifically for them in you monitoring system such as Zenoss or Zabbix. Also – it is a good idea to have your monitoring system to push alerts into your Email Queue;6 as your CSRs will spend most of their time there – alerting them via Email Queue tends to work extremely well.


4 Translation: “we have no idea what is going on, but maybe while you’re doing it – the problem will go away by itself”
5 And of course, nothing prevents you from both running automated notifications, and have procedures for support who should start calling admins.
6 And Email Queue should show such alerts with a topmost priority – more on it below.

 

Reporting

Reporting is one thing which starts small, but later takes most of your Back-End Team efforts (though ongoing integration with your Help Desk can be time-consuming too). Once, I’ve seen a game, which started with just a single report showing all the players – and over several years got to over a thousand of different reports.

The key with reporting is to

Make sure that your reports are CSR-driven7

It is pretty much hopeless to try to predict which reports you’ll need in the future; even if we try to guess it – we’ll be hopelessly wrong 99% of the time <sigh />. As a result – I am a strong proponent of starting to implement any of the reports only when somebody comes and says “hey, we need such and such data presented in such-and-such way”. Not only it will save you efforts – but also it will allow to provide exactly what is necessary for CSRs/Marketing/Management…

Hare pointing out:Reports in games can be divided into two very broad categories: operational reports and analytical reports.From what I’ve seen, reports which can be seen in games, vary greatly; still, they can be divided into two very broad categories: operational reports and analytical reports. Operational reports are usually mostly used by CSRs, and analytical ones – by Marketing/Monetization Teams, and by Management.

Examples of operational reports (those which are not too game-specific) include:

  • All the player data for player X.
  • Detailed play history for player X (usually this report differs from the previous one to avoid cluttering).
  • Stats and history of two players X and Y playing together (these are often used by Security Team, in case if some kind of collusion between players is suspected).
  • History of payments for player X.
  • History of all the payments for payment provider P over time period (this one is necessary for reconciliation with payment providers).
  • Show all the people who ever shared any credit cards with player X.
  • Show all the people who ever shared an IP address with player X. (NB: this report is a risky one, and generally you should check whether the number of such players is within reason, before starting to show a million-entry-long list of the people who happen to share the same proxy).

Examples of analytical reports include:

  • How many new registrations came in (money was generated, etc.) during specified time frame? NB: this report is actually a borderline between the operational and analytical ones, but let’s list it here.
  • Per-day registrations AND active players in such and such country (city, …).
  • How much money have those-players-who-registered-during-specified-time-frame brought over their first 60 days after registration? This kind of reports is common to analyze promotion results.
  • How well are players coming from such-and-such affiliate (or via certain ad source) they are doing (both in terms of time spent on site, and in terms of money generated)?
  • The same thing – but comparing all the affiliates.
  • ROI Return on investment (ROI) is the benefit to an investor resulting from an investment of some resource. A high ROI means the investment's gains compare favorably to its cost.— Wikipedia —Per-affiliates (per-ad-source) ROI.
  • etc., etc.

Implementation-wise, I would argue against using fancy GUI-based reporting tools such as Crystal Reports (Active Reports, etc.); in fact, once upon a time I’ve seen people trying Crystal exactly as a tool to implement game reporting – just to realize that it is waaay too restrictive, and to move away to ASP-based custom-written reports (which worked like a charm BTW).

In particular –

I am clearly insisting on writing all your SQL and NoSQL queries by hand

As soon as this requirement is complied with – I am pretty much agnostic to the specific app server technology you’ll be using; any of {.NET|Java|Python|even-PHP} (with appropriate frameworks if you feel like it) will do.

Further two all-important things to keep in mind are that:

  • From the very beginning, be prepared to run your reports off the replica (more on replicas in Vol. VI’s chapter on Databases)
  • Keep in mind that while operational reports are usually run from a replica-which-is-structurally-identical to the OLTP DB – the analytical reports are often very different:
    • They are using aggregates, and/or
    • They’re using OLAP cubes, and/or
    • They’re using a NoSQL DB

As a result of those structurally-different replicas for analytical reporting – such reports often become a joint effort, with parts of the job shared between the Database Team (which should prepare the data in a structure which will be usable by reporting), and Back-End Team (which is responsible for retrieving the data and displaying it to the end-user)8.


7 Marketing Team-driven, Management-driven, etc.
8 and actual SQL/NoSQL queries can easily end up on the either side, though usually for big databases, Database Team is better positioned to write non-trivial queries.

 

On Reports with Too Many Rows

One very important observation about reports is that

Any report which is longer than a few hundred rows, is not immediately usable by people.

Looking over a table with thousands of rows rarely makes much sense; it means that if your CSRs/Marketing need these-long-reports – most likely, they’re doing some post-processing with the data. And if it happens often enough – it means that their work can be optimized by providing another report which does this post-processing for them.

As a result – it is often a good idea to log number of rows returned on the server-side, then have a script which scans logs to find those reports which are often returning too many rows – and then manually investigate what can be done to simplify the work flow.

Actually, this is one manifestation of a more generic rule of

Being proactive with optimization of the work flow of your CSRs.

This rule is very important to ensure that your CSRs are both efficient and happy – and this, in turn, is extremely important to keep both your management and players happy.

 

On Visualization

Assertive hare:while content of the reports should be driven by CSRs / Marketing / Management – presentation of the reports should be as uniform as possible.When speaking about reports, there are actually two distinct aspects of them: content and visualization. And while, as discussed above, content of the reports should be driven by CSRs/Marketing/Management – presentation of the reports should be as uniform as possible. In particular, ideally:

  • There should be ability to sort by any column
    • Multi-column sorting is even more useful
  • Ability to export any report to CSV (so it can be imported into Excel for further processing9)
  • For all aggregate reports (such as per-country analytical reports discussed above) – there should be a uniform way to visualize report as a graph.

All these features should be done purely on the browser-side by JavaScript, so that no additional SQL queries are ever required when playing with visualization. Implementation-wise – it should be done by using your own “glue” library layer to render your data from your reports (without generating any of JS explicitly(!), and without calling any 3rd-party stuff right from your reports); this approach will allow to add more visualization capabilities (and/or change your 3rd-party visualization library) later, and without rewriting all those hundreds of existing reports.


9 NB: due to potential for cheating, some of the reports may need have this capability disabled (and copy-pasting may be disabled too)

 

[[To Be Continued…

Tired hare:This concludes beta Chapter 26(a) from the upcoming book “Development and Deployment of Multiplayer Online Games (from social games to MMOFPS, with social games in between)”.

Stay tuned for beta Chapter 26(b), where we’ll briefly discuss helpdesk software]]

Don't like this post? Comment↯ below. You do?! Please share: ...on LinkedIn...on Reddit...on Twitter...on Facebook

Acknowledgement

Cartoons by Sergey GordeevIRL from Gordeev Animation Graphics, Prague.

Join our mailing list:

Comments

  1. says

    Footnote 9 doesn’t exist.

    I assume you were going to make a snide comment on people who live inside excel, no matter what you build for them.

    • "No Bugs" Hare says

      ;-). Thanks, fixed. BTW, I don’t say Excel is inherently bad – because building all the functionality of Excel into Back-End is way too much even for such a DIY fan as me :-). Sure, common things DO need to be implemented within the Back-End, but I’m actually very happy if marketing can post-process once-in-a-lifetime-needed-report in Excel.

Leave a Reply

Your email address will not be published. Required fields are marked *