Back End Tools for MOGs

	Author:	“No Bugs” Hare  Follow:
	Job Title:	Sarcastic Architect
	Hobbies:	Thinking Aloud, Arguing with Managers, Annoying HRs, Calling a Spade a Spade, Keeping Tongue in Cheek

[rabbit_ddmog vol=”7″ chap=”Chapter 26(a) from “beta” Volume VII”]

One all-important thing which is very often neglected in MOGs – is Back End Tools. On the other hand, Back End Tools is one thing which is extremely important for your game. After all – it is Back End which controls (and allows to see <smile />) all the money flowing into your game; even more importantly – it is the way your CSRs can access the system, and reply to support requests (ranging from “Help! I lost my password” to “I paid you my $1.99, but got nothing”).

Within this book, we will understand Back End as a set of tools, which provides the following capabilities:

• Control. Whatever your game is – you’ll probably want to change its operational parameters (or more generally – control your game operation at business level). This includes such things as running scheduled tournaments, changing gameplay parameters, announcing server restart to all the relevant players, etc. etc. etc.
• Monitoring. It is all-important that your support team knows the current health of the system at any given point in time (saying to the player complaining that the game is slow, to reboot his computer – while it is your servers which are lagging – is usually not the best idea).
• Reporting. In addition to controlling your game – you will need to get all the data your non-programmers need, from the databases.¹ This includes all kinds of data, ranging from operational data such as “Last sessions played by player X”, to analytical data such as “Give me per-country stats of all the players filtered according to such-and-such criteria”.
• Help Desk. Most of the time, you will find that your Help Desk is just a fancy name for an elaborated-and-highly-customized Email Queue. While in theory, in addition to e-mails you can have tickets, live chat, and even phone call center – in practice, pushing your players into explicit tickets is unrealistic, and live chat / phone is just too expensive to run (with a few narrow exceptions discussed below).

While the spectrum of these tools is rather wide – they have one significant thing in common:

“primary users of your Back-End Tools are your CSRs, your Security Team, your Marketing-&-Monetization Team, and last the Almighty ManagementInstead, primary users of your Back-End Tools are your CSRs, your Security Team, your Marketing&Monetization Team, and last but certainly not least – the Almighty Management. As a result – most often, Back-End Tools are implemented as an internal web site.

Sure, admins and programmers MAY use Back-End Tools too (though in larger companies there is often an administrative prohibition on doing it in production at least for programmers²) – but Back-End Tools should be easily usable by people who have no idea about command line, threads, CIDR addressing notation, and so on.

Now, let’s take a closer look at these capabilities – and see what they might entail; on the other hand – let’s keep in mind that each game has its own unique requirements, and I will be able to mention only a very small subset of things which your specific game will need in the long run.

Control

Controlling your game is a very game-specific task, and all I can do is to give a few examples to demonstrate the range of Control-related Back-End Tools I’ve seen or heard of:

• Announcements (ranging from server restarts to saying “hey guys, our Big Tournament is about to start, you can watch it in real-time”).
• Preparing for server restart (pretty often – stopping creating new games and waiting until current ones finish, a process somewhat similar to Docker’s DRAIN).
• Scheduling tournaments.
• Adjusting gameplay parameters for different lobbies (different countries, …).
• Managing payment methods (percentages of transactions going via different payment methods, disabling a method, etc.)
• Managing player accounts – ranging from player ban (or “ban wave”) to giving players some goodies (for example, extending their subscription as a compensation for server crash during all-important-for-them-tournament).
• Approving uploaded avatar images (that is, if avatar images can be uploaded).
• etc., etc., etc.

A question whether to include admin-level manipulations (such as “let’s add this server/cloud provider/…” to the mix of the servers we’re using”) into Back-End Tools – is an open one. Still, I’d say that if these operations have some impact on gameplay – all the servers (with their parameters) should be at least visible in your back-end; usually – it is also a good idea to allow control of certain gameplay parameters for already-added servers from Back-End Tools. For example, while adding a new cloud provider can be seen as a purely admin task (and can be made without help from Back-End Tools) – pretty often, the same admin will need to add this provider into Back-End Tools, so non-admin Gameplay Team can see it and say via Back-End Tools: “hey, we can see in Monitoring that this provider has frequent connectivity problems, so we won’t run our Important Tournaments there”.

“Implementation-wise, Back-End Control Tools are usually based on a web server, plus an app server behind it – which app server then communicates with your Game ServersImplementation-wise, Back-End Control Tools are usually based on a web server, plus an app server behind it – which app server then communicates with your Game Servers (such as Game World Servers, MatchMaking Servers, Cashier Servers, etc.; more on them in Vol. III’s chapter on Server-Side Architecture).

What to use as your web server – doesn’t matter too much (it is an internal web server, so even IIS will do³); however, you better think from the very beginning on “how we’re going to integrate our app server with our Game Servers”. In other words – if you’re going to have your Back-End app server in Java while your Game Servers are C++, you’d better agree from the very beginning to use protobufs (FlatBuffers, Thrift, …, your-own-infrastructure-supporting-both-Java-and-C++) for communications between your Back-End app server and your Game Servers.

Monitoring

As noted above, monitoring is an important part of the Back-End Tools. In particular, CSRs should:

• Be aware of the current problems on your side to give reasonable advice to the players. For example, if you know that all the players from a large ISP are currently disconnected (which can be easily caused by the ISP’s router failing, and BGP being misconfigured) – it is better to describe the situation to the customer instead of anemic and unprofessional “please run Windows Update and update your drivers”⁴
• Raise the hell if something goes wrong – calling admins/developers/whoever-else in case of alerts raised by your monitoring tools. While such notifications can be done automatically – a well-trained 24×7 support team was seen to be more reliable in this regard.⁵

Technically – most of the time, your CSRs will just use a custom view (“custom device report”, etc.), which you create specifically for them in you monitoring system such as Zenoss or Zabbix. Also – it is a good idea to have your monitoring system to push alerts into your Email Queue;⁶ as your CSRs will spend most of their time there – alerting them via Email Queue tends to work extremely well.

Reporting

Reporting is one thing which starts small, but later takes most of your Back-End Team efforts (though ongoing integration with your Help Desk can be time-consuming too). Once, I’ve seen a game, which started with just a single report showing all the players – and over several years got to over a thousand of different reports.

The key with reporting is to

It is pretty much hopeless to try to predict which reports you’ll need in the future; even if we try to guess it – we’ll be hopelessly wrong 99% of the time <sigh />. As a result – I am a strong proponent of starting to implement any of the reports only when somebody comes and says “hey, we need such and such data presented in such-and-such way”. Not only it will save you efforts – but also it will allow to provide exactly what is necessary for CSRs/Marketing/Management…

“Reports in games can be divided into two very broad categories: operational reports and analytical reports.From what I’ve seen, reports which can be seen in games, vary greatly; still, they can be divided into two very broad categories: operational reports and analytical reports. Operational reports are usually mostly used by CSRs, and analytical ones – by Marketing/Monetization Teams, and by Management.

Examples of operational reports (those which are not too game-specific) include:

• All the player data for player X.
• Detailed play history for player X (usually this report differs from the previous one to avoid cluttering).
• Stats and history of two players X and Y playing together (these are often used by Security Team, in case if some kind of collusion between players is suspected).
• History of payments for player X.
• History of all the payments for payment provider P over time period (this one is necessary for reconciliation with payment providers).
• Show all the people who ever shared any credit cards with player X.
• Show all the people who ever shared an IP address with player X. (NB: this report is a risky one, and generally you should check whether the number of such players is within reason, before starting to show a million-entry-long list of the people who happen to share the same proxy).

Examples of analytical reports include:

• How many new registrations came in (money was generated, etc.) during specified time frame? NB: this report is actually a borderline between the operational and analytical ones, but let’s list it here.
• Per-day registrations AND active players in such and such country (city, …).
• How much money have those-players-who-registered-during-specified-time-frame brought over their first 60 days after registration? This kind of reports is common to analyze promotion results.
• How well are players coming from such-and-such affiliate (or via certain ad source) they are doing (both in terms of time spent on site, and in terms of money generated)?
• The same thing – but comparing all the affiliates.
• ROI Return on investment (ROI) is the benefit to an investor resulting from an investment of some resource. A high ROI means the investment's gains compare favorably to its cost.— Wikipedia —Per-affiliates (per-ad-source) ROI.
• etc., etc.

Implementation-wise, I would argue against using fancy GUI-based reporting tools such as Crystal Reports (Active Reports, etc.); in fact, once upon a time I’ve seen people trying Crystal exactly as a tool to implement game reporting – just to realize that it is waaay too restrictive, and to move away to ASP-based custom-written reports (which worked like a charm BTW).

In particular –

As soon as this requirement is complied with – I am pretty much agnostic to the specific app server technology you’ll be using; any of {.NET|Java|Python|even-PHP} (with appropriate frameworks if you feel like it) will do.

Further two all-important things to keep in mind are that:

• From the very beginning, be prepared to run your reports off the replica (more on replicas in Vol. VI’s chapter on Databases)
• Keep in mind that while operational reports are usually run from a replica-which-is-structurally-identical to the OLTP DB – the analytical reports are often very different:
  • They are using aggregates, and/or
  • They’re using OLAP cubes, and/or
  • They’re using a NoSQL DB

As a result of those structurally-different replicas for analytical reporting – such reports often become a joint effort, with parts of the job shared between the Database Team (which should prepare the data in a structure which will be usable by reporting), and Back-End Team (which is responsible for retrieving the data and displaying it to the end-user)⁸.

On Reports with Too Many Rows

One very important observation about reports is that

Looking over a table with thousands of rows rarely makes much sense; it means that if your CSRs/Marketing need these-long-reports – most likely, they’re doing some post-processing with the data. And if it happens often enough – it means that their work can be optimized by providing another report which does this post-processing for them.

As a result – it is often a good idea to log number of rows returned on the server-side, then have a script which scans logs to find those reports which are often returning too many rows – and then manually investigate what can be done to simplify the work flow.

Actually, this is one manifestation of a more generic rule of

This rule is very important to ensure that your CSRs are both efficient and happy – and this, in turn, is extremely important to keep both your management and players happy.

On Visualization

“while content of the reports should be driven by CSRs / Marketing / Management – presentation of the reports should be as uniform as possible.When speaking about reports, there are actually two distinct aspects of them: content and visualization. And while, as discussed above, content of the reports should be driven by CSRs/Marketing/Management – presentation of the reports should be as uniform as possible. In particular, ideally:

• There should be ability to sort by any column
  • Multi-column sorting is even more useful
• Ability to export any report to CSV (so it can be imported into Excel for further processing⁹)
• For all aggregate reports (such as per-country analytical reports discussed above) – there should be a uniform way to visualize report as a graph.

All these features should be done purely on the browser-side by JavaScript, so that no additional SQL queries are ever required when playing with visualization. Implementation-wise – it should be done by using your own “glue” library layer to render your data from your reports (without generating any of JS explicitly(!), and without calling any 3^rd-party stuff right from your reports); this approach will allow to add more visualization capabilities (and/or change your 3^rd-party visualization library) later, and without rewriting all those hundreds of existing reports.

[[To Be Continued…

This concludes beta Chapter 26(a) from the upcoming book “Development and Deployment of Multiplayer Online Games (from social games to MMOFPS, with social games in between)”.

Stay tuned for beta Chapter 26(b), where we’ll briefly discuss helpdesk software]]

Don't like this post? Comment↯ below. You do?! Please share:

Acknowledgement

Cartoons by Sergey Gordeev from Gordeev Animation Graphics, Prague.