Indirect and Client-Centric Payment Processing. Logging and Reconciliation

	Author:	“No Bugs” Hare  Follow:
	Job Title:	Sarcastic Architect
	Hobbies:	Thinking Aloud, Arguing with Managers, Annoying HRs, Calling a Spade a Spade, Keeping Tongue in Cheek

To navigate through the book, you may want to use Development&Deployment of MOG: Table of Contents.]]

As we’ve seen – direct processing, while fairly straightforward, has its own share of troubles (in particular, PCI DSS, while certainly doable, is a Big Fat Problem™). No wonder that there are alternative ways of payment processing out there; we’ll further separate them into “indirect” and “client-centric”.

Indirect Processing

To avoid complying with PCI DSS (which is indeed a Big Headache for smaller companies) – lots of Payment Providers are using what I name “Indirect Processing”.

Historically, one of the largest providers using Indirect Processing, is Paypal; rather recently, it got a very serious competition, in particular Stripe. I won’t get into an argument which one is better (especially as the situation is changing all the time) – what is important though is that this competition has made things better (both for merchants and for end-users).

The idea behind Indirect Processing is fairly simple (though is not nearly as straightforward as the one of Direct Processing described above). The flow of Indirect Processing (give or take) goes along the following lines:

• Your player indicates that she wants to purchase something for your game. Great!
• You go from your Client to your Server-Side, which eventually results in your Server-Side DB creating some kind of a “transaction request”; this transaction request usually has transaction details (such as “what it is for”, transaction request ID, and amount).
• You redirect your customer to the payment provider’s web site (usually providing transaction request ID).
  • “Unfortunately, neither in-app browser nor opening an URL is entirely good.At this point, if we’re speaking about downloadable Client, you need to open a browser window. In general, you have a choice of opening in-app browser or open an URL (hoping that OS will launch appropriate browser). Unfortunately, neither one of these options is entirely good.
    • In-app browser windows tend to use OS-specific browser without an opportunity for the end-user to override it; moreover – (a) share of non-default browsers (notably Chrome) is very high these days; and (b) if you’re opening a web browser which is not the one that your customer is usually using – all the customer settings (login etc.) from that-browser-player-really-uses – can be lost (which can easily cause your player to drop off from the purchase). Also – in-app browsers are well-known for changing their integration APIs all the time (which in turn often causes crashes after the end-user OS upgrades – ouch!).
    • On the other hand, opening an URL usually does allow to launch customer’s preferred browser – but being in a separate window, it causes some perception disconnect between your app and the browser, which causes some customers to drop off at that point.
    • Overall, in a balance of pros and cons, I would say that opening an URL is a more reliable choice (especially if you take all the possible measures that browser window is opened on top of your app and not underneath) – but YMMV even more than usual.
• Payment provider’s web site authorizes the customer (requests credit card details, whatever) – and makes sure that she authorizes specific payment too. All this happens beyond our control (but can still affect customer drop out rate).
• If customer does authorize the payment – you will get a confirmation coming from payment provider’s Server about it. Ways of this confirmation can vary significantly:
  • Server-Side callback: provider’s web site invoking a special URL on your web site.
  • Polling: you need to poll some URL on provider’s web site (usually using you transaction request ID).
  • Client-Side notification: Client-Side JS calling your web site to notify you that the transaction went through.
    • “Note that under classical Indirect Processing what you get is just an unsigned notification, which means that you MUST NOT trust it.Note that under classical Indirect Processing what you get is just an unsigned notification, which means that you MUST NOT trust it. Therefore, after receiving this notification you still MUST go to the provider’s web site to check whether it is true.
    • Let’s also note that Client-Side notifications are not really reliable in the context of Indirect Processing: if the payment has already been done on Payment Provider’s side – and then Client crashes (gets disconnected, etc. etc.) – you end up with a situation when your customer has paid – but you don’t know about it, so you don’t give her whatever-she-has-paid-for. Doubleplusungood.
      • To avoid it, Client-Side notifications usually need to be complemented by some kind of polling.
      • An alternative approach to addressing this inherent Client unreliability is used by Stripe (also see below on Paypal’s v.zero): when they’re authorizing payment from the customer – they’re not performing the transaction yet (i.e. money don’t change hands). Instead – they give Client an “authorization token”, which you (merchant) then use to actually perform the transaction. With Stripe-style “tokens”, if Client hangs/crashes/etc. – the transaction will never happen, solving quite a few problems (though it still may leave the customer wondering whether the transaction went through, so you better have reporting in place).
        • Note that while the ideology of Paypal’s v.zero/Braintree is similar to that of Stripe, their terminology is very different (which in turn is extremely confusing if migrating from one of them to another one). Stripe’s “token” roughly corresponds to v.zero’s “nonce”, and v.zero’s “token” roughly corresponds to the “transaction request ID” we discussed above.
• After you get the confirmation (and as noted above, you MUST get it not from the Client, but from the payment provider’s Server, with all the channel authentication/TLS etc.) – you can be sure that the transaction did indeed go through, and can proceed with crediting your player’s account with whatever-she-paid-for. Bingo! The payment process is completed.

Of course, this process is inherently more convoluted than Direct Processing; on the other hand – it (a) allows you to process credit cards indirectly, via payment provider – and without going through PCI DSS compliance yourself,¹ and (b) it allows to go beyond credit card processing; for example, if using Paypal – your customers will be able to pay even if they do NOT have a credit card but they have money in their Paypal account.

“These benefits don’t come for free though.These benefits don’t come for free though. From purely technical point of view (i.e. not speaking about fees etc.) – the following (rather minor) caveats the need to be addressed when dealing with Indirect Processing:

• As already noted above – you MUST NOT trust Client, so you MUST get your transaction validation from the payment provider’s Server.
• You will have a lot of “transaction requests” which will never materialize into transactions (i.e. won’t be confirmed, ever). It is perfectly normal, and is not a significant problem per se, just make sure NOT to remove these “transaction requests” from your DB too early (I’ve seen a transaction coming through over a month after it was initiated – and nobody, including payment provider, was able to explain how it could have possibly happened).
• You MUST double-check that the payment details coming in payment confirmation, exactly match those ones stored in your transaction request. Occasionally (and very rarely) – they don’t match, and in such cases you MUST flag the transaction and let your CSRs to deal with it manually. In theory – it should never happen, but I saw it happening way too often (and for very different reasons too) to disregard this problem.
• You MUST be prepared to handle a “stray” transaction coming out of nowhere (i.e. with incorrect transaction request ID, etc.) – once again, just flag it and let your CSRs to deal with it manually. Make sure to implement this handling – otherwise you can end up with refusing to get money (ouch!) and/or unaccounted money (double ouch!).
• Make sure that your reconciliation report (more on reconciliation in general in [[TODO]] section below) aims to use the same transaction time as your provider. Most of the time, they will record the transaction when it’s confirmed (and not when it’s initiated) – and you should do the same to reduce your reconciliation issues.
  • Even with this in mind, be prepared to strange problems during the reconciliation; overall – Indirect Processing tends to cause significantly more problems while reconciling, than Direct Processing.

Client-Centric Processing

As we can see, with Indirect Processing, even if client “tokens” or “nonces” are involved, the final transaction is always confirmed (or even performed) via Server-to-Server connection. Moreover, given that we MUST NEVER trust the Client with payments – it might seem that using such direct Server-2-Server communications is the only feasible way of doing it.

Apparently, it is not the case; in other words –

Two prominent examples of such payment systems are Google Play In-App Billing and Apple In-App Purchases (see [GoogleInAppBilling] and [AppleInAppPurchase] respectively for details).

With Client-Centric payment processing, everything (not really surprisingly) revolves around the Client, and typical transaction flow may look as follows:

• Your player indicates that he wants to purchase something for your game. Great!
• Optional (actually, Google Play-specific): Your Client (app) sends a request to the Payment Provider’s Server, with an ID of the item to be purchased – and receives some kind of “request token” (“pending intent” in Google Play-speak).
• Your Client makes a payment request to a local OS (obviously, for Google Play it is normally Android, and for Apple it is iOS or OS X) – which in turn initiates checkout UI for the purchase. When local OS is done with it (and it handles all the stuff such as credit card details, communication with Payment Provider Servers etc. etc.) – it returns back to the app with an information “whether the purchase did go through”. If it did go through – you get an “confirmation token” (known as “intent” for Google Play purchases, and “receipt” for Apple App Store).
• That’s it – there is no your Server in this picture, just Client and Payment Provider’s Server (and the latter is of not much use for you).

This model may seem completely unsuitable for MOGs (at least to our ones, which are based on authoritative Servers). Even if Client reports the “confirmation token” to our Server – we cannot possibly trust the Client, especially not with the money-related information.

“Fortunately, there is a neat crypto-trick which makes this schema usable for Server-oriented games in general - and multi-player games in particularFortunately, there is a neat crypto-trick which makes this schema usable for Server-oriented games in general (and multi-player games in particular): it is that “confirmation token” may (and normally does) contain a signature. This signature is made with either our private key which we have fed to the Payment Provider’s Server – or with the Payment Provider’s key (the former applies to Google, the latter – to Apple). This signature in the “confirmation token” is that thing which makes us able not to trust the Client anymore. If we (and Payment Provider) were careful enough with our/their private key – it means Client by itself² cannot really create a valid signature, so if the signature in anything-coming-from-Client is valid – we know for sure that the signed message actually originates from Payment Provider’s Server.

With this in mind, quite a few processing models are possible to enable Client-Centric processing for MOGs (while making sure that no purchase is lost even if Client has crashed, wasn’t able to reach our Server, etc.). For example, the following schema is rather bulletproof:

• On each start, our Client requests all the purchases it has, from the Payment Provider’s Server.
  Our Client feeds all the purchases to our Server.
• Server checks all the purchases received from the Client against its own persistent list of purchases for this player (in DB); all the purchases which are already accounted for – are ignored.
• If the purchase Server got from the Client, is new – Server checks the signature (against our public key which matches that private key fed to the Payment Provider), and does two things in the same ACID transaction:
  • Makes a record that this specific purchase has been already accounted for, and…
  • makes all the changes necessary to apply the purchase to the game (such as updating player’s account, adding abilities, items, etc.).

While this schema is quite bulletproof, it works in terms of “purchases”, and doesn’t deal with a peculiarity which applies at least to Google Play In-App purchases: “once purchased, a managed item cannot be purchased again until you consume the item”. To deal with “consuming” the items – the schema above will need to be extended as follows:

• When the Server gets a new purchase (in terms of Google Play – a “managed item”), it decides whether it wants to allow the player to purchase it once again. If yes, the following logic applies:
  • Within the very same ACID transaction, our Server makes another record in its own DB that this purchase should be consumed, and sends back to the Client an instruction so that Client should say Google Play Server that the item is consumed.
  • Also, as this notification can be lost – our Server should check current purchases (those sent by the Client) against “consumed purchases” in its DB – and send back an instruction to “consume” the item each time when Client comes with such a purchase on the list of purchases.

Phew. It wasn’t too easy – but we made it; the schema above doesn’t have any direct interaction between Payment Provider’s Servers and our Servers – and it is still pretty much bulletproof (reliable / credible) both from security point of view, and from reliability point of view (the latter means that all the purchases will be accounted for regardless of Client or network problems and crashes).

[[TODO: Bitcoin processing]]

Logging

One all-important thing to do whenever you’re dealing with money – is to record/log everything³ you’re sending to and receiving from your Payment Provider. Moreover –

“if you’re using TCP to communicate, you should log everything right before it goes into your send() call (and right after it comes out of your recv() callIn other words: if you’re using TCP to communicate, you should log everything right before it goes into your send() call (and right after it comes out of your recv() call⁴); if you’re using HTTP – I still prefer to log at TCP (or TLS) level, but it might fly if you’re logging headers and body (after all of them are already formed – that’s important).

The reason for this “very desirable” point is that whenever you’re dealing with money – any kind of problem can be very difficult to prove to the other side, and having this kind of logging tends to help a damn lot in this regard, and the farther your logging is from the transport – the more of your logic is involved, and more easy is for the Payment Provider to say “it is all your problem”. Usually (i.e. without comprehensive logging right before TCP/TLS), this is what happens 99% of the time. For those who have already been in these shoes of arguing with technical departments of Payment Providers – just one real-world story, which went unbelievably smoothly.

Once upon a time, there was a game, processing millions of dollars on a daily basis. And on one foggy day, a complaint has come from support, saying that “hey, an almighty Payment Provider says that we didn’t comply to their protocols, which caused this transaction to go astray – and causing trouble with our VIP player!”. The amount was very small – but still, a bug is a bug (and unhappy customer is an unhappy customer). Having all the logs, the reply was easy to produce, and went along the following lines:

• With respect to transaction ID XYZ, we can observe the following (with an excerpt from our log files attached):
  • At HH:MM:SS (our time) on MM/DD/YYYY, we sent the following request to you <verbatim portion of the log goes here>
  • In response, at HH:MM:S2, we got the following reply: <another verbatim portion from the log>
  • As it has field X with value Y, then, according to paragraph AA.B in your documentation, we repeated the request with such-and-such values at HH:M3:S3: <yet another portion from the log>

“The point is that with good logging it is perfectly possible to find out whether you were really compliant with the protocol, and to fix the problem (if it is on your side), or to prepare an answer which is really-difficult-to-argue-with (if it is on their side).After this reply, the Payment Provider has never come back (hopefully it was enough for them to identify and fix the bug). Moreover, it is very difficult to see how they can possibly argue with the reply above.⁵ The main takeout here, however, is not to demonstrate that Payment Provider guys make mistakes: the mistake could have easily been on the Game Server side. The point is that with good logging it is perfectly possible to find out whether you were really compliant with the protocol, and to fix the problem (if it is on your side), or to prepare an answer which is really-difficult-to-argue-with (if it is on their side).

Whenever the same problem occurs in a system without comprehensive transport-level logging, the same reply would read: “hey, our DB has such and such information, so we should have sent you such and such requests” – and this is both unconvincing, and doesn’t allow to see for sure whether there is indeed a bug on your side. Proper logging allows to answer the question of “whether it is your problem or their one” – and with 3-rd party interactions, such an answer happens to be absolutely necessary to fix the bug wherever it resides.

Or, and while we’re still discussing logging – in production, make sure to ask your admin guys to store financial logs forever and ever. Most of the time – logs are stored only for a few days, but financial logs are very different from all the other app logs (and rather small too) – so their storing policies can and SHOULD be very different from the rest of the system.

Logging and Sensitive Information. PCI DSS

One thing to be kept in mind with regards to logging is that often not all the information is allowed to be logged. As a result, the implementation of the logging of financial stuff can be severely complicated by a requirement to “mask” sensitive information. One such example is PCI DSS – which IIRC effectively disallows logging of unmasked credit card numbers (that is, if you’re using Direct Processing).

Still, even with this complication in mind, I insist that we MUST log all the exchanges with Payment Provider (even if some fields in the log are masked). In practice, if it is only CC numbers (or other absolutely-sensitive information) which is masked – it doesn’t really affect the process of issue resolution (along the lines described above); in my experience, I’ve only seen once when the specific data in the field has affected payment processing (and believe it or not – it was a problem in a backbone router which apparently dropped the packets with a very specific bit pattern; when in two days router card has failed and was replaced – the packets started to go through without any changes in the software).

Reconciliation

One thing which inevitably comes hand-in-hand with pretty much any payment processing, is reconciliation. It means that every month⁶ somebody needs to compare two pieces of data: (a) list of transactions with Payment Provider for the previous month as reported by them, and (b) the very same list as reported by your system.

“At the very best - this is if you’re using Direct Processing, and if neither your nor their payment system has any bugs, your reconciliation process will still detect some occasional transactions which were recorded in one month by your system and in another one by their system.While it may seem a very easy process (with lists being identical so obviously) – in practice it is not necessarily so. At the very best – this if you’re using Direct Processing, and if neither your nor their payment system has any bugs, your reconciliation process will still detect some occasional transactions which were recorded in one month by your system and in another one by their system.⁷

And if you’re using Indirect Processing or Client-Centric Processing (or if there are bugs in your own payment subsystem) – be prepared to learn quite a few interesting peculiarities of Payment Provider’s system as a result of reconciliation.

Concluding Chapter XVIII

Phew, we discussed a few issues related to payment processing. As anything else in this book – they’re not a rocket science, but if you didn’t know them – they could cause you lots and lots of trouble. To summarize our takeouts from Chapter XVIII:

• Chargebacks are bad, but they need to be balanced with Decline Rates.
• Direct Processing is the most straightforward – but it requires trusting the merchant (us), and as a result – for credit cards it requires PCI DSS compliance.
• While PCI DSS is doable – it is a Big Headache™, so you might want to postpone dealing with it until you’re big enough and have security specialists on your team
• Indirect processing is quite ugly (especially for games-with-downloadable-Client), but it works, and without PCI DSS too.⁸
• Client-Centric processing (especially for App Stores) can be implemented in a secure and reliable manner – in spite of lack of direct Server-2-Server communications.

[[To Be Continued…

This concludes beta Chapter 21(c) from the upcoming book “Development and Deployment of Multiplayer Online Games (from social games to MMOFPS, with social games in between)”. Stay tuned for beta Chapter 18, where we’ll discuss client updates.]]

Don't like this post? Comment↯ below. You do?! Please share:

Acknowledgement

Cartoons by Sergey Gordeev from Gordeev Animation Graphics, Prague.