Eight Ways to Handle Non-Blocking Returns in Message-Passing Programs – with Script

	Author:	“No Bugs” Hare  Follow:
	Job Title:	Sarcastic Architect
	Hobbies:	Thinking Aloud, Arguing with Managers, Annoying HRs, Calling a Spade a Spade, Keeping Tongue in Cheek

Historically, first non-blocking implementations were based on plain messages; such an implementation for our “Item Purchase” example is shown on the slide. Yes, it is THIS bad.

Let’s take a bit closer look at it: 

First, we need to create a struct to store the context of our request (so we can handle return properly). There is nothing really meaningful here – but we still have to write this boilerplate code.

Then, we have to add a map of request_ids to this request data, so we can find the necessary request context when reply arrives back.

While in some cases it is possible to say that there is only one outstanding request per type, in general – it is better to avoid such things as they have a tendency to become unmanageable rather quickly. The same stands for ad-hoc tricks which allow to identify replies by embedding additional information into reply instead of request_id.

As we got our initial request from the Client – we need to save the context into our struct, compose the message using the function prepared by our IDL compiler, send the message, and save our struct into the map.

BTW, in addition to being verbose, this code is also error-prone: this kind of code is not only cumbersome, but is also pretty easy to get wrong.

Then, on receiving reply from DB (Re)Actor, we need to find the request in the map, and check its status

Now, just for a change, we have one line of non-boilerplate meaningful code (the one which was present in our blocking version)

…but as you have already guessed – it is followed by even more boilerplate.

Overall, amount of boilerplate code in Take 1 is enormous: out of over-70(!) lines-of-code in Take 1, only about 10 are meaningful. This becomes especially obvious when placing Take 1 and our “Holy Grail” baseline code side by side…

This slide pretty much summarises the problems with Take 1: amount of boilerplate code is soooo large, that all the business logic is completely buried in it. What’s interesting though – is that I’ve seen perfectly working systems based on this kind of non-blocking handling (though their maintenance costs was a completely different story).

For our Take 2, we’ll consider a technique which is quite popular in game development today – and based on non-blocking VOID-ONLY RPC calls. The idea behind using void (and non-throwing) RPC calls is that as there is no reply whatsoever to such calls – there is no question “what to do when the reply arrives”.

On the other hand – such void-only RPC calls provide only very marginal improvement over plain messages; in particular, to implement a typical request-response exchange, we need to have two void RPC calls – one to send request, and another to send reply back.

As a result, most of the boilerplate stuff from Take 1 such as request context struct,

map of request_ids into these structs,

as well as filling context on sending request, and finding it on receiving reply – are all still here, with all the associated error-prone stuff.

Overall, improvements in readability and reduction of boilerplate code, provided by Take 2, are not that large as we’d like them to be. While 50 lines-of-code is indeed better than 70, it is still five-times more than in our target “Holy Grail” code.

Our Take 3 is about object-oriented callbacks. It is a thing which I was doing myself almost 20 years ago when architecting a pretty large (Re)Actor-based system; the architecture is still in use pretty much without changes – and it does work.

As we can see from slightly increased font size, Take 3 is just a bit less verbose than Take 2. With Take 3, it is all about callback objects – and it takes lots of boilerplate to describe them. We need a callback object to handle return from our non-blocking request to the database…

and another callback object for the non-blocking call to our Game World

And only after we’re done with this meaningless boilerplate (and are over 50% down our Take 3 code) – we get to somewhat-meaningful stuff (it still has lots of boilerplate, but at least it is interspersed with SOMETHING meaningful).

Currently highlighted is the code processing original purchase request from Client…

and here is return handler. One thing which we can see, is that – compared to Takes 1 and 2 – there is no explicit handling of the request_id-to-callback-data maps; in Take 3, while these maps are still present behind the scenes (there is no magic here), we managed to hide them from the view of app-level developer (which is a REALLY BIG relief).

If we compare our Take 3 to our baseline “Holy Grail” code, we’ll see that Take 3, while being less verbose than Take 2, still has like 4-times more lines of code than we’d like to have. On the plus side, however (in addition to being less-error-prone than Takes 1 and 2), we can see that meaningful code in Take 3 is concentrated within a smaller number of lines, which significantly improves its readability.

Overall, Take 3, while being verbose, has been observed to be perfectly usable and perfectly working. I would even go as far as saying that it is the best thing we could do before the advent of C++11, and most importantly – before lambdas.

Up to now, all our Takes were doable even in C++98; now let’s see how we can improve it with the power provided by C++11.

Our Take 4 is all about lambdas, and more lambdas, leading us to the infamous “lambda pyramid”. We can immediately see that the amount of boilerplate code went down very significantly; actually, it is the first time when our non-blocking code became somewhat-readable on a single slide.

Ideologically, the code in Take 4 is actually very similar to the Object-Oriented Callbacks from Take 3: it is just that instead of those verbose manually-written object-oriented callbacks, compiler creates equivalent lambda closures for us behind the scenes.

One important point about C++ and lambdas is that while we’re using lambda capture ”by copy”, it still captures current object (the one referred to by THIS) by reference. Accidentally, this is EXACTLY the behaviour we want for our purposes: we DO want to capture stack variables by value (so they survive return from our cashierPurchaseItem function), but at the same time we DO want to capture our (Re)Actor object by reference, so we can access its CURRENT state while processing non-blocking returns.

If we compare our Take 4 to our “Holy Grail” code, we’ll see that verbosity-wise they’re about the same.

However, readability-wise Take 4 is still far from perfect (and it will become more obvious if we introduce exception handling). Most importantly, the code which is logically linear on the right side – becomes NESTED within our Take 4 code. This (as well as rather counterintuitive indents for what-is-essentially-linear-logical-code) is what earned Take 4 approach the name “lambda pyramid”; as soon as the number of non-blocking operations in original non-linear code goes to 4-5, the whole thing becomes rather poorly readable.

Let’s see how we can improve it further. Our Take 5 is about so-called FUTURES. 

Very very briefly, the idea is to create a placeholder, known as “future”, where the result of some operation which-will-only-become-known-in-the-future can be stored

And as soon as we have these “future” objects – we can specify actions which have to be done as soon as the result within the future becomes available. These actions are often referred to as continuations.

If we compare our Take 5 to our baseline “Holy Grail” code – we’ll see that verbosity of futures is still in check, but, unlike “lambda pyramids”, the code-which-was-linear-in-blocking – is still linear with futures.

I’d certainly say that Take 5 is the best we have so far. BTW, as an added benefit, concepts such as “we need to wait for TWO non-blocking RPCs to complete”, are easily expressible with the futures too.

Last but not least about our Take 5, let’s note that in spite of being CONCEPTUALLY similar to std::future<> in some sense, our ReactorFutures are not identical to it. Most importantly – std::future<> is primarily about THREAD sync, and our ReactorFutures, being used within one single thread, are quite different, and are more similar to Facebook’s folly::Futures (though it SEEMS that future futures from std::experimental will implement continuations, and will become what-we-want).

If we extend the idea of futures a bit further, we can get to constructing the whole code trees from lambdas. The value of such approach becomes obvious as soon as we realise that not all the code is linear, and that there are conditions and exceptions. As we’re very limited on time today, I won’t give an example with conditions – but here is an example comparing our-“Holy Grail” code with exceptions added, to the “Code Builder” approach in our Take 6.

The idea behind “Code Builder” is that we’re effectively “building” the code tree in runtime (in CCode object), and as soon as the code tree is built – we can run it in a perfectly asynchronous manner. Most importantly for our current purposes is that elements of our code tree directly correspond to the elements of our original “Holy Grail” code: 

Whenever we have “try” on the right side – we have “ttry” on the left, whenever we have linear code on the right – we have more-or-less similar code on the left too, and so on. And even if we’d have loops with non-blocking calls within – we’d still have exact 1-to-1 correspondence between blocking code and non-blocking one, and WITH EXACTLY THE SAME NESTING TOO.

Overall, Take 6, while being admittedly more verbose than Take 5, is MUCH more flexible than that, and (before the advent of co_await) could be a good choice for more complicated use cases.

BTW, if we introduce preprocessor to the picture, we can make the code in Take 6 significantly less verbose (while preserving all its good properties). Whether improved readability is worth the trouble of debugging code which uses heavy preprocessor macros – is still unclear, but such a possibility does exist.

Our Take 7 will be about stackful coroutines and/or fibers. Fortunately for us, we don’t have to go into a lengthy discussion about the differences between the two; for us, it is sufficient to say that fibers and stackful coroutines are pretty much the same for our purposes.

In not-exactly-standard but still-widely-cross-platform C++, stackful coroutines are represented by boost::coroutines (though personally I’d prefer my IDL compiler to use boost::context directly).

As we can see, the code in our Take 7 on the first glance looks better than that of the Take 6 and Take 6a. However, it comes at a cost of two important caveats…

First, with stackful fibers/coroutines, it becomes very difficult to serialise state of our Reactor – and to achieve quite a few all-important properties of Reactor (including post-mortem debugging and low-latency fault tolerance) it IS necessary to serialise Reactor state.

Granted, in C++ serialisation is not a picnic even with lambdas, but with stackful co-routines I don’t even know how we can approach this task (BTW, if you do – I would be VERY interested in hearing it after the talk).

Even more importantly, stackful co-routines come with a Big Fat word of caution. While with fibers/coroutines we CAN avoid futures and write the code EXACTLY as a blocking one – we DON’T really want to do it.

When we compare Take 7x (using ‘x’ to denote that it SHOULD NOT be used) to the “Holy Grail” code – we’ll notice that there is one thing missing from the Take 7x: it is those “REENTRY” markers which are necessary to indicate those points where the Reactor state can be modified. This, in turn, as we discussed above, leads to difficult-to-see errors, and to significantly increased code maintenance costs. As a result – I do NOT recommend using Take 7x in real-world projects.

A really short real-world story in this regard. Some time ago, I was presenting the Take 7x option to a billion-dollar company having several million-lines-of mostly-non-blocking (Re)Actor code (the idea was to replace Take-3-they’re-currently-using, with something more palatable). Long story short: right away after seeing this Take 7x – they asked me: “Hey, how we’ll know those points where the state can be modified??“.

It illustrates one of the earlier raised points – that THOSE REENTRY MARKERS are REALLY important for real-world non-blocking development.

Our last Take, Take 8 is related to the concept which is well-known for the other programming languages (such as C#), but is a new-kid-on-the-block for C++. I’m speaking about await (which was renamed into co_await in recent versions by WG21).

As of now – it is NOT a part of standard yet, and changes are still to be expected; on the other hand – implementations are available both for MSVC and Clang.

As we can see, the code in Take 8 is almost-exactly the same as our “Holy Grail” code on the right.

Even our REENTRY markers have their direct counterparts in Take 8.

Still, we have to note that at least as of current C++ proposals, Take 8 is not 100% ideal for our purposes. Most importantly, it has problems with serialising the state of our (Re)Actor, and while the problem is NOT as severe as that of Take 7, it is still going to cause quite a bit of trouble. On the plus side, it MIGHT be possible to serialise Take 8’s state as a part of serialising the whole allocator – but it is rather tricky, relies on certain properties of underlying OS, and as a result – causes complications with ASLR. From our current perspective- it is currently not 100% clear whether it is really GUARANTEED that ALL the co_await frames will REALLY go via a user-definable allocator in ALL the compilers and libraries (also we need a guarantee that await-frames won’t use anything else but allocator, so any on-stack storage for await frames doesn’t happen).