MOGdev: to feature branch or not to feature branch?

	Author:	“No Bugs” Hare  Follow:
	Job Title:	Sarcastic Architect
	Hobbies:	Thinking Aloud, Arguing with Managers, Annoying HRs, Calling a Spade a Spade, Keeping Tongue in Cheek

[rabbit_ddmog vol=”3″ chap=”Chapter 11(a) from “beta” Volume III”]

In the course of our Volumes I to III, we were discussing a lot of architectural issues specific and not-so-specific to MOGs, and now you’ve hopefully already drawn a nice architecture diagram for your multiplayer game.

However, before actually starting coding, there are still a few things to do. Let’s take a look at them one by one.

Version Control

To develop pretty much anything, you do need a version control system (VCS, also known as source control system). I don’t want to go into a discussion why do you need it, just saying that there is a consensus out there on version control being necessary for all the meaningful development environments. Even if you’re a single developer, you still need version control: the version control system will act as a natural backup of your code, plus being able to rollback to that-version-which-worked-just-yesterday, will save you lots of time in the long run. And if you’re working in a team, the benefits of version control are so numerous that nobody out there dares to develop without it.

The very first question about version control is “what to put under your version control system?” And as a rule of thumb, the answer goes like

“hey, what is the mesh we should use with this source code?And yes, “pretty much everything” generally includes assets, such as meshes and textures. Moreover, it is of paramount importance to keep all the assets under the same version control system – and in the same repository – as your source code. While there are people out there advocating using Git for code and SVN for assets – I am firmly against such split-brain approaches. The main problem along this line is that being unable to synchronize two completely separate repositories, we’ll pretty much inevitably run into a situation when answering a question “hey, what is the mesh we should use with this source code?” will take much longer than it should have.

On the other hand, as with most of the rules of thumb out there, there are certain (but usually very narrow) exceptions to both “everything you need to build” and “usually NOT results of the builds” parts of the bold statement above. As an exception to “everything you need to build” part, you might want to keep certain egregiously-large-and-barely-connected-to-your-game things (such as intro videos) outside your version control system,¹ but such cases should be very few and far between. Most importantly,

, that’s one strict requirement; however, as long as you comply with this rule – you MIGHT bend all the other rules a bit.

BTW, if your source significantly depends on the toolchain you’re using (which shouldn’t be the case in theory – but it does happen way too often in practice <sad-face />²) – then to comply with this rule, you may have to get the whole toolchain into your version control (in the extreme case – a whole VM-with-all-the-tools-used-to-reproduce-the-build can become a part of source-controlled universe, but IMO this is rarely necessary for gamedev).

As for exceptions to “not including results-of-your-build” rule of thumb – I’ve seen examples when having YACC-compiled .c files within version control has simplified development flow (i.e. not all developers needed to setup YACC on their local machines), but once again – this is merely a very narrow exception from the common rule of thumb stated above. On the other hand, keeping your compiled executables (and worse – complete installers) within the version control is usually undesirable. At some point, I’ve seen a policy increasing the size of the version control DB beyond the point of being usable (and while an argument of ‘disk is cheap’ does fly to certain extent – the time necessary to checkout, as well as time necessary to backup your version control, is important – and both of them were severely affected by such a policy).

“Note that I am not arguing along the lines of “whether binaries belong to the version control”Note that I am not arguing along the lines of “whether binaries belong to the version control”. Instead – it is about “whatever-is-necessary-to-build” against “what-we-get-as-a-result-of-our-build”. While “whatever-is-necessary-to-build” should include assets, and may include compilers-and-libraries-used-to-build – storing of results of our build (whether binary or not) is rarely necessary. First, if you did a good job with storing “whatever-is-necessary-to-build”, you can reproduce exactly the same build, so you don’t really need to store the results; second –  the value of binary executable / installer for development is extremely limited.

To Git or not to Git? To Feature-Branch or not to Feature-Branch?

In the most of the development world, for quite a few years Git (or sometimes more-or-less-equivalent Mercurial) is being considered a golden standard for source/version control. On the other hand, recently I feel a completely unexpected comeback of SVN.

Actually, if we take a closer look at the heated Git-vs-SVN debates, the main line of meaningful arguments is not really about specifics of the source control system, but is rather about branching.³

And as most of the modern VCS (including Git, Mercurial, and SVN 1.8+) do provide useable branching⁴ – the actual line of argument is not about “what our VCS will allow us to do” anymore, but is about “what we want to do” with regards to branching, and more generally – with regards to our development process.

As a result, we’ll discuss the development processes first, and only then we’ll proceed towards the discussion of the pros and cons of specific VCS.

Before we start, let’s note one all-important property we should keep in mind while discussing development processes and VCS in the context of game development. First, let’s observe that, as noted above – we have to store all the game assets under our version control system. Second – let’s note that the assets are generally handled by non-developers (such as artists). When these two observations are combined, it means that

“In general, it is not a problem to explain the concept of “check-in” and “check-out” to non-developersIn general, it is not a problem to explain the concept of “check-in” and “check-out” to non-developers; after all – even the most conservative gamedev teams are using version control (such as SVN or Perforce), with “check-in” and “check-out” being the cornerstone of all-the-source-control-systems. On the other hand – requiring non-developers to understand complicated processes such as GitFlow (and Linus forbid, “rebase”) is usually out of question.

Development Processes: from Release Branching and Feature Branching to Trunk-Based-Development-with-Continuous-Integration and Feature-Branching-with-Frequent-Integration

Now, let’s take a closer look at those development processes. Over the time, at least four rather distinct approaches to development were observed in the wild:⁵

• Trunk-Based Development (with Release Branching on the side). Actually, Trunk-Based Development (i.e. “everybody actually works on a trunk, and directly commits to trunk”) is a natural thing when you’re working on the project alone – and it was used for larger projects for a while too. In practice, Trunk-Based Development becomes rather frustrating as soon as you have a team with 100+ developers making commits to trunk, where any single commit can break the whole thing; it has led to infamous situations of “hey, nobody can compile until this <censored> guy has his code fixed” <sad-face />…
  • Pretty often (and especially in waterfall-based development processes), Trunk-Based Development is accompanied with the practice of Release Branching; this practice goes back to times when the software was released with a biennial cycle. From the point of view of source control – it means that whenever we had a release, we needed to have two branches: one to develop bugfixes for this release, and another one – to develop our next release.
• Feature Branching. Historically, Feature Branching was hugely popularized by Linux development processes and Git. Essentially, Git is all about the philosophy of a more extreme form of Feature Branching, where everything out there can be seen as a patch. From our point of view – Feature Branching does work, but has a problem that integrating Feature Branch can easily become a significant effort; moreover, the longer Feature Branch lives without integration – the more difficult it will be to integrate it, up to the point of throwing away the whole development of several month in one of the branches <ouch! />.
• Trunk-Based-Development-with-Continuous-Integration. As the time passed, so-called “Continuous Integration” emerged (more on it in [[TODO]] section below), which has helped Trunk-Based-Development in a very significant way. The most important difference of Trunk-Based-Development-with-Continuous-Integration from simple Trunk-Based-Development is that with the help of Continuous Integration, you won’t be allowed to check in anything which breaks compile-and-a-bunch-of-very-basic-tests. As a result – addition of Continuous Integration mostly solves that everybody-waiting-for-you-to-fix-your-bug-which-made-it-to-trunk problem <phew />, which is typical for “pure” Trunk-Based Development.
• Feature-Branching-with-Frequent-Integration. As noted above, one problem inherent to feature branching, is that the branches can easily become out-of-sync, making merge down the road very difficult (up to the point of being outright impossible). To deal with it – we can adopt a policy of Frequent Integration. With such a policy in place (and enforced by tools – more on them in [[TODO]] section below) we don’t sit in a Feature Branches for long, merging them back to trunk as soon as possible (as in “usually – daily, at the very most – once per week”). Moreover, even if we cannot integrate our feature branch yet – we should at least re-sync changes from the trunk into our feature branch. In turn, such a tool-aided policy mostly solves that problem of the long-living-Feature-Branches being difficult to integrate back to the trunk.⁶

Looking at these four approaches I can say that I clearly do not like the first two of them (though even they can work in practice); as for the two last ones –

In other words – both these workflows will work, and “which one to choose” is more-or-less down to preferences of your team. BTW, if we think of it for a few more seconds – we’ll realize that with adding of the Continuous/Frequent Integration, the original approaches actually converged towards a more-usable-process (with the difference between the two approaches declining as the life time of each Feature Branch is reduced – and reducing the life span of Feature Branch as-much-as-possible is already almost-universally recognized as being a Good Thing™, the only remaining argument is about what “as-much-as-possible” really means).

“at least in theory Trunk-Based-Development-with-Continuous-Integration and Feature-Branching-with-Frequent-Integration can co-exist within the same projectMoreover, at least in theory Trunk-Based-Development-with-Continuous-Integration and Feature-Branching-with-Frequent-Integration can co-exist within the same project (though personally I would still suggest to pick one of them to start with⁸).

Now, armed with this information – let’s discuss Trunk-Based-Development-with-Continuous-Integration and Feature-Branching-with-Frequent-Integration in a bit more detail.

Trunk-Based-Development-with-Continuous-Integration

Very roughly, Trunk-Based-Development-with-Continuous-Integration goes as follows:

• Essentially, all we have is ‘trunk’ branch, where all the development goes
  • All the usual “check-in”/”check-out” logic applies
  • By default, all check-ins/check-outs go directly into ‘trunk’
    • There are exceptions, but they’re rare
      • One such exception is hotfixes – which usually involve “release branch”
    • To avoid those “nobody can work because somebody has checked in one non-compilable file” – we use a Continuous-Integration tool (CI tool)
      • CI tool compiles checked-in code, and runs a bunch of very basic unit tests
      • “to work efficiently – CI tool should work (and give a “green light”) not only on each commit, but before the actual commit happensIMPORTANT: to work efficiently – CI tool should work (and give a “green light”) not only on each commit, but before the actual commit happens. Such “pre-tested commits” are extremely important to reduce the number of that “everybody waits for a committed-bug to be fixed” situation.

Trunk-Based Development, when aided with a CI tool doing pre-tested commits (a.k.a. pre-commit test, etc.), has several important advantages over Feature Branches:

• It is simple. Most importantly – it is simple enough to be used for non-developers (such as artists). If you give your artists Perforce or a TortoiseSVN (without any branches in sight) – chances of them revolting against using version control become pretty slim (but give them Git and ask them to “rebase” to re-sync their branch – and you will certainly have a mortgage-size crisis on your hands).
• As everybody works with already-committed code – it allows for much more testing to be performed over that already-committed stuff; this tends to help with ironing those most-pesky integration bugs out.
• Merges are rare – which means that there is less risk of the need to revert a merge (and reverting a merge is a well-known counter-intuitive mess at least in Git and Mercurial, more on it below).

On the negative side – Trunk-Based-Development-with-Continuous-Integration exhibits a few not-so-desirable properties too:

• It complicates work on not-so-trivial features. As there are no “feature branches” – we have to choose between doing-everything-in-a-very-incremental-manner (which is possible, but time-consuming) – or developing for a while without committing at all (which is risky and prevents inter-developer collaboration on features).
• Cherry-picking (i.e. deciding which features should go into the next build) is pretty much impossible
  • OTOH, “feature flags” a.k.a. “feature toggles” (~=”runtime decision to enable/disable a feature”) allow for rather similar functionality, though with two (admittedly relatively minor) drawbacks
    • While I don’t have problems with enabling and disabling ready-to-be-used features using feature flags – relying on feature flags to make sure that code from half-baked features is never ever used is, well, rather risky (if your system can be misconfigured in production – somebody will do it sooner rather than later). In addition, feature flags also create a risk of such never-used code living in trunk for years.
    • on the Client-Side giving away the code prematurely may conflict with our bot fighting efforts (more on it in Vol. VIII’s chapter on Bot Fighting).
  • “With Trunk-Based-Development-with-Continuous-Integration development model running over several years, code tends to become unnecessarily-tightly-coupled more easily than when using Feature BranchesWith Trunk-Based-Development-with-Continuous-Integration development model running over several years, code tends to become unnecessarily-tightly-coupled more easily than when using Feature Branches. While there is an overall tendency for the unattended code to become entangled (pretty much as unattended tow-lines from epigraph to this section) – with Trunk-Based-Development lacking a concept of “feature”, fighting the spaghetti code tends to require more discipline than for Feature-Branch approach.
    • On the other hand – if you’re using (Re)Actors (which, as I am arguing over this whole book, you should <wink />) – clean inter-(Re)Actor interfaces are enforced in a rather strong manner, so at least between (Re)Actors it is not that much of a problem.

Feature-Branches-with-Frequent-Integration

An alternative to Trunk-Based-Development-with-Continuous-Integration is Feature-Branches-with-Frequent-Integration. As a rule of thumb, it goes as follows:

• All the development is done within Feature Branches, which are then merged into the trunk; normally – no direct commits into trunk are allowed.
• Usually, when/if going a Feature Branch route, I am arguing for the “Git Flow” branching model by Vincent Driessen described in . For the very first time, it may look complicated, but for the time being you’ll just need a few pieces of it:
  • master As a rule of thumb, you should merge here only when milestone/release comes. All the commits to the master branch should come from merges from the develop branch. Direct commits (i.e. commits which are not merges from develop) into master branch SHOULD NOT happen.
  • develop The branch which is expected to work. More precisely – it is usually understood as a branch that compiles and passes all the automated/CI tests. ⁹You should merge to develop branch as soon as you get your feature working “for you” (and all the automated regression tests do pass).
    • As for allowing direct commits (not from feature branches) into develop branch – it is arguable. If we do allow them – we’ll be actually using a hybrid between Feature-Branches-with-Frequent-Integration and Trunk-Based-Development-with-Continuous-Integration (and this is not bad, but it is important to understand implications of each model).
    • Whether we use direct commits or not – for all the commits into develop branch we MUST use that pre-commit Continuous-Integration stuff discussed in the Trunk-Based-Development-with-Continuous-Integration section above. Actually, it becomes even more important for merges (in particular, because reverting merges is a mess for pretty much all the VCS out there <sad-face />).
  • feature You should create your own feature branches as you develop new features. These feature branches should be merged into develop branch as soon as your feature (fix, whatever) is ready. Consider feature branch as your private playground where you’re developing the feature until it is ready to be merged into develop branch. Feature branches are generally not required even to compile; however, it is a really good idea to have an ability to run CI tests on a feature branch – and to use this ability on regular basis.

“there are developers out there who prefer to live within their own feature branch for many weeks and monthsA major word of caution with regards to feature branches: there are developers out there who prefer to live within their own feature branch for many weeks and months, often even implementing many different features under the same branch and postponing integration for as long as they can. This is a Really Bad Practice™, and you SHOULD integrate your Feature Branches very often. Moreover, even if merge of your feature branch into develop branch is not possible – at the very least you should sync your branch with trunk (via “sync merge” in SVN, and via “rebase” in Git) on regular basis (at the very most once per 2-3 days), to incorporate changes-made-in-trunk, back into your Feature Branch.

These policies are all good, but the problem is that for a team of 20+, pretty much any policy doesn’t work without police some way to help us to follow it.

Fortunately, there are at least two phenomena which can help with reducing life time of those way-too-long Feature Branches. First, we have to note that the very nature of merge-based source control systems tends to punish those developers who do their merges later. When both you and a fellow developer are working on your respective branches, and she got committed her merge 5 minutes before you, then it becomes your problem to resolve any conflicts which may arise from the changes both of you have made. In most cases for a reasonably mature codebase, there won’t be too many conflicts, but whenever they do happen,

Print this profound truth in a 144pt font and post it on the wall to help your fellow developers merge their feature branches more frequently.

The second way to deal with those Feature-Branches-which-outlive-their-usefulness, is to have an automated tracker of long-lived feature branches – and to send automated reminders at least to those involved and to PM¹⁰ (and if you really want to get rid of these too-long-branches – send the reminder to the whole team to create additional peer pressure). Automated notifications, while not being a silver bullet, do help to push developers in the right direction.

Pros and cons of Feature-Branches-with-Frequent-Integration over Trunk-Based-Development-with-Continuous-Integration actually mirror respective cons and pros discussed in the Trunk-Based-Development-with-Continuous-Integration section above. On the plus side, Feature-Branches-with-Frequent-Integration:

• streamlines development of the not-so-trivial features
• ensures cleaner code in the long run (in particular, it reduces unnecessary tightly coupling)
• provides an ability to cherry-pick features-to-be-compiled-in
  • This, in turn, is a practical prerequisite to rather-important Replay-Based Testing as discussed in Vol. II’s chapter on (Re)Actors.

On the negative side, this model is not really ideal either:

• It is too complicated for non-developers such as artists.
  • To deal with it – we have to say that:
    • At each given moment, each artist¹¹ works in one specific feature branch.
    • We provide simple scripts¹² checking-out and checking-in into this branch from his working directory
    • This way – from the artist’s perspective, the whole process is the same as with trunk-based development, and is simple enough to deal with without going into the complicated world of branches.
  • While merges are less frequent with Feature-Branch-based development – each merge takes more time (especially if it was hold for too long).
    • See above about “how to push developers to commit/push more often”.
  • “amount of naturally occurring de-facto integration testing of the committed code is reduced (and reducing the amount of testing – especially integration testing – is never a good thing)As commits into develop-branch-used-by-everybody tend to happen less frequently than for Trunk-Based Development – it means that amount of naturally occurring de-facto integration testing of the committed code is reduced (and reducing the amount of testing – especially integration testing¹³ – is never a good thing).
  • Merges are frequent, which raises chances of having a merge revert (and they are way too messy at least in Git and Mercurial).

Comparing Feature-Branches-with-Frequent-Integration and Trunk-Based-Development-with-Continuous-Integration

Now, let’s compare our two candidate workflows side by side; such a comparison can be seen in Table 11.1:

As we can see – pros and cons of these two approaches are pretty well-balanced, so “what to choose” becomes more-or-less a matter of whatever-team-prefers (that is, as long as things such as CI and monitoring of the long-living feature branches are implemented).

Also – it is certainly possible to use a “hybrid” approach with more-trivial features going directly to the trunk, and more-complicated-stuff living in their own Feature Branches for up to a few weeks.[[TODO: elaborate on “hybrid”, where you’ll end up anyway <wink />]]

[[To Be Continued…

This concludes beta Chapter 11(a) from the upcoming book “Development and Deployment of Multiplayer Online Games (from social games to MMOFPS, with social games in between)”.

Stay tuned for beta Chapter 11(b), where we’ll continue our discussion on version control for gamedev (and will raise still-sizzling-hot git-vs-SVN-vs-Mercurial-vs-Perforce question in the context of gamedev)]]

Don't like this post? Comment↯ below. You do?! Please share:

Acknowledgement

Cartoons by Sergey Gordeev from Gordeev Animation Graphics, Prague.