Choosing Version Control System for MOG development

	Author:	“No Bugs” Hare  Follow:
	Job Title:	Sarcastic Architect
	Hobbies:	Thinking Aloud, Arguing with Managers, Annoying HRs, Calling a Spade a Spade, Keeping Tongue in Cheek

[rabbit_ddmog vol=”3″ chap=”Chapter 11(b) from “beta” Volume III”]

After we finished our (admittedly very limited) discussion on development flows – we can proceed to discussing specific version control systems. As of 2017, the following four version control systems are widely used for game development (listed in historical order of their respective first releases): Perforce, SVN, Git, and Mercurial. While, as noted above, most of gamedev industry is still leaning towards Perforce and SVN – there were successful games using Git and Mercurial too.

Perforce

I have to admit that I never used Perforce myself; still, I’ll try to summarize arguments which gamedevs routinely provide for using Perforce:

• Unlike most of the version control systems – Perforce is oriented not only towards coders, but also towards non-coders such as designers and artists. And I have to agree that providing designers/artists with a friendly environment is indeed extremely important.
• “Huge projects (those with lots of asset binary files, totaling terabytes) are handled without issues.Huge projects (those with lots of asset binary files, totaling terabytes) are handled without issues.
• Locking files is possible.
  • As asset files (whether they’re binary or text – more on it below) are usually not really mergeable – having two artists to work on the same file is a Bad Idea™. This is where universally-frown-upon-in-programmers-world “lock file” feature comes handy.

On the minus side:

• Perforce branching is reported to be rather ugly (up to the point of being outright unusable); even worse – data loss has been reported to happen during Perforce merges <double-ouch! />. On the other hand, to be fair – most of the Perforce users have reported working with Perforce for many years without problems, though it is unclear how much branching they were using.
  • [[TODO]]: Using Perforce streams instead of branches [https://www.perforce.com/perforce/doc.current/manuals/p4v/streams_overview.html]
• Perforce keeps a track of your working copy on the server; while not a problem for LAN – it is a problem when you have to work remotely (which is more and more often these days)
  • While working offline is possible with Perforce, it is subject to “reconciliation” process when you’re back online, which is well, ugly.
• Perforce has been reported to require to resort to out-of-the-source-control file copying and/or sharing (which is an inherently Bad Thing™) on quite a few occasions.
• Continuous Integration tools are relatively reluctant in supporting Perforce; on the other hand, with Jenkins, TeamCity and Bamboo supporting Perforce – it is not that bad either.
• “Locking files” feature can be abused (in particular, you should have a policy of “not using exclusive checkout” for the code).
• You cannot just delete a file in your working copy – you should do it ONLY via Perforce client; otherwise – you’re in quite a bit of trouble. It is not that big deal – but certainly it is an inconvenience.
• At hundreds-of-dollars-per-user – pricing can get not-so-insignificant (especially if you have part-time users).

SVN

I have to admit that for a long while, I have been a fan of SVN – and I still admire it. From a technical standpoint, SVN is your typical centralized version control system (based on a single centralized server), and it is great because:

• The model is simple
• It is easily usable by non-developers.
  • “For non-developers on Windows Tortoise SVN rulezzz!For non-developers on Windows (and large chunk of your designers will be on Windows) Tortoise SVN rulezzz!
• Handles large multi-terabyte projects well.
• File locking is available.
• Offline work is possible and easy (though offline commits aren’t possible, and neither is offline access to history beyond one last version)
• IMO, SVN sync merge is more intuitive than Git’s rebase (though I admit that this point is debatable and flame-war ridden).
• SVN is built under a strong perception of history being immutable. While it is possible to mess with SVN history, it is difficult (or even impossible?) to do without messing with SVN files directly (i.e. without having admin-level access to svn server box).
• [[TODO: partial checkouts]]
• Path-based access control is possible, including restricting reads on a per-directory/per-file basis. [[TODO: refer to explanation why it is necessary]]

On the minus side:

• While merges reportedly improved on the way towards SVN 1.8 or so – they’re still not as fluid as in Git. At the very least – as far as I know, you still SHOULD avoid renaming files in your branches (otherwise – chances are you’ll get an infamous “tree conflict” <sad-face />).¹ It is not that big deal – but a significant inconvenience if doing feature branching.
• Commits while you’re offline are not possible. This is not as bad as with Perforce (actually, if you’re offline just while you’re typing in while on a train back home – it is not noticeable at all), but if you’re going to be offline for a while as you are developing² – it can become a problem.
• As with Perforce, locking can be abused. To mitigate it, it is possible to:
  • Outright prohibits locking of source files (IIRC, pre-commit script should do it).
  • for non-mergeable files – make sure to write a script sending reminders (CC: PM) such as “you’re holding this file since yesterday – are you sure you really need it for this long?”)
• For an open-source project – SVN’s model doesn’t lend itself well to “pull requests”
  • OTOH, I didn’t see much “pull requests” for intra-company development, even less for gamedev.

Git

After working with SVN for a while, I had to switch to Git – and found it being clearly better-suited for Feature-Branch development model; moreover – Git is also perfectly usable for Trunk-Based-Development – as long as there are only developers who work on the repository. On the other hand, for gamedev-with-assets-and-artists-involved – Git, while being usable, is clearly not the best option.

Pros of Git include:

• Being branch-centered from the very beginning, branch handling in Git is good.
  • Still, reverting branch merge is ugly even in Git <sad-face />, more on it in [[TODO]] section below.
• Offline work is very straightforward, you have full capabilities of commiting and having your history.
  • Of course, it comes at the cost of extra push operation, so if you don’t work offline often – it is not that big deal (especially in 2017, where you have Internet pretty much all the time) 

List of Git negatives, at least when it is used for game development, is longer:

• “Git is not really friendly to non-developers such as artists (that’s to put it very mildly)Git is not really friendly to non-developers such as artists (that’s to put it very mildly). The whole model is significantly more convoluted than that of centralized version control systems such as Perforce or SVN, and without ability to merge those-files-artists-are-working-on – it becomes convoluted-for-no-apparent-reason for their purposes <sad-face />.
• Whatever-you’re-doing, you have to keep the whole repository on your local box (except for Git-LFS files, more on them below); if the whole-project-including-history is large (as in “1T large”) – it can take a looong while to download it.
• As a result, some developers started to support multiple Git repos – one for “lean and mean” code, and another one for docs etc. TBH, I do not like the very idea of having several repos (it starts a slippery road towards “let’s keep all the code in Git, and all the assets in SVN” – which usually qualifies as a Really Bad Idea™ because of lack of sync between two repositories, as discussed above).
• Git-LFS is a kind of crutch (and is not really following the “distributed” nature of the rest of Git).
• Handling of huge-projects-with-lots-of-binary-files is rather ugly with Git. While it did improve with Git Large File Storage (Git-LFS) – it is still not clear how Git-LFS behaves for multi-terabyte real-world projects.
• File locking for non-mergeable files is not available (see below on advisory locks – but they don’t really work well for non-developers).
• Per-file access control is not supported (at least not out-of-the-box). While this is not that much of a problem for open source projects – it is quite an issue, especially for gamedev where we have to resort to security-by-obscurity <sad-face />.
• I positively hate an ability to mess up with (“amend” in Git-speak) already-committed data. IMNSHO, having history immutable is a Really Good Thing™ for any version control system.

For a more detailed discussion on problems-of-Git-for-gamedev-purposes – see, for example, [chris@enemyhideout].

Git and unmergeable files

“For game development (and unlike most of other software development projects), you're likely to have binary files which need to be edited (representing so-called “assets”)For game development (and unlike most of other software development projects), you’re likely to have binary files which need to be edited (representing so-called “assets”; more on assets and asset pipeline in Vol. V’s chapter on Graphics 101). More precisely, it is not only about binary files, but also includes any file which cannot be effectively merged by Git (even simple text-based Wavefront .obj file is not really mergeable in a sense that tracking differences in these files is pretty much useless).

A question “what to do with such files” is not really addressed by Git philosophy. The best way would be to learn how to merge these unmergeable files, but this is so much work that doing it for all the files your artists and game designers need, is hardly realistic <sad-face />; still – make sure that if you’re using Unity, you’re using their SmartMerge (that’s regardless of using Git or not).

The second best option would be to have a ‘lock’ so that only one person really works with the asset file at any given time. However, while locks are supported by Perforce and SVN (and there is a Lock Extension for Mercurial too) – Git’s position with regards of locks is (a) that there won’t be mandatory locks, ever, and (b) that advisory locks are just a way of communication so that should be done out-of-Git <sic! />. The latter statement leads to having chat channels or mailing lists just for the purposes of locking <ouch! />. I strongly disagree with such approaches, because IMNSHO:

To use advisory (non-enforced) locks in Git, I suggest to avoid stuff such as chat channels, and to use manually-editable lock files (located within Git, right near real files) instead. Such a lock file MUST contain the name (id) of the person who locked it, as a part of file contents (i.e. having lock file with just “locked” in it is not sufficient for several reasons). Such an approach does allow to have a strict way of dealing with the unmergeable files (that is, if people who’re working with it, are careful enough to update – and push(!) – lock the file before starting to work with the unmergeable file), and also it doesn’t require any 3rd-party tools (such as an IM or Skype) to work.

For artists/game designers, at the very least this logic must be wrapped into a “I want to work with this file – lock it for me” script (with the script doing all the legwork and saying “Done” or “Sorry, it’s already locked by such-and-such user”).³ And if you like your artists better than that, you can make a Windows shell extension which calls this script for them and displays a nice “Locked” icon.

The approach of lock-files described above is known to work (though having a drawback of creating commits just for locking purposes), but still remains quite a substantial headache. Actually, the headache can be so significant that it might be better to use Perforce, SVN, or Mercurial-with-Lock-Extension (all of them support mandatory locking) just for this reason.

Let’s also note that there is also an issue which is often mentioned in this context, the one about storing large files in Git, but IMO this is a much more minor problem, which can be mostly resolved by using Git LFS plugin.

Issues with reverting Git branch merge commit

One of Git peculiarities is related to revert of Git commit of merging branches. While revert of committed branch merge is not a picnic in any version control system, in Git it is IMO particularly nasty and counter-intuitive.

For a proper discussion of it – take a look at [TorvaldsHamano]; here I’ll provide only a very brief overview. In short – after reverting Git committed branch merge, your system is left in not exactly the same state than it was before the merge(!) – so you need to remember about this reverted merge when you’re doing re-merge, otherwise you’ll get very unexpected results <sad-face />. In practice, it means that with Git you generally should avoid reverting branch merges at all costs. To make things worse – in Git-world this behavior is not considered a bug-which-has-to-be-eventually-fixed (but rather a feature-which-makes-those-who-know-about-it-gurus), so chances of it being fixed are estimated about as high as chances of a cold day in hell <sad-face />.

Mercurial

Last but not least on the list of our contenders-for-version-control, is Mercurial. While Mercurial is ideologically very similar to Git, it certainly has a very different look and feel. In particular, the following pros can be observed about Mercurial in the context of gamedev:

• Mercurial is (almost-)usable by non-developers
  • TortoiseHG helps a lot for those poor Windows-based souls.
• Branching is good (though reverting branch merge is still a mess <sad-face />)
• Offline work is very straightforward too
• Commits are not mutable⁴
• Lock Extension is available (though not distributed with Mercurial by default)
• Per-file access control is supported (though not for reading)

List of Mercurial cons is also impressive:

• “Large files (beyond 100M or so) are handled very inefficientlyLarge files (beyond 100M or so) are handled very inefficiently; Mercurial needs about file-size-multiplied-by-5x-to-10x RAM to operate, so having a 1G asset file is likely to bring quite a few systems to their knees <sad-face />.
• Same as with Git, pretty-much-whatever-you’re-doing (except for large files), you need to have the whole repository on your local box.
• Same as with Git, unmergeable files are a Big Headache(tm).
• Just as with Git-LFS, Mercurial Large Files Extension is a crutch, going against its overall distributed nature.
• Access control restricting reading within repository is not possible <sad-face />.⁵

From what I’ve heard, one big reason why gamedevs are not using Mercurial, is because of that issues with large asset files (see, for example, [SirGru]).

On Open-Source Gamedev

If by any (admittedly rather slim) chance you’re planning to release an open-source game – another extremely-important factor is added into play: namely, “how many people you’ll be able to attract to work on your open-source project?” And in this regard, GitHub is a very clear leader by far, with BitBucket and GitLab fighting for distant second place.

Now, we should observe that all these three services are running Git (only BitBucket providing Mercurial option on the side). Moreover, all the competition-running-SVN such as OSDN and Assembla are lagging far far behind these three (as for Sourceforge – it is no longer recommended due to certain really ugly decisions they made a few years ago ).

This means that

Comparison of Four Major Version Control Systems for Gamedev Purposes

Now, we can summarize our discussion about different version control systems in the context of game development, in the following Table 11.2:

As we can see – unlike for generic software development (where Git still arguably rulezz), for gamedev we have to say that non-distributed version control systems (such as Perforce and SVN) tend to be a more logical choice than DVCS such as Git and Mercurial. When choosing between Perforce and SVN – I’d prefer SVN, but I have to admit that if you’re heading for trunk-based development – Perforce becomes perfectly competitive too.

On the other hand, if your game is going to be open-source – Git (or at least Mercurial) gets an all-important-for-open-source advantage of additional exposure.

Version Control: 3^rd-party Hosting vs In-House

“while sometimes outsourcing is indeed a good idea, some other times it doesn’t really workIn XXI century, overall trend is to have more and more services outsourced; however – while sometimes outsourcing is indeed a good idea, some other times it doesn’t really work. When it comes to outsourcing version control for a game, keep in mind the following pros and cons of such outsourcing (also known as “cloud-based version control”, “SaaS”, etc.):

• Pro: less headaches, plain and simple. With an in-house version control, you need to spend time on configuring it, backing it up, and storing backups safely. It is not that much work – but somebody has to do it if you keep your system in-house.
• Pro: upgrades happen automagically, so you don’t need to spend time on them
• Pro: unless you have a serious admin which handles it – it is less difficult to mess up your version control system.
  • BTW, I’d say that for Git requirements for your admin are higher than for other systems (in other words, Git is substantially easier to mismanage – in particular, due to the mutable histories <ouch! />).
• Con: for a game with lots of assets, and with 3^rd-party hosting – you can be for a looooong wait for each checkout. Even more so if you’re using Git or Mercurial (and if you’re not careful enough to keep all your assets within Git-LFS or Mercurial LargeFiles – it can easily become catastrophic).
• Con: upgrades happen automagically, so you can’t schedule them (so if a problem occurs affecting your system – it will happen at the worst possible time, like “on the day of the release”). Granted, it is rarely a problem for hosted version control – but is quite a problem for other 3^rd-party systems such as Issue Tracking.
• Con: While 3^rd-party hosts such as GitLab or BitBucket don’t have a reason to steal your code – by their nature they are extremely juicy attack targets. And as we as gamedevs have to resort to “security by obscurity” much more often than we’d like to (more on it in Vol. VIII) – the damage from some-hacker-cracking-into-GitLab-or-BitBucket-and-publishing-all-the-source-code-found can be enormous.

Overall, when it comes to version control, it is not that much difference between 3^rd-party hosting and in-house system (~=”you won’t do too wrong choosing any of these routes”). BTW, if you happen to like UI of GitLab or BitBucket but are not fond of 3^rd-party hosting for any of the reasons mentioned above – keep in mind that you can have them installed in-house (a.k.a. “self hosted”) too.

[[TODO: cherry picking (in particular – for replay-based testing as discussed in Vol. II’s Chapter on (Re)Actors)]]

[[To Be Continued…

This concludes beta Chapter 11(b) from the upcoming book “Development and Deployment of Multiplayer Online Games (from social games to MMOFPS, with social games in between)”.

Stay tuned for further parts of Chapter 27, where we’ll continue our discussion on DB optimizations (going into the strange field of app-level caches and app-level replicas)]]

Don't like this post? Comment↯ below. You do?! Please share:

Acknowledgement

Cartoons by Sergey Gordeev from Gordeev Animation Graphics, Prague.