How to Report C++ Compiler Bugs

	Author:	“No Bugs” Hare  Follow:
	Job Title:	Sarcastic Architect
	Hobbies:	Thinking Aloud, Arguing with Managers, Annoying HRs, Calling a Spade a Spade, Keeping Tongue in Cheek

Disclaimer: please do NOT expect any revelations in this post; it is merely a summary of very-obvious and a-bit-less-obvious observations (which still might elude most of us=”C++ developers who run into a compiler bug once or twice in a lifetime”).

In addition to my previous post on C++17 compiler bug hunt attracting quite a bit of attention, it has caused a reasonable question on “how to get your bug not only reported but also fixed”. As a result, I think it is a good idea to state my findings so far with regards to reporting compiler bugs (no warranties of any kind, batteries not included).

Very Obvious

First, let’s see some very obvious requirements for compiler bug reports.

Follow Respective Instructions

First of all, we have to follow respective instructions for reporting; these instructions can be found at:

• for Clang: https://llvm.org/docs/HowToSubmitABug.html
• for GCC: https://gcc.gnu.org/bugs/
• for MSVC: https://docs.microsoft.com/en-us/cpp/how-to-report-a-problem-with-the-visual-cpp-toolset

There is nothing to add here – we have to read and comply (well, to the extent possible <wink />).

General Observations

A few general observations about reporting bugs:

• “the bug MUST be reproduciblethe bug MUST be reproducible (otherwise, there is no real way to handle it). As a result:
  • make sure to specify the version of your compiler in your bug report
  • make sure to specify command line which causes the trouble
  • unless your bug is reproducible with a “preprocessed source” (more on it below) – make sure to provide ALL the dependencies.
    • in such cases, make sure to unzip your planned-submission at least to a completely-separate folder (better – to a box-with-only-bare-compiler-installed) – and run your command line there to see whether the bug does reproduce.
  • bugs related to multithreading in the compiler, are pretty much hopeless <sad-face />. Fortunately, as of now, over three-major-compilers, I am aware of only one-such-bug-suspect (which seems to be mitigated by specifying /cgthreads1 to cl).
• Different bugs require different “repros”. For the alleged bugs in front-end compiler (such as compiler crashes a.k.a. ICEs, refusing to compile what-we-think-should-compile, etc.) usually it is preprocessed .i file which is most convenient (.i can be obtained via -save-temps option for GCC/Clang, or via /P for cl¹) . As for other types of bugs – what exactly to provide in the report, depends (however, it still MUST be reproducible).
• There are some popular non-bugs, so make sure to check if “your” bug falls into this category; for a list – make sure to read GCC Non-Bugs even if you’re using a different compiler.
  • Going beyond GCC list, one bad example of a reported non-bug can be found here: https://developercommunity.visualstudio.com/content/problem/167958/just-insert-a-useless-line-is-right-in-vs2017.html. Make sure NOT to file bug reports such as this one.
• make sure to look for potential duplicates (in addition to those 14 bugs I have reported to date, I’ve run into a few situations which apparently were duplicates – and some weren’t compiler bugs, but my own misreading of the standard (yes, **it happens)).
• bug report SHOULD be as-minimal-as-possible. On the other hand, minimizing a bug report is a separate art, which is not really easy to get right.
  • “however, even if you cannot get your problem minimized - IMO it is still better to submit ithowever, even if you cannot get your problem minimized – IMO it is still better to submit it (as long as it is reproducible, is not a well-known non-bug, and is not an obvious duplicate). While it is more headache for the compiler team to handle bugs with non-minimized reports – they often can do it (especially if the bug looks bad enough); BTW, special thanks to Dimitry Andric from Clang team who did minimization for one of kscope-reported Clang bugs.
  • There are certain tools out there to help with minimization, in particular:
    • C-Reduce and/or delta – useful for bugs such as compiler front-end crashes
    • bugpoint (LLVM/Clang-specific, for reducing test cases which go beyond front-end)
      • I am not aware of similar tools for GCC or MSVC <sad-face /> (if you do – please let me know).

A Bit Less Obvious

Going beyond very obvious stuff above, here are some observations which might or might not be correct, but which seem to stand when dealing with compiler writers:

• regressions are MUCH more easily fixed than new-features-not-working
  • hint: if you can demonstrate it did compile ok under the previous version of the same compiler – it can help A LOT to get it fixed sooner rather than later.
• with very few exceptions, nobody cares to fix anything-but-the-very-latest-and-greatest compiler. Even a hotfix for currently-released one is pretty much hopeless if the top-of-the-trunk doesn’t seem to be affected.
  • hint: whenever possible, try to compile your code with top-of-the-trunk, last preview, or at least with the latest greatest official release. Otherwise – it is likely to be ignored (especially if symptoms you’re describing, are not exactly the same with the latest greatest version).
• problems arising from invalid code tend to get MUCH less attention than those resulting from valid one (especially with MSVC).
  • Even ICEs coming from invalid code don’t have good chances with MSVC (which is not really good, but in practice is not that bad as the other bugs out there).
• “making your bug report correct from the very beginning is importantMaking your bug report correct from the very beginning is important; see, for example, the discussion on [/r/cpp/7zd4jz] for a horror story on how a bug got ignored because of typing “debugger” instead of “destructor” when submitting it.
• Last but not least, two things which wouldn’t be necessary in an ideal world, but which DO play their role in our imperfect one:
  • Make sure to build your reputation with compiler writers, it IS important <sad-face />.
    • In particular, make sure that your first reported bug for a specific compiler is Damn Solid(tm).
  • Apparently, raising the hell about your bug can help (just as one example – for MSVC raising the hell on /r/cpp did work at least once, see [/r/cpp/7zd4jz]).

Now, let’s briefly discuss some of popular types of bugs in compilers:

• assertions/crashes within compilers (a.k.a. ICEs)
  • these are getting fixed relatively easily (there is no need to prove it is a bug). One exception is when our compiler is MSVC, and the code involved is more-or-less obviously invalid.
• problems with compiler refusing valid code.
  • These can be MUCH more difficult to get through, AND require quite a bit of the explanation (we have to convince compiler writer that it is a NON-COMPLIANT behaviour – and it can be very difficult, as quite a few of them – especially GCC – tend to have their own understanding of the language, which is not necessarily coherent with that of developers-using-the-compiler).
    • hint: try to compile your code under all three major compilers; if it does compile under two of them – it might help to prevent your bug from being thrown away outright
      • BTW, IF you managed to minimize your problem – Compiler Explorer allows you to test how-your-code-compiles under different compilers easily. NB: as of now, as far as I know, Compiler Explorer doesn’t allow to run your compiled code, but for minimized front-end bugs it is pretty good, though it uses a relatively old MSVC 19.10 – and currently-existing MSVC 19.12 is quite different from Compiler-Explorer’s 19.10 <sad-face /> (heck, 19.10 doesn’t even support /std:c++17 option).
    • hint: referring to a specific section of the C++ standard (with a brief explanation why it should apply to your code) MIGHT help too.
    • hint: unless the bug is perfectly obvious, it makes perfect sense to discuss it with community (my first choice would be StackOverflow, with c++, language-lawyer, and standards-compliance tags, asking along the lines of “does this code conforms to the standard?”). This discussion can also be a good link to include into your bug report.
• problems with code generation.
  • These are the most difficult to get through. First, each and every compiler writer knows for sure that their compiler codegen is bug-free. Second, with all the excuses granted by the Undefined Behaviour (UB) clauses in the standard, it is very difficult to demonstrate that it isn’t your program which has the bug (and TBH, 99.999% of the time crashes-with-optimizations-on are our problems, not compiler’s).
    • Strictly speaking, anything related to Undefined Behavior (a.k.a. UB) is our bug, not compiler’s one.
      • “did you know that in a case of a signed integer overflow, compiler is allowed to generate code which formats your hard drive?did you know that in a case of a  signed integer overflow, compiler is allowed to generate code which formats your hard drive?
        • if you think it is crazy – I wholeheartedly agree; however, it is not a question for compiler writers, but the one for WG21.
        • to detect some of UBs in runtime – before reporting a codegen bug in GCC or Clang, make sure to compile your program with -fsanitize=undefined and/or -ftrapv and to run it afterwards.
      • in practice, the most annoying bugs in our programs tend to be related to “strict aliasing rule”; if we re-compile our program with -fno-strict-aliasing and it suddenly starts working as a result – much more often than not, it is our bug, not compiler’s one².
  • At least the following seems to stand:
    IF there are no UBs in your “pure” program³ – then results of it being run, should NOT depend on optimizations
    • it means that IF you can demonstrate that you have no UBs in your “pure” program, AND that behavior of the program depends on the /O flags – you might get a chance to convince compiler writers that it is their bug, not yours.
      • Still, it is going to be an uphill battle (mostly thanks to all those non-bug reports cluttering the space of the compilerdev teams).
        • In this regard, IF you can have your test case minimized (which can be really difficult for codegen stuff <sad-face />) – asking community whether-your-code-really-has-no-UBs (see above re. Stack Overflow) – has a potential to help.

Summary

While it takes a lot of time to make our bug reports convincing, I can assure you, it is MUCH worse on the compiler-team side. First, they’re receiving a hundred of bug reports every day, with only a very few of them making any sense. Second, they have to minimize that bug – which is not a trivial exercise at all. Third, they have to find why this particular combination of operators causes trouble (even minimal repros can be large, especially for codegen bugs). And last but certainly not least, they have to fix that specific bug – without introducing any new ones (and given the enormous complexity of modern C++ compilers, it is often very non-trivial).

Bottom line:

Don't like this post? Comment↯ below. You do?! Please share:

Acknowledgement

Cartoons by Sergey Gordeev from Gordeev Animation Graphics, Prague.