Quick Update on C++ Compiler Bug Hunt (overall, 15 bugs reported, 8 already fixed)

 
Author:  Follow: TwitterFacebook
Job Title:Sarcastic Architect
Hobbies:Thinking Aloud, Arguing with Managers, Annoying HRs,
Calling a Spade a Spade, Keeping Tongue in Cheek
 
 

Some time ago, I wrote a post about kscope – mini-project on finding and reporting bugs in modern C++ compilers. The point was that there is a way to use C++ facilities to make C++ code self-mutating (more strictly – pseudo-randomized using externally supplied ITHARE_KSCOPE_SEED macro), which allows finding certain classes of bugs during randomized testing. In the previous post, I wrote about 12 bugs reported, with 3 of them already fixed. I have to admit that since that point, I didn’t have as much time as I’d like to spend on kscope; still, 3 more bugs were reported (2 of them being codegen bugs) – and quite a few were fixed (fortunately, bugfixing is one thing which doesn’t require my participation <wink />).

As a result, as of early May, our table of reported-and-fixed bugs looks as follows:

Clang GCC MSVC
Reported-and-Fixed Bugs (GOOD) #36055 (fixed in trunk in 2 weeks after my report, fix reportedly scheduled to apply to Clang 6.0; THANKS to everybody who contributed to the fix!)

 

#84401 (an enhancement, don’t take it too seriously; reportedly fixed; THANKS to everybody who contributed to the fix!)

#84463 (reportedly fixed for GCC8+; THANKS to everybody who contributed to the fix!)

 

#195484 (reportedly fixed in a week after my report, fix scheduled to apply to VS2017 15.7 Preview 2; THANKS to everybody who contributed to the fix!)

#195579 (reportedly, was fixed before my report in a recent preview)

#196885 (reportedly fixed in 3 days(!) after my report, fix scheduled to apply to VS2017 15.7 Preview 2; THANKS to everybody who contributed to the fix!)

#202584 (reportedly fixed in VS2017 15.7 Preview 4; THANKS to everybody who contributed to the fix!)

#202598 (reportedly fixed in VS2017 15.7 Preview 3; THANKS to everybody who contributed to the fix!)

NOT to be fixed (DEPENDS) #36333 (affected only Clang 5, became obsolete since Clang 6 is released) #195665 (MSFT decided to document it so it is not a bug, but a feature. Very briefly: IF you want your MSVC to be conformant with the standard – you have to DISABLE Edit-and-Continue (mis)feature)
Reported-and-Pending Bugs (BAD) #47488  (it seems that I run into a new manifestation of the old 7-year-old bug, tried to bump it)

#85605 (missing optimization present in Clang)

#195483 (reportedly already-known to MSFT internally but not fixed yet)

#196900 (reportedly already-known to MSFT internally but not fixed yet)

#199554

 

Short observation shows that MSVC has much more bugs detected by this project than GCC and Clang; while exact reasons are of course, unknown, my really wild guess is that’s because codegen in GCC and Clang is already randomly-tested by a comparable-to-kscope (actually, better) tool – CSmith. As CSmith has already reported 79 bugs in GCC and 203 bugs in LLVM (and most of these reported bugs got fixed), it means much fewer chances for kscope to find anything else there. Bottom line:

Automated randomized testing rulezzzz!!!

And of course, as I already wrote, what is really important is not the number of bugs found, but the rate at which bugs are fixed – and in this respect MSVC looks pretty good.

That’s it for today…

 

Join our mailing list:

Leave a Reply

Your email address will not be published. Required fields are marked *