 | Author: | “No Bugs” Hare Follow:   |
| Job Title: | Sarcastic Architect | |
| Hobbies: | Thinking Aloud, Arguing with Managers, Annoying HRs, Calling a Spade a Spade, Keeping Tongue in Cheek | |
Some time ago, I wrote a post about kscope – mini-project on finding and reporting bugs in modern C++ compilers. The point was that there is a way to use C++ facilities to make C++ code self-mutating (more strictly – pseudo-randomized using externally supplied ITHARE_KSCOPE_SEED macro), which allows finding certain classes of bugs during randomized testing. In the previous post, I wrote about 12 bugs reported, with 3 of them already fixed. I have to admit that since that point, I didn’t have as much time as I’d like to spend on kscope; still, 3 more bugs were reported (2 of them being codegen bugs) – and quite a few were fixed (fortunately, bugfixing is one thing which doesn’t require my participation <wink />).
As a result, as of early May, our table of reported-and-fixed bugs looks as follows:
| Clang | GCC | MSVC | |
|---|---|---|---|
| Reported-and-Fixed Bugs (GOOD) |
|
|
|
| NOT to be fixed (DEPENDS) | #36333 (affected only Clang 5, became obsolete since Clang 6 is released) | #195665 (MSFT decided to document it so it is not a bug, but a feature. Very briefly: IF you want your MSVC to be conformant with the standard – you have to DISABLE Edit-and-Continue (mis)feature) | |
| Reported-and-Pending Bugs (BAD) | #47488 (it seems that I run into a new manifestation of the old 7-year-old bug, tried to bump it)
#85605 (missing optimization present in Clang) |
#195483 (reportedly already-known to MSFT internally but not fixed yet)
#196900 (reportedly already-known to MSFT internally but not fixed yet) |
Short observation shows that MSVC has much more bugs detected by this project than GCC and Clang; while exact reasons are of course, unknown, my really wild guess is that’s because codegen in GCC and Clang is already randomly-tested by a comparable-to-kscope (actually, better) tool – CSmith. As CSmith has already reported 79 bugs in GCC and 203 bugs in LLVM (and most of these reported bugs got fixed), it means much fewer chances for kscope to find anything else there. Bottom line:
And of course, as I already wrote, what is really important is not the number of bugs found, but the rate at which bugs are fixed – and in this respect MSVC looks pretty good.
That’s it for today…
Leave a Reply