News

Static Code Analyzer Reportedly Finds 10,000 Open Source Bugs

• By David Ramel
• August 18, 2016

A Russian company behind the PVS-Studio static code analyzer claims to have used the tool to discover more than 10,000 bugs in various open source projects, including well-known offerings such as the Firefox Web browser and the Linux kernel.

The PVS-Studio tool checks for errors in C, C++ and C# code.

Company exec Andrey Karpov was upfront in admitting that his firm -- apparently named OOO "Program Verification Systems" -- uses the tool to scan code and find enough errors to justify writing an article to document the findings and publicize the tool.

"The bugs we found demonstrate that nobody is immune from misprints, inattention or other mistakes," Karpov said in a blog post published today. "Absolutely nobody, and we find confirmations to this point in such projects as Microsoft Code Contracts, Qt, Linux kernel, CryEngine, VirtualBox, LibreOffice, Firefox, Boost, Tor and so on. At the moment we inspected 262 projects. It's official! We found and logged 10,000 bugs!"

The company maintains a list of the open source projects it has analyzed that exhibit enough bugs to pen an explanatory article and tracks discovered bugs in a database. The database lists buggy projects by the type of error code generated from detected problems. It also provides links to examine the code of projects where the bugs are found.

For example, one error code is generated by projects with code that contains identical sub-expressions on both the left and right sides of an operator. Investigating further, you can find the actual code snippets exhibiting this error. Here's one:

if ((res->pw==0)||(res->pw==0)) continue;

Here, the logical OR operator (||) compares the same expression, so the function will continue no matter what, as it's basically saying something like the equivalent of "if x = 10 OR if x = 10, continue." [Author's note: As real programmers have pointed out, I flubbed my attempt at explaining the bug that generates this error code: "it will only continue if x = 10."]

OOO "Program Verification Systems" deliberately chose not to provide an option to view all the bugs found in a single project in order to deflect incorrect impressions about the quality of a project or the capabilities of the analyzer, which is continually evolving and may report a different number of errors for a project at different stages of its evolution.

Karpov also downplayed the number of errors reported by the static analyzer.

"Of course, 10,000 issues in 262 projects is not too much," Karpov said. "It makes 38 issues per project at an average. I should notice that indeed this amount does not mean anything. Code base and quality may vary from project to project. For example, in some projects we find just one issue, while other projects contain hundreds of issues."

Karpov said his company typically writes an article when it finds a lot of bugs in a project (documented in the aforementioned list), but if only a small number are found, it just reports them to the project's community contributors.

"Another important point to note is that to promote static analysis and PVS-Studio we do not need to find as many bugs as possible," Karpov said. "We need to find enough interesting issues to write an article. That is why we always suggest contributors of projects to examine their code more carefully. In fact, non-recurrent inspections are good for demonstration of analyzer capabilities, but in real development process they are of very little use. The whole point of the static analysis is to run it on a regular basis. In this case most of errors can be detected during code writing, and not after 50 hours of debugging or after user's complaints."

This month, the analyzed projects that included enough bugs to warrant articles include Inkscape, CryEngine V and NASA World Wind. Previous projects highlighted in articles include Blender, OpenJDK, 7-Zip, Xamarin Forms, PHP and other recognizable names.

"It's not entirely without any reward that we [analyze] all these projects," the company said. The reports we publish serve as an advertisement for our tools and our company. We make no secret of it. But I believe it's the best advertisement you've ever seen! PVS-Studio does help the open source community."