PHP Versus ASP.NET Benchmarks Drive Debate
A new but unofficial performance benchmark test by a senior Microsoft developer is spurring debate in the ASP.NET and PHP communities, even as it reveals some surprising findings.
Microsoft began embracing PHP nearly three years ago, when partnered with Zend Technologies, steward of the open source PHP Web scripting language, to optimize Windows Server and IIS for PHP, and vice versa. That commitment deepened when Microsoft released the second CTP of the PHP SDK for the Windows Azure cloud services platform.
Last week Joe Stagner, a senior program manager in Microsoft's Developer Tools and Platforms group published findings of his benchmark tests intended to reveal how well ASP.NET and PHP code run on both Linux and Windows. While the benchmarks remain unsanctioned by Microsoft, Stagner said the findings caught him off guard. Among them:
- ASP.NET was universally faster than PHP (on Windows and on Linux) with the exception of File Copy and Attribute operations.
- MySQL Access from PHP on Linux was slightly faster than SQL Server access on Windows (assuming common data types and SELECT statements).
- ASP.NET (C#) operations, object use, library calls and the like were significantly faster that their PHP equivalents.
- PHP RAW statement execution and function calls were faster on Windows than on Linux
While some critics disputed Stagner's objectivity, he insisted that he took his Microsoft hat off when conducting the tests. "I’m not saying “ASP.NET is faster, so you shouldn't choose PHP!" he wrote in a blog posting that revealed his findings. "I’ve always contended that the affable simplicity of PHP had some drawbacks for certain advanced applications."
He denied criticism that his results may be biased because he works for Microsoft. "Contrary to the couple of inevitable allegations that I’m 'just' a Microsoft shill, I’m doing this because I want to know," he noted.
Cache and Critique
Stagner received a number of comments on his site but Andi Gutmans, Zend's co-founder and CEO, said in an email that Stagner "did his best to remain unbiased… and tried to set up a level playing field." But Gutmans did call out Stagner for neglecting to include an op-code or byte-code cache for PHP.
"This is a significant omission because ASP.NET running on Windows is cached and converted to machine code by the JIT compiler (in the CLR), so similar standard caching like that in ASP.NET truly needs to be added to the PHP environment for parity. There [are] a number of easy-to-use, free, byte-code caches, such as our Zend Optimizer+ that comes included free with Zend Server Community Edition, our free PHP stack," he said in an email. "We encourage the author to rerun his tests and publish the results using Zend Server CE."
Cal Evans, a longtime PHP proponent agreed and pointed to an MSDN article about how the CLR among other things compiles ASP.NET into byte-code and caches it. "A more useful test would be to show the performance of ASP.NET vs. using PHP with Alternative PHP Code APC -- an extension available for several years but part of the core language as of PHP 5.3 -- to show both PHP and ASP.NET with byte-code caches enabled," he added.
Zend's Gutmans argued that "Web workloads not only run fast on PHP, but scale massively." But responding to those critiques, Stagner said this week he re-ran the tests using op-code caching. The result "showed a small improvement in performance, some stuff up to 25 percent faster, but overall it was far less than I expected," he noted in a blog posting today. Stagner said he intends to do further tests. "Though I think the tests that I’ve run so far are fair, they are incomplete and their incompleteness prevents us from seeing the whole story," he noted.
Healthy Dialog
Benchmarks aside, the PHP and ASP.NET communities seem to have a healthy dialog, according to Evans. "Almost everyone in the PHP community is thrilled at the fact that Microsoft has been supporting the efforts of the PHP Windows team," he noted. "Microsoft has been reaching out and I think at every turn we have responded in a positive way. As a PHP developer, having PHP run well on Windows only expands my options. I personally look forward to the day when PHP is a first class citizen on Windows."
Still, Stagner questioned whether the PHP cache issue was a key differentiator. "Can Andi Gutmans make PHP run as fast as ASP.NET? Sure he probably can but that's not the point," he said in the interview.
The purpose of the test was not to provide evidence that Microsoft's technology works better, he said, but rather to provide real data to feed back into the product group. Moreover, he pointed out that PHP's execution performance was about the same on Windows as on Linux. "The bottleneck was MySQL access," he said, singling out the MySQL drivers.
Moving forward, he said the next phase of his testing will consist of "more meaningful tests" that create and render an entire page, rather than stressing core language features like looping and function calls. "I will graduate more to load balancing, in terms of not how fast does this page run, but how fast does a hundred simultaneous requests or 10,000 simultaneous requests for this page load," Stagner said.
Stagner hopes to explore how the two platforms compare in terms of user concurrency, a topic of particular interest to hosting providers.
While Stagner's benchmark campaign has ignited debate, some, like IDC analyst Al Hilwa, question the utility of benchmarks in the day-to-day decision making of dev shops.
"People are rarely in a position to start from scratch and chose new languages," Hilwa said. "Most of the considerations are to do with skill and the existing knowledge in working with an application, and not so much with absolute prowess, whether it be for performance or any particular aspect of a language."