By Jason Snell

January 10, 2017 12:12 PM PT

Consumer Reports battery test uncovers an Apple bug

Note: This story has not been updated for several years.

Consumer Reports withheld its recommendation on the new MacBook Pros based on poor battery tests, and today we know the cause. A bug in Safari, combined with the specific test configurations of the Consumer Reports laptops, were behind the strange results. Consumer Reports says it will re-test and revisit its ratings.

At Macworld we built a lot of different lab tests over the years. It’s hard to test real-world performance in automated tests. You want to produce a result that represents what regular people would experience when using the product, but it’s a constant battle against software and hardware that’s designed to reduce power consumption at every turn. You can’t just use a human to do the testing, because in addition to being wildly inefficient (these tests take a days to perform, per system), they won’t be exactly the same on all the different systems.

In the case of Consumer Reports battery testing, they used a web-browsing test that required the disabling of Safari’s cache in order to simulate multiple page loads. It’s not a configuration any regular user would use, but the intent is to simulate something every user does—loading a bunch of different web pages. They just wanted to do it without having to create hundreds or thousands of different sample web pages on a test server.

The problem with these test scripts is that the farther you take them away from how regular people use their computers, the more you risk your data not being relevant to real-world use cases. Consumer Reports turns off auto-dimming features on laptop displays, for instance—but what if Apple’s auto-dimming algorithm was notably superior to ones found on PC laptops? In that case, the battery savings of that feature would be discarded in the name of testing consistency. Is it worth it? It’s a balancing act. There are judgment calls like that at every turn when you’re building lab tests.

(Apple has its own internal performance testing group, of course. It’s a group that includes several of my former Macworld and MacUser colleagues, and they’re no strangers to how these sorts of tests work.)

Unfortunately, the Consumer Reports test script encountered what Apple called “an obscure and intermittent bug reloading icons which created inconsistent results.” Very few users ever disable the browser cache, which is probably why this bug slipped through.

My guess is that this bug is more likely the cause of the battery-life disparity than anything specifically weird or unfair in Consumer Reports’ laptops tests, but I suppose we’ll see when it revisits its findings.

If you appreciate articles like this one, support us by becoming a Six Colors subscriber. Subscribers get access to an exclusive podcast, members-only stories, and a special community.