News

Apple TestFlight Now Does A/B App Testing

• By David Ramel
• April 14, 2017

Apple's updated TestFlight tool now lets iOS developers conduct A/B testing of their mobile apps thanks to new multiple build functionality.

The company says its TestFlight offers "beta testing made simple," letting developers invite up to 2,000 testers to try out their apps before they're released in the App Store.

Up until this week, though, there was no easy way to use it for A/B testing, in which developers create two builds of the same app with different features or design in each and then subject them to user testing to gauge how the different options affect usage based on statistical analysis.

Such A/B testing has been long been a desire of iOS developers, as evidenced by a Quora question ("How do I do A/B testing via TestFlight?") that went unanswered and a 2015 question on the Stack Overflow site: "Apple TestFlight: Is it allowed to upload 2 beta apps for an A/B test?" Although various workarounds were mentioned in the following discussion of the latter question, the short answer was: "You can not test two builds simultaneously."

Until this week, that is.

"TestFlight in iTunes Connect now provides multiple build support, enhanced group capabilities, and improved tester management—making it even easier to test your apps," Apple announced in a post Tuesday.

Regarding the multiple builds functionality, Apple said, "TestFlight now lets you distribute and test multiple builds at the same time, so testers can choose from a number of builds to test."

The company also announced updates for grouping functionality and tester management:

• Groups: TestFlight groups have changed. You can now do more with them, like create groups of TestFlight users, and each group can test a different build. To get you started, we've added all of your existing external testers to the group "External Testers," which you can edit at any time.
• Tester management: Testers can continue testing a build when it goes live on the App Store, minimizing disruptions. iTunes Connect users can also access all active builds, letting them seamlessly compare different builds and you can easily resend invitations to testers who have not yet accepted their invitation.

TestFlight lets developers internally or externally test up to 100 apps at a time, with builds staying active for 90 days after upload. In addition to iOS apps, the tool supports multiple builds of watchOS, tvOS and iMessage apps at the same time.