7 Top Device Clouds for Mobile App Testing

Testing mobile apps can be a pain, especially with the wide range of OS versions in the Android ecosystem and myriad devices with different screen sizes, resolutions, capabilities and so on -- while iOS developers, of course, enjoy a far narrower scope of testing requirements.

Although device emulators are available for single-device manual testing, such as the Android Virtual Device (AVD) built into Android Studio, and the popular third-party option for Android, GenyMotion, they don't allow for automated testing on a variety of real devices.

Such testing is made easier with the cloud computing phenomenon, allowing access to large device clouds or farms featuring hundreds or even thousands of real devices, along with emulators and simulators. Mobile developers don't have to worry about the impractical proposition of rounding up their own devices for testing.

Mobile app device clouds come in all shapes and sizes, with different pricing models, target audiences, feature sets and use cases. The landscape -- or rather, cloudscape -- can be confusing, so we've summarized some of the popular options for you. Read on to see what solution fits your needs.

• Company's Service Description: AWS Device Farm is an app testing service that lets you test and interact with your Android, iOS, and Web apps on many devices at once, or reproduce issues on a device in real time. View video, screenshots, logs, and performance data to pinpoint and fix issues before shipping your app.
• Pricing: Three levels are offered: pay as you go, $0.17/device minute, with first 1,000 minutes free; unlimited testing, starts at $250/month; private devices (test on dedicated devices deployed exclusively for your account), starts at $200/month.
• Available Devices: 206 Android and Fire OS devices; 152 iOS devices. An interactive device list is available.
• Notes: Two primary testing methods are provided -- automated testing on a variety of frameworks and remote access to devices on which developers can load, run and interactively test Android apps in real time. It tests native and hybrid apps, including apps created with PhoneGap, Titanium, Xamarin, Unity and other frameworks. If developers don't have their own tests, a built-in fuzz test is available. Tests are run on real, non-rooted devices, a mixture of OEM and carrier-branded offerings.

Firebase Test Lab for Android
Formerly called the Google Cloud Test Lab, this has been integrated into the Firebase development platform that also offers a variety of other back-end services to support mobile app development.

• Company's Service Description: Firebase Test Lab for Android provides cloud-based infrastructure for testing Android apps. With one operation, you can initiate testing of your app across a wide variety of devices and device configurations. Test results -- including logs, videos, and screenshots -- are made available in your project in the Firebase console. Even if you haven't written any test code for your app, Test Lab can exercise your app automatically, looking for crashes.
• Pricing: Firebase comes in three plans: Spark, for free, with limitations; Flame, $25 per month; and Blaze, a pay-as-you-go model.
• Available Devices: The list of available Android devices is available through a command-line tool.
• Notes: It can use developer-provided instrumentation tests or its built-in Robo test that analyzes the structure of an app's UI and then explores it by automatically simulating user activities. It's integrated with Android Studio, the Firebase console and the gcloud command line. Developers can also use it with Continuous Integration (CI) systems. It provides real production devices that are flashed with updated Android API levels or locale settings that developers specify so they can road-test apps against a real-world collection of real devices and device configurations.

Xamarin Test Cloud
Acquired by Microsoft last year, Xamarin enables cross-platform app development using the C# programming language.

• Company's Service Description: Find bugs before you ship with automated tests running on the devices your customers use. You can test everything your users can do, from pinching and swiping to double-tapping and scrolling. Find performance problems on any OS automatically with step-by step memory and performance tracking. Beautiful reports make it easy to know when you're ready for launch.
• Pricing: Plans with varying levels of limitations and capabilities include: Newbie, $99/month; Small Startup, $379/month; Growing Business, $799/month; and Enterprise, contact for a quote. A 30-day trial is available.
• Available Devices: It currently lists 2,624 devices, running iOS 8 through iOS 10 and many Android versions from Honeycomb to Jelly Bean.
• Notes: Developers can write tests in C#, Ruby with Cucumber, or Java with Appium. Xamarin Studio and Visual Studio can be used for C# tests, and Xamarin Test Cloud templates are integrated into those IDEs. It can integrate with standard NUnit test runners. Code can be shared for cross-platform tests between iOS and Android. For automated tests, Xamarin engineers can provide hands-on coaching to get projects started. Functionality includes changing GPS locations, pressing physical buttons, activating device cameras and rotating the device between portrait and landscape modes.

Kobiton
While some device clouds are provided only from within more full-featured app development and testing platforms, this offering from Atlanta-based startup Kobiton Inc. is all about device testing.

• Company's Service Description: Kobiton is a mobile cloud platform that enables users to perform manual or automated testing on physical iOS and Android devices.
• Pricing: Three plans are available: Indie, $10/month or $100 annually; Team, $199/month or $2,000 annually; Growing Biz, $499/month or $5,000 annually. A 15-day trial for all is available.
• Available Devices: 160 devices are available in several families, including iPhone7, Google Pixel, Galaxy S7 Edge and iPads.
• Notes: It supports Web automation or manual testing of Web, native and hybrid apps. All devices are real, no emulators. Devices can't be used to make phone calls or send SMS messages, but device cameras can be used. Application files up to 500 MB are supported. Automatically created activity logs provide detailed reports and screenshots of issues in real time so developers can release apps sooner. Developers can manage and access internal devices anytime, anywhere.

Perfecto
Perfecto Mobile Ltd. provides device testing in several ways, including its Continuous Quality Lab. The company offers manual testing, test automation, mobile performance testing and real-device monitoring.

• Company's Service Description: Our Continuous Quality Lab helps you easily mobilize your brand by perfecting the digital experiences that define it. Our 14 highly-secure data centers are home to thousands of real devices that are accessed from any browser -- simply leave the device management to us. Perfecto has saved our customers over $156M each year through the elimination of manual testing.
• Pricing: Three plans include: Basic, $99/month/user or $1,188 per year/billed annually; Advanced, $299/month/user or $3,588 per year/billed annually; and Enterprise, call for quote. A two-hour free trial is available.
• Available Devices: More than 10,000 smartphones, tablets and phablets.
• Notes: All devices are real, no emulators. It offers "DevTunnels" to debug Android/iOS/Web mobile apps on real devices under realistic conditions in order to speed the process of validating new features and fixing bugs that only show up under specific environments or conditions. Developers can execute tests written in Espresso, XCTest, Selenium or other test frameworks on multiple platform versions and form factors directly from their workstations. It provides extensions for Appium and Visual Studio, a lab for Eclipse, a Jenkins plugin, an Android Studio plugin and more.

Sauce Labs
At Sauce Labs, mobile testing is just one of a host of services, including an open source automated testing platform, cross-browser testing, JavaScript unit testing, professional services and more.

• Company's Service Description: Test your mobile apps across iOS and Android real devices, emulators and simulators on our public cloud or on dedicated devices in a private cloud. Run automated tests to accelerate your releases and perform manual spot checking to isolate issues quickly. Access hundreds of different devices without the hassles of procurement or maintenance.
• Pricing: Many different models are available for monthly or annual pricing plans. For the monthly plan, options include: Manual, $29/month; Individual, $199/month; Expert, $399/month; Professional, $499/month; Small Team, $699/month; Enterprise, contact company. A 14-day free trial is available.
• Available Devices: 308 devices are available, featuring Android from version 2.2.2 to 7.1.1 and iOS from 7.1.2 to 10.0.
• Notes: Both real devices and simulators/emulators are available. With last year's acquisition of TestObject, it supports manual real device testing and secure testing on private clouds. Company claims it's the "only test cloud that lets you automate your native, hybrid and mobile Web tests across emulators, simulators and real devices." Developers can run automated mobile tests across native or hybrid apps with Appium (cross-platform framework for iOS and Android apps), Espresso (Google's framework for testing Android apps) or Robotium (providing a record and playback test case creator for Android apps). All are open source frameworks.

Experitest
Experitest Ltd. offers online and onsite versions of its SeeTestCloud, along with automated and manual mobile functional testing, and mobile performance testing products.

• Company's Service Description: In today's mobile economy, you need to release new applications often. Low quality or bad experience are highly penalized by users. Making applications that work well in the wild from the first download requires development and testing with speed and quality in mind. Experitest mobile cloud testing tools allow you to introduce new devices and operating systems faster, automate more tests, and improves cross team collaboration. All, so you can release better apps, faster.
• Pricing: No plan comparisons are available as with the other offerings. Get more information here. 30-day free trials are available.
• Available Devices: The company says it provides access to hundreds of remote mobile devices hosted at Experitest datacenters. The online test cloud page lists 56 devices that are available, including a Fire HD 8 device from Amazon, iPhones, iPads, many Android phones, Blackberry devices and more.
• Notes: Plugins area available for Selenium Webdriver, Visual Studio, RFT, TestComplete, Java, Perl, Python, Ruby HP LoadRunner. It can generate detailed and shareable test reports including videos and images. It lets developers collaborate with screen view sharing to allow simple bug detection. Mass parallel tests can integrate with existing ALM environments. Non-functional testers can test their application performance under real network conditions and monitor device vitals. Supported OSes include: Android 2.1 – Android N (7) (including); iOS7 – iOS10.2 (Including); and Windows Phone 8.

Posted by David Ramel on May 5, 2017