ScreamingToaster

Introduction

I just released the first version of Wicked Blogging App for BlackBerry a week ago, and while I was going through the final release phase of the product, I was thinking about how much testing goes into the products that we make. It’s not really apparent to the end user the rigors that we put our apps and services through to make sure that they work on a multitude of devices, in different conditions, across many different types of networks. For the end user, as long as the user experience is great and they don’t see any hiccups, we’ve done our job well!

Just a personal observation. All software is not created equal. I use many different apps in my daily life, and I find that there is no standard baseline of quality across any set of apps. It totally depends on the software, and the people who created it. One of the most awesome pieces of software I use is IntellJ IDEA. It’s a Java development environment that’s unparalleled in it’s power and intelligence, ease of use, and performance. On the other hand I’ve used Vista’s Windows Movie Maker, which is a total piece of shit. I just wanted to make my customers aware of the painstaking effort that goes into releasing a final product at ScreamingToaster.

The process

At ScreamingToaster, we test our stuff iteratively, as code is being developed. We don’t wait for a code freeze before testing, and we run regression test cases on all our code every time we make a major change. This means testing out every single feature in the BB app as well as the services that it interacts with, which as you can imagine can be very time consuming. We test things like new account creation, user profile syncing on more than one BB, camera integration, Twitter integration, payments and licensing, for each and every version that we test. We have a dedicated test environment for our software, that’s completely separate from our production environment. We also have a whole suite of BB devices that we test the software on in our test environment, not just emulators running on our development machines. There’s no substitute for real world testing. Just because something works in the emulator doesn’t mean shit. We have to test in real world devices on real world networks, with real world problems, like battery life issues, app performance on slower hardware, slow and unreliable networks, etc.

Carrier limitations

We also have to test for things provided by the carrier, which might not be uniform across all regions. Eg, assisted GPS is a carrier provided feature. For ATT’s networks in the US, not all 3G markets have this feature enabled! Our software has to take that into account when we provide our geo-location features. The app has to degrade gracefully in the absence of geo location information. We have a complex algorithm that we use to determine someone’s location when many different things fail, like if AGPS fails, we try and use cellid lookups. If that fails, we go to autonomous GPS lookups, etc. The app and services do some really intricate and complex things that are totally hidden from the user experience. Older devices on Verizon’s network don’t allow 3rd party apps to access GPS, so it gets really tricky to deal with all these issues.

Network problems

Our BB apps always prefer WiFi to cellular, but they switch back and forth between the two without user intervention. Our ONE Platform (which powers our apps and services/cloud) is very smart in the way it deals with network issues. Our BB apps are very forgiving to network timeout issues, and network hiccups, and try to maintain your session as long as is possible. Eg, you might be at home connected to WiFi which Wicked is running, and then you walk outside and disconnect from WiFi and connect to cellular. Wicked handles this switch without any user intervention, and your session is not interrupted! These switches can happen as many times as you want, and it will handle them. While on cellular or WiFi, you might experience some network outages, because you’re not in coverage. Again, Wicked is aware of such conditions and tries to work over crappy network connections, without interrupting your user experience or your session with the cloud. When the network is so bad that Wicked can’t continue, then it gracefully bows out and displays a message to you explaining the problem, and exits after you see this message. Bad network connections can happen when you put your Javelin in your pocket when you’re in your car, when you walk into an elevator, etc. There are so many scenarios that we have to account for in real world testing. It’s not easy and it takes a lot of time.

Device to device interaction

For us, it’s not enough to test software in ‘isolation’. All our software connects to our cloud (the ONE Platform), and realtime sync of information is a standard feature of all apps powered by this platform, so is payments and licensing, and geolocation. Not only do we have to test our software with 1 user logged into 1 BB, we have to test with 3 users logged into 3 BB devices, interacting with a multitude of other users on such devices as well! Granted, that most of our users don’t even use these usage scenarios, but we are confident that as consumers become more familiar with our software and start to leverage these enhanced capabilities, they won’t be able to live without them. This is why we test things in ways that most users are probably not using right now, but they will in the future.

Here’s a picture of some of the devices we had hooked up (not all of them are shown in this picture) testing the ‘Who’s around me’ feature in Wicked. We had to test what happens when users login and logout, when they change their presence state from “Everybody” to “Nobody”, and move around town and their location is updated, and then as they change their status messages and icons, etc. etc. etc. As you can tell this gets complicated really quickly. Not only do we have to ensure each individual function works, we have to make sure they all work with the same user signed into multiple BB’s, etc. Again, our customers will never see this, as long as ‘Who’s around me’ works without any glitches, they will be happy with it. The user experience is very simple, the software that makes this possible is very complex.

Performance

This is one of my most favorite things to do with my software – tune it for maximum performance. All our software is run at 4x normal rate on test hardware . Think of a car engine that you can only rev to 1000 rpm; for testing we rev it up to over 4000x rpm at idle . Yes, we stress test the crap out of our software, more than you ever will . Eg, for testing we push out updates at 400 % the rate and volume that will ever be experienced in production use! We then do some crazy optimizations for load and scalability, on both our services, and BB apps, to ensure that our apps can handle 100,000’s of users being logged in concurrently constantly updating their information and interacting with the system in real time. In production use, we see just a fraction of this load, but by testing for such extremes, we can make our software use as little bandwidth, CPU, and memory as we can possibly get out of it! While delivering the best user experience. This means crazy optimizations in the way Wicked interacts with the cloud, it means turning off animation in certain parts of the user experience, because it would impact overall performance, etc. There is so much tuning and balancing that goes behind the scenes you would really be surprised how complex it is to get the balance just right. Also, we test on slower hardware and networks, like the Javelin 8900 on 2.5G network, and ensure that everything works great on it. Now when you put this app on a Tour or Bold on 3G, it FLIES!

Summary

I hope I provided some insight into how we polish and refine our products before they are made ready for public consumption. This is also why we don’t believe that giving our apps away for free is a sustainable business model for us. There’s more to an app and a company and people than the price tag. As savvy consumers, you have to look at all that you’re getting. You’ve paid a lot of money for a BB, and even more for the voice/data plan just to make it go. When you put an app on it, make sure that this app will add value to your purchase, and not take away from it by bricking the device, making it unresponsive, etc. Some people like to think about the upfront cost of an app, while forgetting the entire lifecycle cost. We don’t. We are committed to delivering a tremendous amount of functionality, fantastic customer service and support, at a great value. Doesn’t mean it’s free, but if you add up what you’re paying for what you’re getting, you will be happy at the bargain you are getting it at.

Nazmul, President, ScreamingToaster.