Applying User Testing to Make UX Decisions in Real Time

Oct 22, 2017 | Discussion

On a recent project, we (the project team) were having difficulty deciding on a particular method of navigation for our new app. It was for an app that would replace one currently being used by millions of customers of a large Telecom company, and there was a wide variance of opinion on which method of navigation was best. And the stakes for decision were high: the app had been through numerous redesigns and had never really lived up to user expectations, largely due to confusing navigation.

So how could we decide which method of navigation would be most understandable and engaging for users? And how could we sell our decision to skeptical stakeholders? By testing, of course.


The Contenders

After some discussion and research into highly successful apps that users love, we came up with three basic contenders. First, the iOS standard bottom bar navigation as featured in most leading iOS apps. Second, the almost universal web standard for mobile navigation, the “hamburger” menu. Third but not least, we came up with a custom made design based on a circular menu that would appear when the user tapped an icon at the bottom of the screen (that just happened to look like the client’s logo).


The Method

We built three identical prototypes, the only difference between them being the navigation paradigms described above. Users had the ability to tap through all three designs on their phones to explore the experiences. The three mobile prototypes were tested in person and remotely, using, and users were asked a short series of survey questions.

Users were selected carefully: telecom customers who had recently used their provider’s billing app (our clients’ or another), as well as front line customer service agents who had been dealing with customers using the current client app. We tested the prototypes against users’ experience with their current telecom provider’s app, as well as each of the three prototypes against each other.

By gathering qualitative data through user zoom, and qualitative feedback through user interviews with customers and front line customer service agents, we were able to learn a lot about how users would respond to each. And what’s more, we were able to come up with a pretty reliable basis on which to make our decision and sell it to our stakeholders.


The Winner


And the winner is: iOS standard navigation by a nose! All of the details are in the User Testing Results presentation. It was a very close result. It turned out that all three navigation paradigms were numerically almost identical in performance. Users could complete all of the tasks nearly all of the time with any of the three methods. Bit we also asked the qualitative NPS question “How likely are you to recommend this app to a friend or colleague?”. This is where the apps differentiated themselves. Users were more likely to recommend to iOS tab bar. Reasons why can be debated over a beer any time, but the stats don’t lie. Side note: the tab bar was a lot more economical to implement and maintain than the other two which would have required developing custom controls, which helped to nudge it in front with the project team.