Agile UX Testing @ Zalando

by Carina Kuhr - 30 May 2014

Some weeks ago we – Zalando’s UserLab Team – introduced a new user experience research approach to our stakeholders that supports the fast pace and flexibility of their agile product development processes. In this post I would like to share our ideas and findings about the development and implementation of this agile UX testing approach with you. But let me give you a brief introduction of the UserLab’s role at Zalando first:

The daily business of the UserLab Team

As part of the Onsite Testing Team, the UserLab is the place where Zalando’s most recent features and innovations are shown to users before going live or before being A/B tested. In the UserLab we talk to customers, observe their online behavior when browsing on Zalando, we analyze use cases, point out problems and eventually we give recommendations to the product teams. In order to test all kinds of hypotheses about the users, we apply a lot of different methods like interviews, focus groups, eye tracking, surveys or card sorting. We believe in a user centered design approach where combining qualitative (subjective) and quantitative (objective) user research helps developers, designers and product managers to make the right design decisions and consequently ensure good usability, high relevance and positive user experiences for our websites and apps. So we try to serve as an interface between the people who develop the features and the people who use the features, and make sure that the users’ characteristics, goals, expectations and preferences are considered in the product development phases.

../images/agile_schedule_ux_post_2.png

Our experiences, challenges and goals

The UserLab’s testing projects of the last two years have had a positive impact on the user centered design process at Zalando, but still the biggest challenge for us is to make sure that qualitative user testing becomes a prerequisite in product development and to establish continuing awareness for the benefits of it. Especially for those stakeholders who are not familiar with qualitative testing yet. More and more product teams are involving us in their design decisions and we are pretty enthusiastic about the growing interest in user experience testing at Zalando right now. Product teams appreciate the possibility of outsourcing user research to us and receiving a detailed analysis, however they often need to make quicker design decisions than a thorough usability analysis of the results of 8-10 complex user interviews allows. So we sat down, reflected on past projects, made a list of requirements and thought about how a new, more agile testing approach could meet these challenges.

So this is our mission statement for agile testing:

  • Possibility for quick, resource-efficient and focused user research of high quality
  • Reconcile user feedback with agile product development and establish user testing as a regular part of the rhythm
  • Make results and insights more tangible, work on solutions instead of producing findings
  • Provide regular performance measurement and determine success of recommendations
  • Effectively intertwine agile product development and user experience methods
  • Raise awareness for benefits of combining qualitative testing and quantitative testing
  • Create structures that make it easier for the product teams to develop products with having the user in mind --> develop towards a value proposition rather than a set of functions
  • Empower the product teams to do their own UX testing and provide advice wherever needed
  • Make it possible to start working right after testing without having to wait for the analysis
  • And consequently: help to produce products that fulfill Zalando’s business goals

Agile Testing Principles at Zalando

Having these goals in mind, we built a project structure in which user testing is planned, conducted and analyzed within not more than four days. We borrowed ideas and principles from the ‘agile manifesto’ [1] as well as from the ‘Lean UX’ [2] framework and came up with a set of rules for our own agile testing approach.

Collaboration and Participation

If we want agile testing projects to be successful, it is essential that user researcher and product teams conduct user research together. This means that product teams take over more responsibility in testing and analysis than they did in the past. This shift has to happen for a couple of reasons:

  • It strengthens the product teams’ empathy for the user and bridges the emotional gap between the people who build the product and the people who use it
  • Information is not filtered through deliverables and their interpretation; gathering user research becomes a firsthand experience for PMs, designers and developers, consequently increasing the quality of learning
  • Seeing a user fail to understand the product has much more impact than reading a report about users’ failure to understand a product
  • It creates awareness for the possibilities and constraints of qualitative testing (which makes our work a little easier)
  • It facilitates team discussions about the KPIs of the product

And last but not least, agile testing wouldn’t be possible in the first place if the UserLab Team couldn’t rely on the additional resources of the product teams.

Focus

In an agile testing project we don’t want to give a detailed evaluation of a feature with all its possible usability issues; we have other methods for that. Agile testing should be more of an opportunity for the product teams to quickly validate assumptions and move on with the next step in their product development. Short and efficient meetings and a brief product profile (‘product mapping’) that is filled out at the beginning of each agile testing project make sure that the team focuses on the most important research questions and hypotheses. Also, we resign from producing documents like sophisticated testing plans and detailed power point reports. Instead the focus is on the outcomes of the testing and the solution of problems that users have with our products.

Iteration

Implementing testing on a regular basis makes sure that teams show their product to typical users from time to time. Additionally, it helps the team in their decision-making and takes away pressure from the team members who are responsible for making the right choices. However, when planning the organization and timing of tests, we had to make some compromises at the cost of flexibility. Right now we have a pretty strict schedule in which each of the teams has a predetermined testing spot every four weeks. We are working on solutions to make the schedule more adaptable to the urgency of the to-be-tested topics in the future.

Lo-fi prototypes

Since agile UX testing is not about getting a detailed usability analysis, the prototype does not need to have all functionalities. Actually, we promote testing low-fidelity prototypes, because this helps to focus on the main pain points that the product teams want to solve with user testing. The testing object only needs to provide the core features for testing the assumptions about the user’s behavior or attitude. This can even be done with sketches, wireframes or paper-prototypes.

Transparency

We facilitate collaboration between the UserLab and product team in every step of testing. The test should be planned, conducted and analyzed together. Every team member is warmly welcome in the observation room and invited to join the discussion. Our goal is to make planning, testing and results transparent to every one of the own as well as other product teams. Now, after the first couple of agile testing projects, we realized that we aren’t quite where we want to be yet in providing transparency, but we have some ideas that will soon be tested.

Principles put into practice

../images/agile_schedule_ux_post_1.png
[1]Fowler, M., & Highsmith, J. (2001). The agile manifesto. Software Development, 9(8), 28–35.
[2]Gothelf, J., & Seiden, J. (2013). Lean UX: Applying Lean Principles to Improve User Experience. Cambridge: O’Reilly

Similar blog posts