👗 Founded in 2008, Zalando SE is a German multi-national E-commerce company headquartered in Berlin, Germany, it also has tech hubs in Dublin, Ireland; Dortmund, Germany, and Helsinki, Finland.
The company follows a platform approach, offering fashion and lifestyle products to customers in 23 European markets.
Zalando has raised a total of $615.9M in funding over 8 rounds. Their latest funding was raised on Nov 11, 2013, from a Private Equity round. Zalando is registered under the ticker ETR:ZAL since the 2014 IPO.
.png)
The mission’s main objective: improve overall customer experience and increase engagement and retention. One of the main goals of the Outfits Program is to create experiences that aim to redefine fashion inspiration and help customers create their new favorite outfits.The team designed a series of A/B experiments that were deployed on different areas of the Zalando webpage with the aim to observe customer behavior and draw the necessary conclusions to improve internal metrics.
With an overlap of two weeks, we shadowed the leaving PM for a week and then took over from week 2 while having the outgoing PM available for questions. We successfully ran ceremonies and the refinement sessions. We agreed on an internal roadmap where the PMs in that business unit were our stakeholders. Naturally, we liaised with the Head of Product and Portfolio Manager on priorities, and people/capacity needs.
The three experiments presented in this case all had something in common: they used A/B testing to optimize the best way to show outfits generated by an internal machine learning algorithm called Algorithmic Outfits to customers.
1. A/B/C experiment: Algorithmic Outfits Dynamically Created on the Home Page Based on Viewed vs Wishlisted vs Purchased Items (Web)
The Problem
The algorithmic outfits component on Home (page) provides outfit inspiration based on the items users had purchased in the past 6 months, which is only visible for logged-in users. Due to the limited number of users who have made a recent purchase, this component is only available for a smaller percentage of users. In order to ensure that more users can benefit from outfit inspiration, we leveraged the Algorithmic Outfits on a couple of new use cases: wishlisted and recently viewed items.
This means that all home visitors who have added at least one item to their wishlist or visited at least one Product Details Page in the past 60 days would be eligible for the component. Based on our estimation, this would enable the Algorithmic Outfits component for roughly 60% of Home users (about 6 digits daily users)
The Hypothesis
If users are shown outfits based on different clothing items, they will engage more with the content (also across multiple sessions), because the content will be refreshed more often (because users add items to the wishlist and visit Product Details Pages more often than making a purchase).
The Treatment
Control: Only purchased items
Treatment 1: Purchased and wish-listed items
Treatment 2: Purchased and viewed items
All treatments included both logged-in and logged-out users, and during the test, some users could only see certain items (e.g. logged-out users can only see clicked items).
Results
Recommendations
Due to the better positive uplifts across all metrics, the "viewed items" variant should be rolled out.
2. A/B Carousels vs no Carousels on the Outfit Page
The Problem
When Zalando users reach the Algorithmic Outfits on the Outfit page (A page where users can see outfits with more details), they have no other outfits to browse. Data collected by the analytics team showed that our most engaged customers (2+ interactions) find the next outfit to explore through the Outfits page. The Algorithmic Outfits do not offer any exploration options, disrupting the experience for our customers. Based on an analysis of the source of the traffic to the Outfit page, the team hypothesized that putting an Algorithmic Outfit carousel option on the Outfit page would increase traffic and engagement to the Outfit page. As shown in the picture below, there are potential growth loops that we could leverage if the content created is interesting enough for users.
The Hypothesis
We believe that allowing customers to see other outfits **on the Algorithmic Outfits on the Outfit page will result in an increase in interactions per user and share of users with multiple interaction days (the number of outfit customers with 2+ visit days with outfit interactions). We will know we have succeeded when there’s a statistically significant increase in the above KPI.
The Treatment
Control: no carousels as this was the current version
Treatment 1: two carousels, mimicking the structure of successful similar page types
Results and Conclusions
Recommendations
It was concluded that even though one of the guardrail KPIs was negatively affected, this could be explained by the higher visits on the Outfit page. Based on (1) the positive impact on the success and the other engagement KPIs, (2) the positive impact on the number of page views as well as (3) the stable interactions with the top part of the page, the treatment variant was implemented.
3. Learning from a failed test: A/B test on Carousel with 4 Algorithmic Outfits on the Product Details Page
Unlike the previous experiments shown in this case, the third test presented shows how experiments can confirm when NOT to pursue an initiative, and instead give learnings from it.
The Problem
Considering that Product Details Page to Outfit Page to Product Details Page is the most common journey performed by users, it made sense for the team to explore adding an Algorithmic Outfits carousel on the Product Details Page to see if customers engaged and strengthen the loop between Product Details Page and the Outfit page.
The image below shows how the user journey would be improved for our customers: from the Product Details Page to Algorithmic Outfits on the Outfit page. With this, the team aimed to decrease the exit rate, as well as increase time on page and user interaction (engagement). By providing more entry points into the Outfit page through the carousel placed on the Product Details Page, the estimated scale of impact would be x5 more traffic from the Product Details Page to the Outfit page.
The Hypothesis
We believe that our customers will engage more with a carousel of images containing a variety of personalized outfits as these will provide inspiration to users, in the many ways they can combine the item they are interested in with other relevant Zalando items to complete the look.
We also believe that some of the customers will click on the non-human outfits, which will lead them straight to the Algorithmic Outfits on the Outfit Page displaying five or more relevant Zalando items that compose the selected outfit, and by clicking on any of these items they will land on the Product Detail Page of that specific item, which generates an ongoing loop.
The Treatment
Control: On the Product Detail Page - ****All carousels in the existing order. No changes
Treatment 1: Show the Algorithmic Outfit carousel in the first position.
Results and conclusion
Recommendations
The test had tremendous success on engagement metrics. However, this solution hit some of the guardrail metrics negatively. The revenue per user dropped by a single-digit percentage (x.xx%), which would translate into a significant amount of revenue loss annually. For this reason, the decision was to not roll out the changes.
An important lesson learned is that when doing A/B tests, especially in bigger organizations like Zalando, you cannot lose sight of the business metrics, even if the overall result for the metrics that matter to your product scope benefit from the change. Furthermore, when introducing new features, the PM should always be aware of the customer funnel and check if the feature is relevant to the overall objective of that page for the company. In this case, the Product Details Page seems to be in a place where the customer is more advanced in their decision-making process, and the carousel may have distracted the customer from purchasing.
We onboarded a new PM to the team. And another one for the mobile app team that we were supporting additionally (in side-quest mode). We integrated them with the proper documentation and a list of ongoing/upcoming priorities to give them the scope of their responsibilities, manage their expectations, and allow them to start delivering as soon as possible.
💡 xx% (double-digit) increase of the traffic and engagement KPIs based on the learnings and success of the A/B tests.
💡 Invalidated a hypothesis on where users expect to see Algorithmic Outfits and where they would best interact with and use them.
💡 Successfully onboarded a new full-time Senior PM. Set them up for success with good handover documentation and a list of ongoing/upcoming priorities to hit the ground running.
.svg){kind=small}
