Why We’re So Keen on Cohort Analysis at Nozzle

“How do I know whether the changes I made are having an impact?”

This is the most common question we hear from Amazon sellers - and for good reason! Fortunately, cohort analysis is here to help.

With so much data now coupled with ever more external factors to consider - like shipping costs and related supply chain issues - it’s becoming ever more difficult to join the dots between cause and effect on Amazon.

Most Amazon sellers would point to the practice of split testing (aka A/B testing) as a way to try to answer some of these questions. Split testing is certainly useful, but answers a different type of question when compared to cohort analysis. Both should be used to drive decisions.

Split Testing (or A/B Testing) vs Cohort Analysis 

Cohort analysis allows you to group users by similar behaviors or traits so that you can analyze how these related groups (cohorts) perform against each other over a period of time.

Split-testing allows you to compare the performance of two different versions of your listing at the same time. 

For instance, if you want to test whether a lifestyle image on your listing results in a higher conversion rate compared to a product image, you can use a split test to see. All you need to ensure is:

1. The two features you want to test can be tested at the same time (i.e. the product image and the lifestyle image); and 
2. The test subjects are randomly assigned. 

Fortunately Amazon already has an in-built tool called ”Manage Experiments” where you can set up your split tests for your product titles and your product listing (with A+ content tests coming soon).

There are also a few third party tools that can run these split tests for you. 

Image: Amazon Seller Central Manage Experiments

However if you want to set up other split tests, say on the PPC side, this isn’t possible. Multiple ad groups/campaigns targeting the same keywords isn’t a clean measure. 

Using Cohort Analysis 

Cohort analysis is a way of grouping data according to a specific variable. In this case we’d be grouping your customers according to the month in which they first purchased so that we can measure what happens to them over time. For example, if you compare user behaviors between customers who first bought in December vs customers who first bought in January, you're going to get wildly different results. This is due to the impact of the Christmas shopping season. You could use a cohort analysis to separate customers into  “December” and ”January” cohorts and track their progress over time. Generally, we see that the December cohort has a lower customer retention rate and a lower customer lifetime value (profit per customer) because they’re more interested in good deals for the festive season.  

Cohort analysis is extremely valuable to quantify the impact of the changes you make over time. 

In the table below, we can see that the average lifetime value per customer across all ASINs for the period Dec ‘20 - Sept ‘21 is $10.40 after 3 months (highlighted in green). This provides a good starting point to compare cohorts. 

Image: Cohort Analysis Dashboard, Nozzle Tool

If we look at each cohort on a month by month basis, we can start making a few observations: 

Image: Cohort Analysis Dashboard, Nozzle Tool

The table above sorts your entire customer base according to when that customer first bought from you. For instance, someone who first buys in May ‘21 will always (and only) be part of the May ‘21 cohort. When that person buys again, say in Aug ‘21 (3 months later), all the profits and data continue to be added to the May ‘21 cohort only. 

Data Points

Using this principle, we track a few other meaningful data points for each cohort:

• Customer retention rate (”R%”): This is the number of people who buy at least twice anytime from their first purchase. Read more about why Customer Retention Rate matters.
• Total new customers: This is the number of new customers acquired in that month. We use a strict definition of whether we have ever seen this customer before. If not, they’re new-to-brand.
• Ad spend: We add up how much you’re spending for that ASIN/category/account per month by adding up sponsored brands, sponsored brands video, sponsored display and sponsored product ad spend. Read more on how much you should be spending on your PPC advertising.
• Customer acquisition cost (CAC): This tracks how much you’re spending on advertising to acquire a new customer for that month. It’s simply the ad spend column divided by the total new customers column. Read our Introduction to CLV and CAC on Amazon.
• First purchase (and all subsequent columns): This is the customer lifetime value at a specific point in time. The first purchase is obviously on the first purchase but if you go to, say, 3 months, you can track the average lifetime value after 3 months. 

‍

Observations

Looking at this table there are a few observations we can make:

• It looks like your newer cohorts (Aug ‘21 and Sept ‘21 - highlighted in blue) are taking longer to break even. We see this when we compare how much you had to pay in those months to acquire a new customer vs how profitable the average customer is. The CAC column is highlighted in yellow. The “breakeven month” for a cohort is highlighted in green. For Aug ‘21 we can see that it cost $10.65 to acquire a customer. The cohort only surpassed the $10.65 mark after 2 months (being $12.42). It’s a similar story for the Sept ‘12 cohort only breaking even after 2 months. Historically we see that in general it’s taken 1 month to break even. So the question is whether something has changed lately to cause it to increase by 1 month?
• It seems that the newer cohorts starting from May ‘21 have seen a drop off in retention rates. Retention rates are highlighted in pink. This started at 28% but decreased all the way to 11% for the Sept ‘21 cohort. Were you running more promotions than usual? We’ve noticed a drop in retention rates when more and steeper promotions are run.
• If we pick a specific period, say 3 months, we can compare whether our older customers are more or less valuable than our newer customers at the same point in time (i.e. 3 months).

Conclusion

This is all interesting…but how does this analysis help me improve my business by testing things?

Good question! So far we’ve just been talking about how cohort analysis can help you with historical customers. However it can also be used for forward looking analysis. For instance, let's say your average lifetime value after 3 months is $10.40 and you want to run an experiment to see if you can increase this. Starting from the month of October ‘21 (i.e. the next month), you’re going to introduce a change to increase your customer lifetime value. This could be increasing your re-marketing efforts via sponsored display. You can then simply track the cohort for 3 months and compare the retention rates and the lifetime value to see if the change made a difference. 

We recommend running 1 experiment at a time for the period you want to measure (3 months in our example). You can run 1 experiment per ASIN so you can actually be running multiple experiments that stretch across your entire catalog.

Our clients are constantly finding new initiatives to test (new packaging, new inserts, new PPC targeting rules for instance). Cohort analysis allows them to track the impact of each of these on various cohorts.

Hopefully by now we’ve convinced you of the value that cohort analysis can bring  - whether it’s in a Nozzle environment or anywhere else!

If you want to check out our Cohort Analysis dashboard make sure to check out our Nozzle Customer Analytics tool for FREE with our 14-day free trial!