This a guest post from Maneesh Sethi, Founder, and CEO of Pavlok (https://www.pavlok.com), an innovative company that creates wearables and technology designed to help people break bad habits and change their behavior for the better. Victory Medium and Pavlok have worked together for a number of years – this original a/b test was conducted nearly 18 months ago in Spring 2017, but we decided to embargo the results due to competitors attempting to model Pavlok’s industry-leading strategies for behavior optimization.
With the news this month that Amazon is finally in a beta test for an offsite conversion pixel they are calling their “Amazon Attribution Tool,” it seemed like the perfect opportunity to explain how Pavlok and Victory Medium used Google Analytics, Google Optimize and Facebook ads to test the checkout purchase flow of Shopify vs Amazon.
Imagine your top line revenue being cut in half. You’ve lost half of your customers and your profitability has been significantly slashed. Even worse, imagine being totally blind and not even knowing everything you’ve lost.It may sound like a nightmarish scenario you’d never allow to happen at your business. But not checking out assumptions is actually something that can happen to the best of us when we’re overwhelmed with the day-to-day rigors of running a business.Pavlok’s cutting-edge wearable (Purchase at pavlok.com) has helped over 50,000 people quit their bad habits, become more productive, and wake up on time. We’ve built a company that helps people, and we do it by always working to improve our hardware, software and sometimes even the process for people to purchase our product.In early 2017, Pavlok conducted an A/B test between Shopify and Amazon that kept us from losing tens of thousands of customers and hundreds of thousands of dollars. And it started with a hypothesis …
Customers Prefer Amazon, Right?
The test began in March 2017 and it was one of Pavlok’s most ambitious undertakings to date. However, before explaining this test and the results, we want to add a slight disclaimer about Amazon and how we use it.We’re huge fans of Amazon and continue to use Amazon Payments and sell directly on Amazon when we have the inventory to support it.What our Amazon vs Shopify test was trying to determine was whether we should sell only via Amazon, and effectively turn off our current on-site checkout flow, and then eventually testing whether some hybrid Amazon checkout flow was feasible. Our hypothesis was that selling on Amazon would add credibility and that first-time visitors, in particular, would be more willing to make a purchase via Amazon than our site.We had a healthy sense of skepticism that Amazon would increase the number of people purchasing Pavlok’s, but they are the largest e-commerce retailer on the internet for a reason: namely, that they make buying online easy.
Behind The Scenes (Of Our Tests)
(A) Shopify Checkout Flow from a Facebook Ad
50% of website visitors from our Facebook advertisements were segmented in the test for a Shopify checkout and tracked via this process:
- A user was shown an advertisement on Facebook with a URL parameter in the landing page URL that forced the “Shopify checkout flow” test on Google Optimize (Google’s support article on query parameter targeting is here)
- A user came to Pavlok.com via the Facebook advertisement – and no matter what page (homepage, blog post, or other landing pages) all of the product details page links went to Version A of our Pavlok product details page with an “Add to Cart” button that linked to the Shopify checkout.
- A user added to cart and checked out via a native Shopify checkout page.
- After purchase, a user was redirected to the “Thank You” page and pixels fired tracking the purchase and conversion events from both Facebook (for direct advertising attribution tracking) and from Google Analytics Advanced E-commerce.
(B) Amazon Checkout Flow from a Facebook ad
50% of website visitors from our Facebook advertisements were segmented in the test for an Amazon Product Details Page & Purchase Funnel and tracked via this process:
- A user was shown an advertisement on Facebook with a URL parameter in the landing page URL that forced the “Amazon checkout flow” test on Google Optimize.
- A user came to Pavlok.com via the Facebook advertisement – and no matter what page (homepage, blog post, or other landing pages) all of the product details page links went directly to our Amazon product details page (Archived Amazon Product Link) and the URL being used to send the prospective customer to Amazon included an Amazon affiliate link specific for that advertising campaign. This process to use an affiliate link helped us test a few different Facebook campaigns and Amazon purchase flows with just new Google Optimize test variations with a unique Amazon affiliate product link URL and a query parameter used for targeting in Google Optimize.
- A user added to cart from our Amazon.com product details page and checked out via the native Amazon checkout.
- After purchase, a user was redirected to the “Thank You” page on Amazon but no pixels from Facebook or Google fired due to Amazon’s previous (and current) policies — so the purchase events appended to Amazon affiliate links had to be manually added as conversions from the Facebook ads in the dashboard tracking this test. This attribution tracking was not ideal, but we knew how many units we were selling and the Amazon affiliate link data was closely aligning with the increase in sales via our test, and determined the Amazon affiliate links seemed to be tracking accurately for fine-grained ad-campaign/amazon-affiliate-URL segments.
Testing two platforms that don’t want to play nice
To run this experiment, we had several technical hurdles to overcome — largely the challenge of running advertising campaigns on Facebook and tracking the purchases on a Shopify purchase flow compared to an Amazon purchase flow.For whatever reason, Amazon doesn’t let businesses place 3rd party advertising pixels from Facebook and Google on after-purchase thank you pages, and they only let you use Amazon tracking links, so all the data lives on Amazon. This was the case in 2017, and it’s largely the case in 2018. This means that if we wanted to run Facebook advertising campaigns or have email marketing campaigns, we needed dozens of custom tracking links from Amazon. And then we’d need to match up the Amazon data to these other sources.Normally when running Facebook ads with the Facebook pixel setup with a Shopify checkout flow, the dynamic revenue from every purchase in Shopify is automatically synced to the Facebook advertising account. In Facebook advertising reports, you can literally see that X ad campaign brought in $Y in revenue. Shopify’s implementation of the Facebook pixel is a wonderfully easy setup (like 5 seconds to copy and paste) and then there is complete transparency between Facebook ads and Shopify revenue.When running Facebook ads that drive to a website like Pavlok and then subsequently drive visitors to Amazon to purchase, it’s extremely difficult to track the purchase conversions, and it has to be done via Amazon Affiliate Links. But even with those links, we had to create a unique Google Optimize variation for each Amazon affiliate link and therefore hit variation limits. This website-source-to-Amazon product details page tracking challenge makes optimizing a marketing campaign a little slower, but we had such an audacious test that it was likely going to have a big swing in one direction or another, and decided to continue.
Running a 6-Figure E-commerce A/B Test: Shopify vs Amazon
In March 2017, we had been seeing a steady increase in buyers from Amazon for months but weren’t sure how deep to integrate the native Amazon purchase funnel into our existing website purchase funnels and product details pages. We originally ran a narrow test to only traffic from Facebook ads, but after some early positive data, we decided we wanted to send all of our website visitors through this test — and in March turned on the 50/50 split of all our traffic between Shopify and Amazon purchase flows.The experiment basically A/B tested whether someone trying to buy a Pavlok would be directed to use the Amazon.com native purchase flow, or whether they would be directed to use the Shopify native purchase flow.
Amazon vs Shopify Product Page Test Details + The “Purchase on Amazon” CTA
We ended up running about 6 unique versions of pages during this test — largely trying to figure out how to fairly test an Amazon purchase flow vs a Shopify purchase flow. The core thing we realized during the test was that users sometimes need more than 1 sales page before deciding to purchase — so by sending a user to Amazon prematurely, that user had a lower purchase conversion rate than if that same user had been shown one additional sales/product page on our website before having a button that clearly told the user it would take them to Amazon to complete the purchase. Once we realized that there was a distinct benefit to a slightly warmer user being sent to Amazon for the trusted buying experience, we kept that version of the test on for just over 2.5 weeks so that the Amazon vs Shopify checkout flow test was as architecturally sound as possible. High-Level A/B Test Details: Amazon vs ShopifyOn one version of our test, if a visitor went to Pavlok.com, half were shown Shopify purchase buttons and half were directed to our Amazon product details page, if they clicked “Purchase your Pavlok” Here’s a screenshot showing these buttons that were part of the test:
One group of users were shown a test where after clicking the “Purchase Your Pavlok” button, they were shown our Shopify product details page that looked like the image below — the yellow “Add to Cart” button took users to our Amazon product details page. We had some slight variations of this “Add to Cart” button and ended up using one for most of the month that said “Purchase on Amazon” — it was obviously a bit more clear for users and had a consistently better conversion rate than a simple “Add to Cart.”
On the above Shopify product details page and in other variations with a direct to Amazon funnel, visitors were redirected to Amazon to complete the purchase on a page that looked like this:
The Shocking Test Results
During our month of testing in March, we sent about half of our total website traffic to Amazon to try and complete their purchase, instead of completing their purchase via our normal Shopify checkout flow. Our sales were gutted for visitors who were sent to Amazon during the test. In the graph below, you can see how significantly our revenue dropped during this month of testing.
The Pavlok sales conversion rate over the 6-months prior to the March test was 0.88%. This means that 0.88% of unique visitors who come to Pavlok.com or any of our pages end up purchasing.During our March test with the Amazon checkout flow, the sales conversion rate for unique visitors on our website was 0.58%. In short, driving customers to an Amazon product details page to purchase instead of completing the purchase on our website via Shopify, resulted in a 34% decrease in our unique website visitor conversion rate compared to the previous 6-month average. March 2017 was officially the worst month of sales for Pavlok in over a year….thankfully it was just an outlier!
Amazon vs Shopify Checkout Conclusions
Pavlok has now returned to our normal checkout flow that lives on our Pavlok.com website instead of on Amazon, and we merely have Amazon payments as a checkout option and occasionally make inventory available to sell on Amazon.The test to drive some of our website traffic to Amazon product details pages, and encourage people to purchase only via Amazon, dramatically underperformed compared to our normal process of driving website traffic to our own product details pages and checkout flow via Shopify.As a small business, Pavlok has leaned heavily on fantastic e-commerce companies like Shopify, PayPal, and Amazon to give customers confidence in their transactions. Our goal is to find e-commerce solutions that help us grow our business, and also are easy for our customers to use.From our perspective, we’ve grown an even deeper appreciation for how our e-commerce platform and checkout tools impact customer trust and purchase completion, and how many marketing strategies are complemented by Shopify’s strong 3rd party analytics and tracking integrations, compared to Amazon. Most e-commerce platforms allow 3rd party pixels to fire, otherwise, it’s nearly impossible to track the effectiveness of those ads. We still have our products listed on Amazon for direct sales when inventory allows it and have Amazon as a purchase option on our Shopify checkout, but it would take a massive change in Amazon’s product details page options, pixel tracking features, and merchant checkout flows to get us to consider driving any significant traffic into their pages again. We’re not sure if this even matters to Amazon since nothing has changed in over a year, and since they want to encourage buying native Amazon ads (which seem to be a good investment for some brands), but we know enough peers and e-commerce stores that use some sort of hybrid business website + Amazon product details pages for their e-commerce, and we wanted to make sure to share our results for those types of business owners considering a new investment in Amazon product details pages.
Here’s to breaking more bad habits in 2017 and beyond… and using data to help improve your business! Maneesh Sethi, Pavlok Founder & CEO (https://www.pavlok.com)*(Test architecture and technical edits by Zach Edwards, Victory Medium Founder & CEO)
Recent Comments