The Crucial Role of Feedback Loops in A_B Testing - VWO Webinar (1).pdf

This session is designed for optimizers who are ready to move beyond rigid testing roadmaps and embrace a more dynamic, iterative approach. We'll delve into the principles of Kaizen, the Japanese philosophy of continuous improvement, and how it can revolutionize your testing practices.



In this webinar, our expert speaker, Richard Joe of Kaizen CRO, will show you how to extract valuable insights from both quantitative and qualitative data post-test analysis. We'll guide you through creating a 're-test-launch-learn' cycle, ensuring that each experiment builds on the last for ongoing optimization. By adopting a growth mindset, as championed by Carol Dweck, you'll view failures as stepping stones to success, ultimately driving higher conversion rates and richer learnings.



Join us to transform your A/B testing approach and achieve sustained growth through effective feedback loops. This session will equip you with the tools and mindset needed to navigate the complexities of experimentation and maximize your results.

1. Richard Joe
Founder, Kaizen CRO

[email protected]
The Crucial Role
of Feedback
Loops in A/B
Testing

2. Introduction

3. Iterate Beyond the Roadmap
Adopt a Growth Mindset

4. Limitations of Rigid Roadmaps



Launch a test, learn from it but get into the habit of shelving tests and not iterating
on them (win or lose) and move onto the next test on your roadmap
Problem: Lose out on further Learnings, Conversions and Hypothesis validation
You’ll probably recognise this situation

5. Power of Iterative Testing Cycles



Result: Squeeze more learnings and conversions from your test ideas, learn more and
get more wins
Process: Launch a test, learn from it, reiterate on it and launch follow up test

6. Introduction to Kaizen




Japanese word kaizen means 'improvement' or 'change for better


Common Usage: ‘Continuous Improvement’ in business activities

7. Kaizen



Embodies what CRO and experimentation are all about


Continually optimise website conversion, learn more about our audiences
preferences and improve the overall user experience.

8. Kaizen Approach to A/B Testing



Result: Squeeze more learnings and conversions from your test ideas, learn more and
get more wins
Process: Launch a test, learn from it, reiterate on it and launch follow up test

9. Kaizen Approach to A/B Testing




“Habits are the compound interest of self-improvement. If you can get 1% better
each day for one year, you’ll end up 37 times better by the time you’re done” - James
Clear, Atomic Habits

10. Compound Effect



The principle that underscores the monumental impact of small, consistent actions
over time, leading to significant outcomes.
The kaizen approach to A/B testing can have a
positive compound effect over time

11. How Do We Apply the Principle of Kaizen Within
Feedback Loops?

12. Feedback Loop



A feedback loop is a cause-and-effect system in which the outputs of a system
feedback in as inputs and prompt new cycles



Feedback loops exist throughout the world, from thermostat control systems to
customer feedback loops in business.

13. Feedback Loop



2 Main types



1) Positive: A positive feedback loop is a closed loop in which the outputs accelerate
a process







2) Negative: A negative feedback loop is one in which outputs slow down a process
and preserve a stable state

14. Feedback Loop



Example of Positive and Negative loops

15. Feedback Loop



A good CRO will want to avoid iterating on negative feedback loops and iterate on
positive ones.

16. Feedback Loop



Business example










Pricing feedback loop: If a company sets a price too high, customer demand
may fall. This feedback loop can then prompt the company to lower their prices
which in turn can prompt customers to make a purchase

17. Applying Kaizen to A/B Testing Feedback Loops



Follow-up Test → Launch → Learn cycle










Process:

18. Applying Kaizen to A/B Testing Feedback Loops



Step 1: Launch initial test










Steps:
Step 2: After test has completed conduct
post-test analysis
Step 3: From your learnings, ask yourself what
did you learn? What can you iterate on for next
time?

19. Applying Kaizen to A/B Testing Feedback Loops



Step 4: Develop an iterative hypothesis for
launching a follow-up test










Steps:
Step 5: Iterate your changes within the
testing tool.

Step 6: Launch follow-up test

20. Applying Kaizen to A/B Testing Feedback Loops



Step 7: Once the test is completed, again do
post-test analysis. Did you gain further
learnings or insights?










Steps:
Step 8: If possible from your learnings,
get into an iterative cycle of testing

21. Working Example: Sticky CTA



●Sticky CTA on Bank Savings page
●Result for Primary Metric: 10% increase in users starting an application












1st Cycle: Initial Test
●Follow-up Test Hypothesis: Sticky CTA’s may work in getting users to sign up
for another account because they are a useful form of choice architecture.
●Learnings: Sticky CTAs can be a useful form of choice architecture in
encouraging users to take action

22. Working Example: Sticky CTA



1st Cycle: Initial Test

23. Working Example: Sticky CTA



●Sticky CTA on Term Deposits page
●Result for Primary Metric: 15% increase in users starting an application












2nd Cycle: Follow-up Test
●Learnings: Sticky CTAs are a useful form of choice architecture in
getting users to take action in setting up a term deposit

24. Working Example: Sticky CTA



2nd Cycle: Follow-up Test

25. Working Example: Sticky CTA



Further Learnings: Screen recording data has shown on
mobile a trend of rage clicks of users closing the Sticky
CTA









2nd Cycle: Follow-up Test
Re-test Hypothesis: If we reduce the size of the Sticky
CTA on mobile, we will see an increase in users starting
an application because users aren’t being annoyed

26. Working Example: Sticky CTA



●Follow-up Test: Reduce size of Sticky CTA on mobile by
30%









3rd Cycle: Follow-up Test
●Result: We saw a 5% increase in applications started
on mobile and a downward trend in rage clicks
●Learnings: Reducing the Size of Sticky CTAs on
mobile can help with conversion and user experience
if they’re annoying users

27. Adopt a Growth Mindset



Embrace challenges and view failure as a learning opportunity, applying
these lessons to future tests for better outcomes.

28. Adopt a Growth Mindset



Definition: “Individuals who believe their talents can be developed (through
hard work, good strategies, and input from others) have a growth mindset.” -
Dr Carol Dweck

29. Adopt a Growth Mindset



Fixed Mindset: Is one that assumes abilities and understanding are relatively
fixed. Those with a fixed mindset may not believe that intelligence can be
enhanced, or that you either “have it or you don’t” when it comes to abilities
and talents

30. Adopt a Growth Mindset



Failure: Accept that this will happen when you experiment

Focus on learnings and not just winning tests
What can you do moving forward?

31. Adopt a Growth Mindset



Failing Forward
“Fail early, fail often, but always
fail forward” - John C. Maxwell

32. Adopt a Growth Mindset



Sticky CTA
In reality we re-iterated first on the everyday account page . Results were
inconclusive


Remember: if we had a fixed mindset, it would have been easy to give up at that point

Re-iterated again for Term Deposits and results were positive

33. How Long Should You Iterate For?



Analogy of squeezing the lemon - Daphne Tideman

Initial first squeezes will give you good returns

Squeeze too long, diminishing returns occur

34. How Long Should You Iterate For?



Consider resource constraints: Time, focus, development costs

What is the degree of experimentation maturity at your organisation?

Juggle higher priority tests

35. In Summary



Following rigid roadmaps leaves fruit hanging on the table

Adopting a Kaizen approach to feedback loops in A/B testing can unlock benefits

Growth mindset is important in failing forward
[email protected]
https://www.linkedin.com/in/richard-joe/