Denial, anger, bargaining, depression, and acceptance: The A/B testing lifecycle.

If you build it, will they come?

At Yipit, we are constantly looking to build cool new stuff without burning cycles on ideas destined for failure.

To stay on the right side of the ledger in this regard,  we’ve deeply tied our development process to analytics.  The company motto is “be a scientist”, which incidentally makes for great memes during our weekly company meetings.

Field of dreams

Always be testing

To be considered during sprint planning, a feature normally must relate back to a testable hypothesis which attempts to predict its impact on key company metrics.  If the upside of a hypothesis is promising enough,we will move quickly to prototype the feature, utilizing a combination of internal and third party A/B testing tools.

As a developer at Yipit, I have become intimately familiar with the psychological ups and downs of experimentation.

Much like a someone suffering from grief, running serial A/B tests can drive you toward a path of extreme emotional states.  Here are a handful of the tell-tale steps that you may recognize in some of your A/B testing co-workers:

Denial

Wait, it’s down 10%?  That makes no sense.  Some javascript must be broken.  Is the HTML valid?  Did our acceptance testing miss something with IE(6|7|8)? Shit, what is the usability like on your iPhone?

Let me go ahead and bother my friends outside of my company and ask them to check out the app to confirm everything is in order.  They’ll gladly interrupt their lives to figure out whether the modal window is opening in 300ms or less.

Anger

Hmm, it’s all working?  Well, the new variation would be performing better if Google stopped sabotaging it.  Their stupid algorithm is heavily weighting views toward the control version. I understand that the variable group is underperforming by 20% and Google Content Experiments are set up to stop the bleeding, but just give it a fair shake for a little bit!

Maybe if I refresh the page enough times, things will change.  I just wish it were not 3AM in the morning…

Bargaining

Even if the variation is 3% worse than the control group, we can still justify the feature.  It may not be ideal in all respects but it fits into our overall strategy and it looks great aesthetically.  You could make the argument that it helps [insert some fringe metric] while hurting the goal we’ve defined here.  Maybe we should reassess our KPIs?

Worst case scenario, at least we’ve learned something here for future tests [no, you have not].  We are now one tweak away from cracking this wide open [no, you are not].

Depression

This is an abject failure, I realize that now.  The change would single handedly undo months worth of good work by other people at the company.

I guess that I can’t intuit everything about user experience on the internet. All of my days spent reading generalized ramblings on UI/UX were for naught.

I have no discernible skill in this dimension that is meaningfully different than any other intelligent person who cares to spend the time.  Who am I and what is my place in this online ecosystem?

Acceptance

Yes, the test has failed, time to move on to something else.  Speaking of which, the logged out homepage looks pretty suboptimal.  Wait, one second, how could I have been so blind?  It’s obvious, I just have to push these pixels down while jazzing up the copy [start the whole cycle again].

On a more serious note

The most important lesson that I have learned in my life has been that you should train yourself to avoid results oriented thinking at all costs.  If the process is good (and the A/B tests plentiful), it will all work out for you in the long run.

Speaking of which, I am running an A/B Test on this blog as we speak.  How do those 50% of you receiving free iPhone 5s feel right about now?

Enjoy this post? Follow me on Twitter (@bjpless)

5 thoughts on “Denial, anger, bargaining, depression, and acceptance: The A/B testing lifecycle.

  1. Pingback: Balancing Design Instinct + Judgment vs. A/B Testing for Design Changes | DevOpsANGLE
  2. Pingback: The dos and don'ts in A/B testing | Usersnap Blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s