Day 121 in MIT Sloan Fellows Class 2023, Digital Product Management 7 - "EXPERIMENT"

Today's class was a guest speaker series from Pete Koomen, visiting group partner of Y combinator and founder of Optimizely.

Obama 2008 campaign

Obama's $60 million dollar experiment - Optimizely

There were a bunch of testing plans in Obama's campaign from call-to-action buttons to visuals.

A/B Testing Got Obama $60 Million. You Can Do It Too | Mailmunch

The most interesting result from this test is optimal visual including banners and videos.

The result is a little bit surprising one from campaign team's perspective, but it demonstrated the power of the experiement beautifully.

Majoirty of campaign members forecasted video would win the A/B test. Why? 

They were strongly biased by their efforts and their passion to cheer Obama. 

The winner is here.

Some points I need to highlight.

• People supporting Democratic party tend to prefer this banner to others.
• At that time, video viewing was not necessarily a comfortable experience as it is.
• Video was much harder to create rather than static banner(curse of knowledge)

The winner banner had a sign-up rate of 11.6%as compared to the original page which had 8.26%. That’s a 40.6% improvement right there. It resulted in additional 2,880,000 email addresses on their email list which translated into an additional $60 million in donations.

Seven habits of experimental culture

1. Asking about experiment result
2. Using experiments to protect big idea
3. Celebrating wins
4. Celebrating losses
5. Hiring for an experimentation mindset
6. Context over control
7. Optimizing for experiment velocity
   1. Inventing a centralized analytics stack
   2. Hiring statisticians
   3. Lowering the approval bar for experiment launches
   4. Building out an experimentation center of excellence
   5. Holding open experiment reviews

In other words, I can translate them into my words as follows.

1. Involve leadership! and leadership should seriously commit experiment
2. Starting from a big hypothesis.
3. Visualize your win
4. Don't be afraid of failure, but reflect on it seriously.
5. Tell both story and truth 
6. Good experiments always have a good success metric. For example, Netflix and Booking.com have similar experimental culture. Due to the difference in success metric(subscription vs transaction), it's priority is totally different and they figured out it through countless experiments. Netflix prioritizes simple and stress-free UI to retain users, whereas Booking.com raise urgency to make transactions.
7. Do many experiments as possible.

Process breakdown and necessary resources

1. Idea
2. Plan
3. Run
4. Digest

For the first step, you need creativity. 

For the second step, data person and operation need to collaborate each other.

For the third step, you need to get approval from managers and don't let them disturb your experiments

For the fourth step, data guy and management aligned with how you interpret the result into various next steps.