“Now, when I’m talking to companies about personalization, I find they are all excited about the latest machine learning algorithms, but very few seem to have got the basics in place to be able to tell what’s working and what isn’t.”
https://medium.com/swlh/ab-testing-so-you-know-what-really-works-662f5c18b354
@mweagle interesting how the scale that a company has can be used to drive product innovation since they have more customer data to run these experiments. On the flip side I'm using some legacy software and it seems like the opposite - they just kept adding features more and more and the onboarding process is brutal
@smaroukis Good point. Scale can definitely accelerate learning. I think it also takes org discipline and coordination to reduce confounding experimental results.
I hear you about long-running software accumulating “features”, which often means user friction.
@mweagle Thanks for posting this Matt, I wrote it a few years ago and still haven’t seen anyone else approach universal AB testing as well as Netflix.
@adrianco Thanks for writing it
@adrianco Sorry - Hypothesis-Driven Development. The inductive & experimental approach you describe resonates with me and it's my mental frame when I hear “data-driven decisions". In contrast to the more "feature factory” approach (ref: https://medium.com/@johnpcutler/12-signs-youre-working-in-a-feature-factory-44a5b938d6a2) which seems unfortunately prevalent.
Interested to learn (if you can share) how Netflix committed to the more experimental frame.
@mweagle Oh, it was from day 1, the founding CEO was a data driven product guy. See “That Will Never Work” by Marc Randolph.
@adrianco Thx - added to queue