How Certain Are You That Your Idea Will Win?
All optimization work starts with you and your degree of certainty about particular ideas. This opens up the door for immediate implementation of ideas when we are truly certain (and accepts that not everything has to be tested). Here is an emerging certainty scale we are beginning to use on projects.
The Optimization Certainty Scale
We are starting to use this scale to help us prioritize ideas, as well help to make implementation decisions. Half of the scale is reserved for an initial subjective expression, and the other half requires quality tests to be used to back up a higher certainty claim. The highest certainty mark I'll ever make may be a 9, which requires three strong supporting winning test, and 1 losing test. This is so, because I believe that even more certainty can be extracted from avoiding something that a relevant losing test uncovered. And why avoid a 10? Because such high certainty closes too many doors. :)
How Do You View Your Own Certainty?
Please share your own thoughts.
Posted by 
Comments
Scott Barnett 8 years ago ↑0↓0
I think there should be at least 2 dimensions used to determine whether something is tested: 1) certainty, 2) importance.
You've covered certainty here and left out importance. I recommend adding it to this conceptual model because for low value items sometimes it's just best to guess and not both testing. For very high value items, it's important to test even if you're 95% it'll work out because the 5% chance of being wrong is so costly that it's worth the time/effort to run the test.
Reply
Jakub Linowski 8 years ago ↑0↓0
Scott, thanks for the comment. We've already separated subjective certainty from certainty that is based on tests in our prioritization work sheets. We then can add them together as well. I think this subjective expression of certainty allows to express importance in some way. Agree?
Reply
bryandavidk 8 years ago ↑0↓0
This linear description could be a great summary of A/B test results
Reply
Patrick Boens 8 years ago ↑0↓0
Interesting, Jakub. However I don't agree with it, or only partially. I believe that things cannot be tested alone (idea 1, 2 and 3 in your case): they must be tested in a context. I would therefore suggest that the scale you use gets adapted to include that extra dimension.
By the way this resembles Prediction machines. ????
PS: I read all your posts ... Keep up with the good work
Reply
Jakub Linowski 8 years ago ↑0↓0
Hi Patrick, can you elaborate why you think a particular idea cannot be tested alone? Isn't that challenging the concept of controlled experimentation?
Reply