Running scientifically controlled experiments on your website is one of the best ways to ensure your site continues to evolve and improve. Successful split testing requires a repeatable framework upon which your ideas can be systematically tested and validated over time.
In my daily testing work, I rely on the classic Scientific Method (the same one you learned in grade school) to ensure I’m asking the right business questions. The Scientific Method will also help you turn those important business questions into viable testing ideas.
Let’s take a closer look at the Scientific Method and how it applies to a real test we recently ran.
Step 1: Question – In this step, we start asking business questions. In order for us to ask the right questions, we’ll have to use all the data we collected from our various tools. This might include using web analytics, exit surveys, heat maps, scroll maps, form analytics, and user testing as well as our own heuristic experience as professional testers.
For our test example, the business question we asked was “How can we get more visitors to click on the “Request a Demo” button? The client knew they had a better chance of the prospect purchasing the software once it was demoed, so getting more people to request a demo in the first place was critical.
Step 2: Hypothesis –A hypothesis is a supposition or proposed explanation made on the basis of limited evidence as a starting point for further investigation. A hypothesis must always be tested in order to validate the idea, but a weak hypothesis often leads to subtle or inconclusive test results. Notice in the below example that the strong test hypothesis is directly tied to requesting a demo, while the weak one is not.
Weak Test Hypothesis: We should test making the button maroon instead of blue so it stands out more.
Strong Test Hypothesis: By changing the button text from “Get a Demo” to “Free Demo”, and changing the color to maroon, we’ll see an increase in software demo requests.
Step 3: Experiment – Once we have a test idea with a strong hypothesis that is tied to a business metric we wish to improve, the next step is to test the hypothesis to see if it is true or false (or possibly indeterminate). For our example, we diverted half the traffic from the original page and routed it through our test variation below.
Step 4: Analyze – Once our experiment has concluded, we need to analyze the results based on statistical validity. In our day-to-day testing, we typically stop a test once the test variation achieves a 90% statistical significance. This means there is a 90% chance that the improvement being seen is not just due to random fluctuations.
Step 5: Conclude – The last step in the classic Scientific Method is to formulate a conclusion. There are several ways a test can turn out.
- The test variation beats the original page with a 90% or higher statistical significance. This is the best possible outcome. On average, only about 1 out of every 4-5 tests will achieve this status.
- The test variation is consistently showing improvement over the original, but the test isn’t achieving statistical significance, even after running for a long period of time. Depending on the circumstances, we may call the test ‘Non-Negative’ and recommend standardizing on the test variation. At other times, we may call the test indeterminate and recommend keeping to the original.
- The test variation is not proven to be any better or worse than the original. In these cases, the test is called a tie or ‘indeterminate.’
- The test variation actually underperforms as compared to the original. While this doesn’t happen very often, it does occur. A good testing team will take the time to learn from these types of “failed” experiments and these learnings can often lead to additional test ideas or test iterations.
For our test example, the maroon button labeled “Free Demo” beat out the original button by 96%. Below are the actual test results. So, our conclusion was to standardize on this new button color and wording.
When you work with the Americaneagle.com team, we’ll help guide you through all 5 steps of the scientific method and make specific recommendations on every test that we run.
Click here to register for a free test. We’ll provide a test idea, build out the test in our testing platform and discuss the results once the test has completed.
