Validation of Virtual for A/B testing
Shopnosis.io team is constantly validating our data & analytics, to develop a highly predictive and highly scalable in-store execution A/B testing solution.
Validation is done on several levels:
- A/A testing
- A/B test of POS marketing creatives: performance in a physical aisle vs Shopnosis.io virtual aisle
A/A testing
A/A testing is an important first step to make sure A/B tests run correctly:
- Making sure the methodology isn't capturing too much statistical noise
- Sampling and recruitment are conducted properly
We did this by comparing two identical variations, with no changes between the control and test group
- Introduced a new (test) brand, with 3 SKUs and 10 facings
- Test brand statistics are in the table below
- Next to the test product, we compared brand shares for ALL brands in the aisle. 39 out of 40 brands show no sig differences between two cells (90% confidence level)
Control | Test group | |
Test brand penetration | 5.56% | 5.05% |
Test brand market share | 2.57% | 2.12% |
Best performing SKU market share | 1.71% | 1.69% |
A/B test of POS marketing creatives: performance in a physical aisle vs Shopnosis.io virtual aisle
- Coming soon
Shopper feedback
Finally, good shopper experience is one of the important factors for scalable A/B tests.
Surveys are the most widely used method for doing shopper marketing A/B tests. Therefore we wanted to compare the experience between surveys and virtual shopping, by asking 400 US shoppers a simple question:
“How would you compare this virtual shopping interview to a typical questionnaire based survey?”
~70% of shoppers find the virtual shopper experience more enjoyable than a typical survey!
Some open ended answers feedback:
...never done that kinda things before but it was definitely interesting
...it worked great actually better than other shopping app surveys
It looks brilliant!
Very different and fun
Yes, this is amazing
No issue...very smooth
Have more of these type of surveys!