Test Campaign Result Accuracy - Test Group Sizing - Part II
An approach to size test groups for a campaign has been presented in [Test campaign result accuracy - test group sizing - Part I].
However, how can one be sure that the size of the customer group used (the sample), is sufficient to provide statistically accurate results.
Having carried out a test campaign on a sample (customer group) of a given size, one can estimate the range of the expected response rate.
If the test campaign has been run on a group of size N and the response rate measured was p, the standard deviation is calculated by the following formula SEP = SQRT(p*(1-p) /N) .
For example if the group size N has been 60 thousand and the response rate p was 4 %, then the standard deviation (SEP) is 0,08%.
This means that one can be 68% confident that the response rate will range between 3,92% and 4,08% (within one standard deviation) or 95% confident that it will range between 3,84% and 4,16%.
(the confidence level of 95% is the probability to fall within the response rate range and is found approximately 2 standard deviations from the mean).
As can be understood by the formula above, the larger the size N of the group, the smaller the standard deviation and the narrower the confidence internal.
Using larger groups (larger samples) leads to higher confidence in the evaluation made.
Up to this point, we have discussed the case of a single test group.
What if a test campaign aims at the comparative evaluation of two alternative customer selection models in order to identify the model which has higher predictive power.
In that case, the test yields two response rates of the two test groups used.
If the response rates differ substantially than the conclusion is clear.
However if the two results are located close to each other, then the confidence intervals may overlap, thus leading to no clear conclusion.
In that case, one should calculate the confidence interval of each group in order to check if the two overlap.
However, how can one be sure that the size of the customer group used (the sample), is sufficient to provide statistically accurate results.
Having carried out a test campaign on a sample (customer group) of a given size, one can estimate the range of the expected response rate.
If the test campaign has been run on a group of size N and the response rate measured was p, the standard deviation is calculated by the following formula SEP = SQRT(p*(1-p) /N) .
For example if the group size N has been 60 thousand and the response rate p was 4 %, then the standard deviation (SEP) is 0,08%.
This means that one can be 68% confident that the response rate will range between 3,92% and 4,08% (within one standard deviation) or 95% confident that it will range between 3,84% and 4,16%.
(the confidence level of 95% is the probability to fall within the response rate range and is found approximately 2 standard deviations from the mean).
As can be understood by the formula above, the larger the size N of the group, the smaller the standard deviation and the narrower the confidence internal.
Using larger groups (larger samples) leads to higher confidence in the evaluation made.
Up to this point, we have discussed the case of a single test group.
What if a test campaign aims at the comparative evaluation of two alternative customer selection models in order to identify the model which has higher predictive power.
In that case, the test yields two response rates of the two test groups used.
If the response rates differ substantially than the conclusion is clear.
However if the two results are located close to each other, then the confidence intervals may overlap, thus leading to no clear conclusion.
In that case, one should calculate the confidence interval of each group in order to check if the two overlap.