Automating The Evaluation Of Web Site User Experience: Collecting Qualitative/quantitative Ux
This is the 4th & final part of Automating the Evaluation of Web Site User Experience: Collecting Qualitative/Quantitative UX. In this post will expand on the Balancing Data Quantity and Analysis Quality.
Balancing Data Quantity and Analysis Quality
A WebEffective user experience test contains a collection of sessions, and each session contains a collection of pages. Sessions can be analyzed individually at a very high level of detail, including VCR-style replay of the exact pages each visitor saw. Alternatively, session data can be aggregated and analyzed for patterns using analytic tools such as path models and page exit summaries. All analyses can be done after the data has
been collected because Keynote preserves the data streams that constitute the original sessions. Thus, reconstructing and exploring usability data in depth is always possible when new insights occur. WebEffectives unique process of HTML redirection addresses critical issues in participant sampling. For example, a typical usability test implementation automatically redirects a group of visitors who come to a Web site home page
through Keynote. The sample is selected randomly using a simple script with a random number generator.
Because the sample is drawn randomly from the overall Web visitor population, it is a perfectly representative sample of the total target audience. Making this assumption allows researchers to generalize from the sample to apply the results to real users
Individuals within any sample will have differences that make them unique, and to the extent they are unique, they do not represent the group as whole. Hence, it is imperative to observe a sample that is sufficiently large that individual variations become statistical noise relative to the underlying pattern. As an extreme example, imagine doing a before-and-after test with just two participants, one in each group. All of the variation in
their behavior or stated opinions could be attributed to personal differences instead of changes to the Web site
Measurement effects also can be removed completely from the data collection process using WebEffective.
Participants use their browsers as they normally do without any modifications to their computer system or
browser. This ensures that their intentions at the Web site and behavior are natural, not the result of special
guidance or observation
Depending on the degree of measurement precision required, several hundred observations may be necessary to make statistically valid comparisons between groups, with past observations, or with set standards and goals. Numbers of this magnitude are far beyond the time and financial resources available to most conventional usability labs. But, the automated session recording enabled by WebEffective gives researchers the hundreds of sessions needed to empirically estimate the importance of various Web design issues. When collecting hundreds or thousands of sessions in a sample, the method of collection is crucial to the final quality of the data gathered for analysis. Partial tracking of where users go can be done by inserting beacons at various points in a Web site, but recording the actual session content with WebEffective explains how visitors transitioned between points and what they experienced in between. User experience researchers are also interested in what visitors think.
Conclusions:
Automating user experience testing of any Web site is a difficult challenge at best, especially in light of the need to balance data quality and quantity: gathering rich data is essential to deriving meaning and understanding, and a sufficient quantity of data is essential to making findings valid and statistically significant.
Technological innovation has begun to eliminate the need to sacrifice either of these important data characteristics. With its user experience assessment service WebEffective, Keynote has made significant strides in automating the collection of data that researchers need and in giving researchers the right tools to evaluate large quantities of exceptionally rich data. Such new data gathering techniques are ushering in a new approach to understanding user interaction with Web sites and the effectiveness of e-business systems
Balancing Data Quantity and Analysis Quality
A WebEffective user experience test contains a collection of sessions, and each session contains a collection of pages. Sessions can be analyzed individually at a very high level of detail, including VCR-style replay of the exact pages each visitor saw. Alternatively, session data can be aggregated and analyzed for patterns using analytic tools such as path models and page exit summaries. All analyses can be done after the data has
been collected because Keynote preserves the data streams that constitute the original sessions. Thus, reconstructing and exploring usability data in depth is always possible when new insights occur. WebEffectives unique process of HTML redirection addresses critical issues in participant sampling. For example, a typical usability test implementation automatically redirects a group of visitors who come to a Web site home page
through Keynote. The sample is selected randomly using a simple script with a random number generator.
Because the sample is drawn randomly from the overall Web visitor population, it is a perfectly representative sample of the total target audience. Making this assumption allows researchers to generalize from the sample to apply the results to real users
Individuals within any sample will have differences that make them unique, and to the extent they are unique, they do not represent the group as whole. Hence, it is imperative to observe a sample that is sufficiently large that individual variations become statistical noise relative to the underlying pattern. As an extreme example, imagine doing a before-and-after test with just two participants, one in each group. All of the variation in
their behavior or stated opinions could be attributed to personal differences instead of changes to the Web site
Measurement effects also can be removed completely from the data collection process using WebEffective.
Participants use their browsers as they normally do without any modifications to their computer system or
browser. This ensures that their intentions at the Web site and behavior are natural, not the result of special
guidance or observation
Depending on the degree of measurement precision required, several hundred observations may be necessary to make statistically valid comparisons between groups, with past observations, or with set standards and goals. Numbers of this magnitude are far beyond the time and financial resources available to most conventional usability labs. But, the automated session recording enabled by WebEffective gives researchers the hundreds of sessions needed to empirically estimate the importance of various Web design issues. When collecting hundreds or thousands of sessions in a sample, the method of collection is crucial to the final quality of the data gathered for analysis. Partial tracking of where users go can be done by inserting beacons at various points in a Web site, but recording the actual session content with WebEffective explains how visitors transitioned between points and what they experienced in between. User experience researchers are also interested in what visitors think.
Conclusions:
Automating user experience testing of any Web site is a difficult challenge at best, especially in light of the need to balance data quality and quantity: gathering rich data is essential to deriving meaning and understanding, and a sufficient quantity of data is essential to making findings valid and statistically significant.
Technological innovation has begun to eliminate the need to sacrifice either of these important data characteristics. With its user experience assessment service WebEffective, Keynote has made significant strides in automating the collection of data that researchers need and in giving researchers the right tools to evaluate large quantities of exceptionally rich data. Such new data gathering techniques are ushering in a new approach to understanding user interaction with Web sites and the effectiveness of e-business systems