Jacob O. Wobbrock
Leah Findlater
Darren Gergle
ABSTRACT
Nonparametric data from multi-factor experiments arise often in human-computer interaction (HCI). Examples may include error counts, Likert responses, and preference tallies. But because multiple factors are involved, common nonparametric tests (e.g., Friedman) are inadequate, as they are unable to examine interaction effects. While some statistical techniques exist to handle such data, these techniques are not widely available and are complex. To address these concerns, we present the Aligned Rank Transform (ART) for nonparametric factorial data analysis in HCI. The ART relies on a preprocessing step that "aligns" data before applying averaged ranks, after which point common ANOVA procedures can be used, making the ART accessible to anyone familiar with the F-test. Unlike most articles on the ART, which only address two factors, we generalize the ART to N factors. We also provide ARTool and ARTweb, desktop and Web-based programs for aligning and ranking data. Our re-examination of some published HCI results exhibits advantages of the ART.
- Conover, W. J. and Iman, R. L. (1981). Rank transformations as a bridge between parametric and nonparametric statistics. The American Statistician 35 (3), 124--129.Google Scholar
- Fawcett, R. F. and Salter, K. C. (1984). A Monte Carlo study of the F test and three tests based on ranks of treatment effects in randomized block designs. Communications in Statistics: Simulation and Computation 13 (2), 213--225.Google ScholarCross Ref
- Findlater, L., Moffatt, K., McGrenere, J. and Dawson, J. (2009). Ephemeral adaptation: The use of gradual onset to improve menu selection performance. Proc. CHI '09. New York: ACM Press, 1655--1664. Google ScholarDigital Library
- Higgins, J. J., Blair, R. C. and Tashtoush, S. (1990). The aligned rank transform procedure. Proc. Conf. on Applied Statistics in Agriculture. Kansas State, 185--195.Google ScholarCross Ref
- Higgins, J. J. and Tashtoush, S. (1994). An aligned rank transform test for interaction. Nonlinear World 1 (2), 201--211.Google Scholar
- Hodges, J. L. and Lehmann, E. L. (1962). Rank methods for combination of independent experiments in the analysis of variance. Annals of Mathematical Statistics 33 (2), 482--497.Google ScholarCross Ref
- Kaptein, M., Nass, C. and Markopoulos, P. (2010). Powerful and consistent analysis of Likert-type rating scales. Proc. CHI '10. New York: ACM Press, 2391--2394. Google ScholarDigital Library
- Littell, R. C., Henry, P. R. and Ammerman, C. B. (1998). Statistical analysis of repeated measures data using SAS procedures. J. Animal Science 76 (4), 1216--1231.Google ScholarCross Ref
- MacKenzie, I. S. and Zhang, S. X. (1999). The design and evaluation of a high-performance soft keyboard. Proc. CHI '99. New York: ACM Press, 25--31. Google ScholarDigital Library
- Salter, K. C. and Fawcett, R. F. (1985). A robust and powerful rank test of treatment effects in balanced incomplete block designs. Communications in Statistics: Simulation and Computation 14 (4), 807--828.Google ScholarCross Ref
- Salter, K. C. and Fawcett, R. F. (1993). The art test of interaction: A robust and powerful rank test of interaction in factorial models. Communications in Statistics: Simulation and Computation 22 (1), 137--153.Google ScholarCross Ref
- Sawilowsky, S. S. (1990). Nonparametric tests of interaction in experimental design. Review of Educational Research 60 (1), 91--126.Google ScholarCross Ref
- Vermunt, J. K. (1997). Log-linear Models for Event Histories. Thousand Oaks, CA: Sage Publications.Google Scholar
- Wobbrock, J. O., Wilson, A. D. and Li, Y. (2007). Gestures without libraries, toolkits or training: A $1 recognizer for user interface prototypes. Proc. UIST '07. New York: ACM Press, 159--168. Google ScholarDigital Library
Index Terms
- The aligned rank transform for nonparametric factorial analyses using only anova procedures
Recommendations
The Levene test based-leakage assessment
AbstractThe secret information is split into several parts(multivariate) in the high-order mask. The test vector leakage assessment (TVLA) relied on Welch's t-test(T-test), the analysis of variance (ANOVA) and normalized inter-class variance (NICV) ...
Highlights- a new method that quickly and reliably evaluate the security of IT products.
- The Levene test used to in leakage assessment of side channel in high order mask.
- Compared with the original detection method, the new method can reduce ...
Analysis of Variance in Nonparametric Regression Models
In a nonparametric regression model with a multivariate explanatory variable we consider the problem of testing the hypothesis that specific interactions in a canonical decomposition of the model vanish. A simple consistent test is developed which is ...
A spreadsheet template compatible with Microsoft Excel and iWork Numbers that returns the simultaneous confidence intervals for all pairwise differences between multiple sample means
The objective of the method described in this paper is to develop a spreadsheet template for the purpose of comparing multiple sample means. An initial analysis of variance (ANOVA) test on the data returns F-the test statistic. If F is larger than the ...
Comments