User study

The previous chapters described how to develop technology. This chapter will explain how to evaluate technology. In previous chapters on AI (Chapter 3), computer vision (Chapter 4) and information retrieval (Chapter 5), we came across a few evaluation metrics like precision, recall and classification accuracy. However, when we evaluate human–machine interaction, the human element adds new challenges. Humans are neither fully deterministic nor random. We are (or at least hope to be) rational, meaning we try to do what we are supposed to do. However, we get ‘used to’ something or get ‘bored’ with something. Even after months of training, sometimes people can exceed expectations and similarly commit mistakes. In this chapter, we will learn to analyze interactions considering the variability in human activities. All of us are aware of collecting data and comparing means, but we will learn to compare means along with variance. The chapter is by no means a replacement for a standard statistics course, but we will mainly highlight how to design user studies, preprocess data, select appropriate statistical hypothesis tests and report results. The statistical tests are not specific to any statistical package, but once one prepares an appropriate data table, any software can be used to analyze results.