Tony Dang I want you to act as a research paper summarizer. I will provide you with a research paper on a specific topic, and you will create a summary of the main points and findings of the paper. Your summary should be concise and should accurately and objectively communicate the key points of the paper. You should not include any personal opinions or interpretations in your summary, but rather focus on objectively presenting the information from the paper. Your summary should be written in your own words and should not include any direct quotes from the paper. Please ensure that your summary is clear, concise, and accurately reflects the content of the original paper.

Abstract An observer’s eye movements are often informative about how the observer interacts with and processes a visual stimulus. Here, we are specifically interested in what eye movements reveal about how the content of information visualizations is processed. Conversely, by pooling over many observers’ worth of eye movements, what can we learn about the general effectiveness of different visualizations and the underlying design principles employed? The contribution of this manuscript is to consider these questions at a large data scale, with thousands of eye fixations on hundreds of diverse information visualizations. We survey existing methods and metrics for collective eye movement analysis, and consider what each can tell us about the overall effectiveness of different information visualizations and designs at this large data scale. 1 Introduction Eye movements can provide us with clues about the elements of a visual display that people pay attention to, what they spend most time on, and how they redirect their attention between elements. The eyes can also be used as indicators of higher-level cognitive processing like memory, comprehension, and problem solving [21, 23, 32, 39, 40, 54]. Z. Bylinskii () • A. Oliva Computer Science and Artificial Intelligence Lab, Massachusetts Institute of Technology, 32 Vassar St., Boston, MA, USA e-mail: zoya@mit.edu; oliva@mit.edu M.A. Borkin College of Computer and Information Science, Northeastern University, 360 Huntington Ave., Boston, MA, USA e-mail: m.borkin@neu.edu N.W. Kim • H. Pfister School of Engineering & Applied Sciences, Harvard University, 33 Oxford Street, Boston, MA, USA e-mail: namwkim@seas.harvard.edu; pfister@seas.harvard.edu © Springer International Publishing AG 2017 M. Burch et al. (eds.), Eye Tracking and Visualization, Mathematics and Visualization, DOI 10.1007/978-3-319-47024-5_14 235 236 Z. Bylinskii et al. Eye movement analyses have been used to study the perception of natural scenes, simple artificial stimuli, webpages, user interfaces, and increasingly, infor- mation visualizations. In human-computer interaction (HCI), eye tracking has often been used for evaluating the usability of systems and studying the related question of interface design [13, 19, 29, 47]. Duchowski provides a survey of different eye-tracking applications in domains ranging from industrial engineering to marketing [13]. In the visualization community, eye-tracking analyses have been used to inde- pendently evaluate different visualizations (e.g., graphs [25–27, 39, 49], node-link diagrams [1], tree diagrams [8], parallel coordinates [62]) and to directly compare visualization types [6, 11, 17]. Eye tracking has also been used to understand how a person visually perceives, explores, searches, and remembers a visualization, providing a window into the cognitive processes involved when interacting with visualizations [1, 3, 6, 11, 26, 37, 49, 50, 53]. Information visualizations are specifically designed to be parsed and understood by human observers. Visualizations can be created to help convey a specific message to a general audience, or to help data analysts extract trends and meaning from the data. As visualizations are amenable to specific tasks, observer performance on those tasks can be directly measured (e.g., ability to find a specific piece of information, to solve an analysis task, to remember the content for later retrieval, etc.). Eye movement analyses can then be used to provide possible explanations of task performance (e.g., why a task was completed quicker with one visualization design as compared to another), as complementary performance measurements that take into account human perception. Eye movements can provide a window into the cognitive processing taking place when an observer examines an information visualization. Although different eye movement metrics have been previously reviewed within the context of different tasks [1, 17, 29, 51], in this manuscript we focus specifically on eye fixation metrics that can be used for collective analysis (the aggregation of data across a population of observers and visualizations) of information visualiza- tion designs. We provide a review of metrics that can be used for the quantitative comparison of different visualization designs in a large data setting. Unlike many previous studies, our analyses are broad, spanning a large diversity of visualization types and sources. We discuss and visualize ways in which different metrics can be used to evaluate the effectiveness of different visualization designs, and we use the MASSVIS dataset [6] to provide some specific examples. The review provided in this manuscript is intended to motivate further research into large-scale eye movement analysis for the broad comparison and evaluation of visualization designs. Eye Fixation Metrics for Large Scale Evaluation 237 2 Methods 2.1 Visualization Data We used the MASSVIS dataset of 393 labeled target visualizations,1 spanning four different source categories: government and world organizations, news media, infographics, and scientific publications [6]. These visualizations were manually labeled using the LabelMe system [59] and Borkin et al.’s visualization taxon- omy [7] (Fig. 1a). Labels classify visualization elements as: data encoding, data- related components (e.g., axes, annotations, legends), textual elements (e.g., title, axis labels, paragraphs), pictograms or human recognizable objects, or graphical elements with no data encoding function. Labels can overlap in that a single region can have a number of labels (e.g., an annotation on a graph has an annotation label and a graph label). Labels are available for analyses as segmented polygons. 2.2 Eye-tracking Experiments We used eye movements collected during the encoding experimental phase from the study by Borkin et al. [6]. During this phase, each visualization was shown to participants for 10 s, producing an average of 37.4 (SD: 3.2) eye fixations per visualization, or an average 623 (SD: 93) total fixations per visualization. This duration proved to be of sufficient length for a participant to read the visualization’s title, axes, annotations, etc., as well as explore the data encoding, and short enough to avoid too much redundancy in fixation patterns and explorative strategies. Participants were told to remember as many details of each visualization as possible for subsequent experimental phases. During the recognition and recall phases, respectively, participants completed a memory task and were asked to write descriptions of the visualizations they remembered. We do not directly use this additional data in the present manuscript, but refer to the conclusions made from the eye movement analyses in the context of memory performances. Eye movements of 33 participants were recorded on 393 target visualizations, with an average of 16.7 viewers (SD: 1.98) per visualization. Equipment included an SR Research EyeLink1000 desktop eye-tracker [63] with a chin-rest mount 22 in from a 19 in CRT monitor (1280 1024 pixels). For each eye fixation, available for analysis are its spatial location in pixel coordinates, duration in milliseconds, and ordering within the entire viewing episode (scanpath). 1Dataset available at http://massvis.mit.edu. 238 Z. Bylinskii et al. Fig. 1 We plot the fixations of a single observer for demonstration purposes, to visually depict a few key terms used throughout this manuscript. (a) The images we use are labeled with AOIs (Areas of Interest), which are elements like the title, axes, and legend. (b) Fixations are the discrete locations that an observer’s eyes have landed on at some point during the viewing episode. (c) Multiple consecutive fixations that land on the same AOIs of an image can be further clustered into gazes. The size of the gaze marker is proportional to the number of fixations making up the gaze, with the marker centered at the mean of those fixation locations. (d) A scanpath is the sequence of fixations made. Here, to denote the temporal ordering, fixations are connected by lines, numerically labeled, and colored such that the earliest are in red and the latest in yellow 2.3 Metrics and Visualizations Depending on the analysis being performed, different aspects of eye movement behavior can be measured including fixation locations, fixation durations, and saccades.2 Fixations are discrete samples of where an eye was looking on a visual display obtained from continuous eye movement data3 (Fig. 1b). By segmenting the visual stimulus into elements or Areas of Interest, denoted AOI, fixations falling on different AOIs can be separately analyzed (Fig. 1a). Consecutive fixations on a specific region or AOI can be further clustered into gazes (Fig. 1c). 2Saccades are intervals between fixations: the motion of the eyes from one fixation point to the next. The analysis of saccades is beyond the scope of the present manuscript, for which additional metrics would be necessary [40, 51]. 3The eye has to be recorded as “still” according to prespecified parameters [24, 60]. We use the standard thresholds set by the EyeLink Eyetracker [63]. Eye Fixation Metrics for Large Scale Evaluation 239 Apart from summarizing the number and duration of fixations on a visual design or its constituent elements, the spatial and sequential aspects of a viewing episode can be used to compute additional measurements of eye movement behavior for visual design analysis. For instance, the spatial distribution of fixations can be captured by the moments of the distribution or the coverage (proportion of visual design fixated at a particular threshold value, Sect. 3.3). The temporal ordering (sequence) of fixations is often referred to as the scanpath [45] and is common for analyzing search tasks (Fig. 1d). For instance, one can consider the sequence of AOIs observers fixate while searching for a target or a specific piece of information. Quantitative eye movement measurements used by previous visualization studies are summarized in Table 1. A review of the most common eye measurements across usability studies more generally is provided by Jacob and Karn [29]. The 5 most common metrics reported across 24 usability studies also appear in Table 1. Different metrics emphasize different aspects of eye movement behavior, which are in turn linked to different underlying cognitive processes. The number or density of fixations allocated to a visual area has been linked to its importance [29, 52]; fixation duration in a visual area has been linked to the area’s information content or complexity [32]; and the transitions between fixations have been found to be related to the search behavior and expectations of the viewer [15, 44, 54]. Patterns in the fixation data of a group of observers can also be used to highlight design Table 1 Eye movement metrics commonly reported in usability studies [29] and for evaluation and comparison of information visualizations. Different perception studies have used these metrics to make conclusions about the importance and noticeability of different visual elements, and to reason about the difficulty of the perception task and the complexity of the visual design [51]. AOI refers to an Area of Interest, which can be a component of a graph like the title, axis, or legend Visualization Quantitative measurements studies Possible interpretations Summary measurements Total number of fixationsa [17, 39] Efficiency of searching or engagement [12, 19, 32] Total number of gazes [11] Complexity of inferential process [11] Mean fixation durationa Complexity or engagement [32] AOI measurements Fixations on AOIsa (propor- tion or number) [8, 37, 62] Element importance or noticeability [52] Gazes on AOIsa (proportion or number) [11] Element importance or noticeability [29] Viewing time on AOIsa (pro- portion or total) [11, 37, 62] Information content, complexity, or engagement [32] Time to first fixation on an AOI [17, 39, 62] Attention-getting properties [10] Mostly qualitative analysis [25–27, 49, 53] Relative complexity or efficiency of dif- ferent designs aThe marked metrics are the 5 most commonly-reported across a total of 24 usability studies surveyed by Jacob and Karn [29] 240 Z. Bylinskii et al. features or diagnose potential problems. For instance, the order of fixations has been found to be indicative of the efficiency of the arrangement of visual elements [15]. A visualization designer might be interested in ensuring that the important elements are more likely to be fixated early. The use of different types of visualizations for highlighting properties of eye movement data have also been useful for complementing and facilitating analysis over groups of observers [1, 18, 41, 57, 64, 66, 68, 69]. A number of previous visualization studies relied mostly on such qualitative analyses (Table 1). Blascheck et al. provide a review of visualizations and visual analytics tools for eye movement data [3]. While visualizations can facilitate data exploration, inferences made from eye movement data are more meaningful when supported by quantitative metrics. For the explorative analysis of the MASSVIS eye movement data, we utilize fixation heatmaps due to their versatility, scalability, and interpretability. Fixation heatmaps are constructed by aggregating a set of fixations and placing a Gaussian4 at each fixation location. The result is a continuous distribution that can be plotted on top of the image to highlight elements receiving the most attention. This simple visu- alization is particularly amenable to collective analysis, allowing us to visualize the fixations of any number of observers on a single image. To highlight different trends in the eye movements, we aggregate over different subsets of the data: distinct fixa- tion durations (Fig. 2), time points during the viewing episode (Fig. 3), and observers (Fig. 4). Our coverage plots are also just thresholded fixation heatmaps (Fig. 5). We note that eye movement analyses are most informative in the context of an objective task that an observer performs. In such cases, eye movements are more likely to be related to task completion itself. Furthermore, eye movement analyses can be used to complement, and provide possible explanations for, other objective performance measurements (e.g., speed or accuracy of task completion). Considered in isolation, eye movement measurements can be open to interpretation, and thus they should complement, not replace, other measurements. For example, the eye movements from the MASSVIS dataset were collected in the context of memory and recall tasks. Participants’ fixations were recorded as they examined visualizations, knowing they would have to retrieve the details from memory later. In this manuscript, our focus is on the eye movement metrics themselves and how they can be used for the evaluation and comparison of information visualizations more broadly. We use the MASSVIS dataset for demonstrative examples.

Give me a bullet point summary of the research methods.