Eye Movement analysis with HMMs (EMHMM)

fixeg

Recent research has reported substantial individual differences in eye movement patterns in cognitive tasks. Thus, it is important to take these individual differences into account in eye movement data analysis. In this project we use hidden Markov models (HMM) to analyze eye movement data.  In our approach, each individual’s eye movements are modeled with an HMM, including both person-specific regions of interests (ROIs) and transitions among the ROIs. Individual HMMs are then clustered to discover common patterns among individuals.  The similarities between an individual’s eye movement patterns and the group behavior can be quantitatively assessed (using likelihood), and correlated with other behavioral data (e.g., recognition accuracy).

Through clustering individuals’s HMMs, our approach finds two common patterns in face recognition: holistic (looking mostly at the face center) and analytic (looking mostly at the two eyes in addition to the mouth).

Holistic
holistic
Analytic
analytic

The frequency of participants adopting the two patterns did not differ significantly between Asians and Caucasians. Significantly more participants showed similar eye movement patterns when viewing own- and other-race faces than different patterns, suggesting little modulation from culture. In contrast to the scan path theory, which posits that eye movements during learning have to be recapitulated during recognition for the recognition to be successful, participants who used the same or different patterns during learning and recognition did not differ in recognition performance. Interestingly, analytic patterns were associated with better face recognition performance and higher activation in brain regions important for top-down control of visual attention, whereas holistic patterns were associated with aging and lower cognitive status in older adults.

HLLvsFAR
HLLvsMOCA

We have also applied similar models to analyze eye movements on webpages in order to discover differences in information system usage among experienced/inexperienced and pressure/non-pressured users.

EMHMM with Co-clustering

The above approach assumes that the feature layouts of the stimuli are similar to each other (e.g., the faces are aligned), so that aggregating fixations across the images makes sense.  In some experiment designs, this is not possible, e.g., in scene viewing experiments where each image has a different layout. To handle images with different layouts, we apply co-clustering to group individuals that share similar eye gaze patterns among all images. With co-clustering, representative HMMs are estimated for each image separately, and individuals are grouped together if they have similar eye gaze (on each image) to other group members. Here is an illustration:

Each subject Si has an HMM to summarize their eye movements on each stimuli Ij. Here the ellipses represent the ROIs. Co-clustering will group subjects whose eye movement patterns are similar to one another when viewing each stimuli. In this toy example, S1 and S2 are grouped together because they have similar eye movements when looking at I1, I2, and I3. Similarly, S3 and S4 are grouped together.

We applied the co-clustering algorithm to a scene-viewing experiment, and discover two strategies, Explorative and Focused. Here are examples of the two groups for two different stimuli.

The explorative group tends to look at the center of the Koala’s face, while the focused group looks more at the eye region.  Once we have the group HMMs, the similarity of each subject’s eye movements to the two groups can be quantified and used for further analysis.

Downloads

This is the MATLAB toolbox for analyzing eye movement data using hidden Markov models. It includes code for learning HMMs for individuals, clustering individuals’ HMMs into groups, and co-clustering of HMMs into groups. There are also functions for performing statistical tests and analyzing the HMMs.

Research Labs

Publications

Journals

Peer-Reviewed Conference Proceedings / Poster Sessions

Invited Talks/Conference Oral Presentations

  • Deep Neural Net + Hidden Markov Model (DNN + HMM): A Novel Framework for understanding human learning.
    Janet H. Hsiao,
    CogSci 2022 Hong Kong Meetup & Symposium, July 2022.
  • Eye Movement analysis with Hidden Markov Models (EMHMM) and Its Applications in Cognitive Research.
    Antoni B. Chan,
    CogSci 2021 Hong Kong Meetup & Symposium, July 2021.
  • Mini-Course in Eye Movement Analysis With Hidden Markov Models.
    Janet H. Hsiao and Antoni B. Chan,
    National Taiwan University, Aug 12-16, 2019.
  • EMHMM: Eye Movement Analysis with Hidden Markov Models and Its Applications in Cognitive Research.
    Janet H. Hsiao and Antoni B. Chan,
    Tutorial in The Annual Meeting of the Cognitive Science Society (CogSci), July 2019.
  • Understanding Eye Movement Patterns in Face Recognition Using Hidden Markov Models.
    Janet H. Hsiao,
    Symposium on Understanding Individual Differences in Eye Movement Patterns, Asia Pacific Conference on Vision, July 2017.
  • Classifying eye gaze patterns and inferring individual preferences using hidden Markov models.
    Antoni B. Chan,
    Symposium on Understanding Individual Differences in Eye Movement Patterns, Asia Pacific Conference on Vision, July 2017.
  • Understanding eye movement patterns in face recognition using hidden Markov models.
    Janet H. Hsiao, Antoni B. Chan,
    Invited Department Seminar at the Department of Psychology, National Taiwan University, 2017.
  • On Associations Between Eye Movement Patterns and Face Recognition Performance: The Effects of Culture and Age.
    Janet H. Hsiao, Antoni B. Chan, Tim Chuk, Tatia M.C. Lee, Cynthia Y.H. Chan,
    Invited talk at the German Graduate School of Management and Law (GGS), 2016.
  • Eye Movement Pattern in Face Recognition is Associated with Cognitive Decline in the Elderly.
    Janet H. Hsiao, Cynthia Y.H. Chan, Antoni B. Chan, Tatia M.C. Lee,
    Invited talk at the HKU International Alzheimer’s Disease Conference, 2015.
  • Eye Movement Pattern in Face Recognition is Associated with Cognitive Decline in the Elderly.
    Janet H. Hsiao, Cynthia Y.H. Chan, Antoni B. Chan, Tatia M.C. Lee,
    Invited talk at the Department of Information Systems and Information Economics, Goethe University Frankfurt, 2015.
  • Understanding eye movements in face recognition with hidden Markov model.
    Janet H. Hsiao, Antoni B. Chan, Tim Chuk,
    Invited talk at the 6th Chinese International Conference on Eye Movements (CICEM), Beijing, China, 2014.

Acknowledgements

We are grateful for support from the Research Grant Council of Hong Kong SAR: #17609117, #17402814 and HKU 745210H for J.H. Hsiao; CityU 110513 and G-CityU109/14 for A.B. Chan. We also thank the HKU Seed Funding Programme for Basic Research (Project numbers 201311159131 and 201811159165) to J.H. Hsiao.  We also thank the Strategic Research Grant from City University of Hong Kong (Project No. 7005218) to A.B. Chan.