Eye Movement analysis with HMMs (EMHMM)

fixeg

Recent research has reported substantial individual differences in eye movement patterns in cognitive tasks. Thus, it is important to take these individual differences into account in eye movement data analysis. In this project we use hidden Markov models (HMM) to analyze eye movement data.  In our approach, each individual’s eye movements are modeled with an HMM, including both person-specific regions of interests (ROIs) and transitions among the ROIs. Individual HMMs are then clustered to discover common patterns among individuals.  The similarities between an individual’s eye movement patterns and the group behavior can be quantitatively assessed (using likelihood), and correlated with other behavioral data (e.g., recognition accuracy).

Through clustering individuals’s HMMs, our approach finds two common patterns in face recognition: holistic (looking mostly at the face center) and analytic (looking mostly at the two eyes in addition to the mouth).

Holistic
holistic
Analytic
analytic

The frequency of participants adopting the two patterns did not differ significantly between Asians and Caucasians. Significantly more participants showed similar eye movement patterns when viewing own- and other-race faces than different patterns, suggesting little modulation from culture. In contrast to the scan path theory, which posits that eye movements during learning have to be recapitulated during recognition for the recognition to be successful, participants who used the same or different patterns during learning and recognition did not differ in recognition performance. Interestingly, analytic patterns were associated with better face recognition performance and higher activation in brain regions important for top-down control of visual attention, whereas holistic patterns were associated with aging and lower cognitive status in older adults.

HLLvsFAR
HLLvsMOCA

We have also applied similar models to analyze eye movements on webpages in order to discover differences in information system usage among experienced/inexperienced and pressure/non-pressured users.

Downloads

This is the MATLAB toolbox for analyzing eye movement data using hidden Markov models. It includes code for learning HMMs for individuals, as well as clustering indivduals’ HMMs into groups.

Research Labs

Publications

Journals

  • Hidden Markov model analysis reveals the association between eye movement patterns and face recognition performance across cultures.
    Tim Chuk, Kate Crookes, William G. Hayward, Antoni B. Chan, and Janet H. Hsiao,
    Cognition, to appear 2017.
  • Is having similar eye movement patterns during face learning and recognition beneficial for recognition performance? Evidence from hidden Markov modeling.
    Tim Chuk, Antoni B. Chan, and Janet H. Hsiao,
    Vision Research, to appear, 2017.
  • Scanpath modeling and classification with Hidden Markov Models.
    Antoine Coutrot, Janet H. Hsiao, and Antoni B. Chan,
    Behavior Research Methods, to appear.
  • Understanding eye movements in face recognition using hidden Markov models.
    Tim Chuk, Antoni B. Chan, and Janet H. Hsiao,
    Journal of Vision, 14(11):8, Sep 2014.

Peer-Reviewed Conference Proceedings / Poster Sessions

Invited Talks/Conference Oral Presentations

  • Understanding Eye Movement Patterns in Face Recognition Using Hidden Markov Models.
    Janet H. Hsiao,
    Symposium on Understanding Individual Differences in Eye Movement Patterns, Asia Pacific Conference on Vision, July 2017.
  • Classifying eye gaze patterns and inferring individual preferences using hidden Markov models.
    Antoni B. Chan,
    Symposium on Understanding Individual Differences in Eye Movement Patterns, Asia Pacific Conference on Vision, July 2017.
  • Understanding eye movement patterns in face recognition using hidden Markov models.
    Janet H. Hsiao, Antoni B. Chan,
    Invited Department Seminar at the Department of Psychology, National Taiwan University, 2017.
  • On Associations Between Eye Movement Patterns and Face Recognition Performance: The Effects of Culture and Age.
    Janet H. Hsiao, Antoni B. Chan, Tim Chuk, Tatia M.C. Lee, Cynthia Y.H. Chan,
    Invited talk at the German Graduate School of Management and Law (GGS), 2016.
  • Eye Movement Pattern in Face Recognition is Associated with Cognitive Decline in the Elderly.
    Janet H. Hsiao, Cynthia Y.H. Chan, Antoni B. Chan, Tatia M.C. Lee,
    Invited talk at the HKU International Alzheimer’s Disease Conference, 2015.
  • Eye Movement Pattern in Face Recognition is Associated with Cognitive Decline in the Elderly.
    Janet H. Hsiao, Cynthia Y.H. Chan, Antoni B. Chan, Tatia M.C. Lee,
    Invited talk at the Department of Information Systems and Information Economics, Goethe University Frankfurt, 2015.
  • Understanding eye movements in face recognition with hidden Markov model.
    Janet H. Hsiao, Antoni B. Chan, Tim Chuk,
    Invited talk at the 6th Chinese International Conference on Eye Movements (CICEM), Beijing, China, 2014.