.png){kind=icon}
.png){kind=icon}
Return
Asymmetric Multi-Task Learning for Interpretable Gaze-Driven Grasping Action Forecasting
.png){kind=icon}
OA
AI
DOI：10.1109/JBHI.2024.3430810.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
This work tackles the automatic prediction of grasping intention of humans observing their environment. Our target application is the assistance to people with motor disabilities and potential cognitive impairments, using assistive robotics. Our proposal leverages the analysis of human attention captured in the form of gaze fixations recorded by an eye-tracker on the first person video, as the anticipation of prehension actions is a well studied and well known phenomenon. We propose a multi-task system that simultaneously addresses the prediction of human attention in the near future, and the anticipation of grasping actions. Visual attention is modeled as a competitive process between a discrete set of states, each one associated to a well-known gaze movement pattern from visual psychology. We additionally consider an asymmetric multi-task problem, where attention modeling is an auxiliary task that helps to regularize the learning process of the main action prediction task, and propose a constrained multi-task loss that naturally deals with this asymmetry. Our model shows superior performance than other losses for dynamic multi-task learning, current dominant deep architectures for general action forecasting and particularly-tailored models for predicting grasping intention. In particular, it provides state-of-the-art performance in three datasets for egocentric action anticipation, with an average precision of 0.569 and 0.524 in GITW and Sharon datasets, respectively, and an accuracy of 89.2% and a success rate of 51.7% in Invisible dataset.
Keywords:
Task analysis
Predictive models
Visualization
Multitasking
Grasping
Forecasting
Hidden Markov models
Constrained loss
grasping action forecasting
interpretable attention
multi-task learning
Journal
IF:
6.8
Papers:
4.5K
Citations:
2.0W
View more organizations