SpatioTemporal Reconstruction of Interacting People for pErceiving Systems

Descrizione del progetto

Interazione uomo-oggetto per sistemi più intelligenti

La percezione dell’interazione uomo-oggetto (HOI) è essenziale per i sistemi intelligenti che assistono le persone nell’esecuzione di compiti. Tuttavia, l’accurata percezione di queste interazioni da immagini a colori del corpo intero presenta delle difficoltà dovute a occlusioni, sfocature da movimento, ambiguità di profondità e dettagli a bassa risoluzione. I precedenti tentativi si sono concentrati sulla valutazione separata del corpo umano o dell’oggetto, spesso ignorando aspetti cruciali quali i dettagli della mano o del corpo. Per ovviare a queste difficoltà, il progetto STRIPES, finanziato dal CER, svilupperà modelli 3D sia per gli oggetti che per gli esseri umani, consentendo la stima dell’HOI da immagini e video a colori naturali. Nei prossimi 5 anni, STRIPES migliorerà la ricostruzione dell’HOI, consentendo a robot assistivi e assistenti virtuali di affiancare le persone nelle attività quotidiane in modo più efficace.

Obiettivo

People constantly interact with objects to perform tasks. To help people accomplish these, computers need to perceive Human-Object Interactions (HOI), and for this, they need to reconstruct HOI from whole-body color images of people interacting with objects or scenes. This is challenging, due to the occlusions between bodies and objects, motion blur, depth ambiguities, and the low image resolution of hands and graspable object parts. There has been significant prior work on estimating 3D humans without considering objects, and estimating 3D objects without considering humans. Little prior work estimates these jointly, but, for tractability, focuses either on interacting hands, ignoring the body, or on interacting bodies, ignoring hands. Only recent work addresses dexterous interaction of whole bodies, but instruments bodies with intrusive markers or sensors, and uses non-standard cameras to capture video of interactions. Moreover, reconstruction lacks hand detail that is crucial for grasping, and videos are captured in constrained settings, consequently, methods trained on these struggle generalizing. My goal is to infer HOI from natural whole-body images/videos. To this end, I present an ambitious 5-year research agenda with novelties in four directions: (1) developing strong generative 3D shape models for objects and humans for a novel HOI representation; (2) developing methods that estimate 3D HOI from a color image with rich contact and proximal awareness; (3) instilling spatiotemporal reasoning into the heart of these for estimating 4D HOI from color video; and (4) extending these methods to also infer their own confidence that will be correlated with the reconstruction quality. The outcome will be novel and robust methods for HOI reconstruction from natural images/videos. This will fill an important gap, enabling future intelligent systems to amplify people’s skills and help them accomplish tasks, e.g. for assistive robots or virtual 3D assistants or trainers.