Analysis of skeletal kinematics for vision-based motion capturing

Projektinformationen

SeerPredict

ID Finanzhilfevereinbarung: 957112

DOI

10.3030/957112

Projekt abgeschlossen

EK-Unterschriftsdatum 15 September 2020

Startdatum 1 Oktober 2020

Enddatum 31 März 2022

Finanziert unter

• INDUSTRIAL LEADERSHIP - Innovation In SMEs

Gesamtkosten

Keine Daten

EU-Beitrag

€ 171 187,50

Koordiniert durch

SEERVISION AG
Switzerland

Periodic Reporting for period 1 - SeerPredict (Analysis of skeletal kinematics for vision-based motion capturing)

Berichtszeitraum: 2020-10-01 bis 2022-03-31

Zusammenfassung vom Kontext und den Gesamtzielen des Projekts

Seervision builds solutions to automate video production workflows, enabling corporations and universities to create more and more high-quality live video content with minimal overhead. Our AI-driven technology detects people in the frame and creates a skeleton based on which it autonomously moves cameras to best frame and follow the on-screen talent in a variety of productions from streamed townhalls to lecture recordings. Just like camera men, we try to predict how a person will move to be able to act quickly enough and keep them framed nicely throughout the production. Interactions between on-screen talent are a key challenge for this, especially on smaller stages, because the machine learning systems can get confused and fail to smoothly keep track of the person that should be filmed.
With SeerPredict we develop novel machine learning algorithms to improve the reliability and quality of the skeleton estimation and prediction resulting in substantial improvements to the talent tracking performance and overall autonomy of Seervision’s video production solution, making high-quality live video more accessible and affordable.

Arbeit, die ab Beginn des Projekts bis zum Ende des durch den Bericht erfassten Berichtszeitraums geleistet wurde, und die wichtigsten bis dahin erzielten Ergebnisse

As part of the SeerPredict project, we have developed, implemented and validated two key improvements of Seervision’s visual perception and tracking solution:
1) We have developed a novel software module that identifies person-to-person occlusions, especially during crossings, as they occur and mitigates their effect on the quality of the extracted skeleton, enhances the robustness and smoothness of the tracking.
2) We have re-designed the architecture of Seervision’s visual perception and tracking solution, incorporating recent advances in the computer vision research literature. This has reduced the delay that needs to be compensated with skeletal prediction, yielding state-of-the-art performance. In addition, it has also reduced the overall computational overhead.
The developed software has been extensively tested under laboratory conditions, during internal productions and demos at Seervision’s in-house studio and finally with selected customers who have seen a clear improvement in the quality and performance of the talent tracking.

Fortschritte, die über den aktuellen Stand der Technik hinausgehen und voraussichtliche potenzielle Auswirkungen (einschließlich der bis dato erzielten sozioökonomischen Auswirkungen und weiter gefassten gesellschaftlichen Auswirkungen des Projekts)

The algorithms developed during this project advance the state-of-the-art of subject tracking for video production use cases. Moreover, the developed software module and improved architecture has been rolled-out into production and helps Seervision maintain its position as the state-of-the-art, premium solution. Moreover, the advances during this project enable further initiatives that will have substantial impact on Seervision’s solution and business.

Person-to-person occulsion during testing at Seervision's offices