Perspective of a real-time structural analysis tool assistant based on Computer Vision and Human Intuition

Project Information

FORSEES

Grant agreement ID: 845064

DOI

10.3030/845064

Project closed

EC signature date 27 March 2019

Start date 1 October 2019

End date 12 January 2023

Funded under

• EXCELLENT SCIENCE - Marie Skłodowska-Curie Actions

Total cost

€ 251 002,56

EU contribution

€ 251 002,56

Coordinated by

POLITECNICO DI TORINO
Italy

Periodic Reporting for period 2 - FORSEES (Perspective of a real-time structural analysis tool assistant based on Computer Vision and Human Intuition)

Reporting period: 2021-10-01 to 2022-09-30

Summary of the context and overall objectives of the project

FORSEES is project dedicated to predict (or to foresee) forces flux on simple structural arrangements only by visual analysis. Amongst human senses, vision is the most developed one, producing a wealth of information for human cognition. We decided to capitalize on visual data to develop a Computer Vision (CV) tool for Structural Engineering (SE) applications to fill the diagnostic gap in the sector with other fields of study (like medicine) and to exploit its educational potentials. The project has been tackled by multiple angles to answer to fundamental questions and to identify standalone scientific advancements: 1) does exist or can be developed a human structural intuition (a sense that identify equilibrium in a fast thinking scheme)? and 2) can it be used to interface modern CV diagnostic system for SE assessments? Finally, 3) could a system that identify and visualize forces in real time be helpful for teaching purposes and enhance equilibrium perception in engineering education? The project started by merging the state of the art on visual analyses in structural applications in a CV framework and then expanded it to a Neuroscience approach. For example, Rapid Visual Screening (RSV) is a methodology widely used in engineering for seismic vulnerability assessment of the building stock that requires squadron of engineers deployed into an area to fill a survey with building features relevant for earthquake behavior. In collaboration with Senseable City Lab (SCL), we were able to replicate RSV results and exploit the power of CV analyses, using street level imagery (i.e. Google Street View) to assess seismic vulnerability of different world. We were also able to identify what features trained engineers and general public find more relevant for safety perception. For a deeper analyses in fundamentals of structural mechanics, we investigated the possibility to reconcile methods like photoelasticity or graphic statics with CV methods and cognition. We researched the analogy with the Free Energy Principle (FEP), that defines the cognitive set of mind+brain as an inference engine, since it can be modeled through a variational equation formally identical to that of the elastic potential of solids. By limiting the equation to the visual sense only, we aimed to formalize an equivalence between the terms of entropy, divergence, surprise, energy and similarity to the FEP terms. To validate this, problems known to SE have been used to reconstruct and identify forces fluxes, altogether with strain and stresses identification.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

In this period, the scientific framing of the structural intuition in terms of mathematical formulation and validation has been carried out. A significant portion of the returning phase has been dedicated to investigate and establish a Neuroscience analogy with visual solution and identification of Forces.Therefore a step-by-step approach has been laid out to work on every term of the FEP and then grouped together for a complete solution. We consider this result the main accomplishment of the project, that both stands as a beyond state of the art accomplishment with a highly scientific content and the opening of a new research conversation. The dissemination of this result required enormous effort in preparation for 2 paper for well regarded scientific journal. Concerning the exploitation of the developed methodologies, it has been mainly capitalized on the outcomes of the outgoing period, exploiting the CV classification methodologies acquired for the urban analyses and that lead to the preparation of 3 separate publications in SE journals (main field of the beneficiary researcher and department). The influence of the multidisciplinary nature of MIT-SCL brought also to a number of 2 other publications ready for broader spectrum journals - concerning the differences between AI structural assessment vs. human perception and the impact of seismic risk on urban morphology. We successfully transposed the methodologies used in the RSV to other pathologies of interest in SE practice such as bridge damage or arch stability. To expand this application to a more general CV tool for structural analysis, a series of tasks has been performed by coupling experiments and numerical simulation of simple systems. Part of this work served as a starting point on the potential of machine learning strategies for instability detection and will be presented on a workshop held in Turin in 2023 (IWSS). At the same Workshop, the main accomplishments ad advancements of FORSEES will be presented by the beneficiary in 2 separate contributions. In the following two years the beneficiary already submitted contributions for 2 conferences (WCEE and IASS) to disseminate the scientific project results of FORSEES. An interactive website with the results will be updated regularly to keep engaging a wide and general public and the related CV tool will be used as prototype in a innovative teaching course and released under request with an open source framework.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

Forces geometric visualization in one of the key problem in Structural Mechanics and its advancement went along the progress of the subject itself. Cauchy stress tensor revolutionized the understanding of solid's internal state of equilibrium and the true concept of strength of materials, even if it's recognized the fictitious nature of it. We suggested a new perspective to the problem of forces identification (and related stress and strain) in solids based on some recent discoveries in the Neuroscience. The analogy we used is between the FEP and Elastic Potential, defining a structure behavior as the inner agent to sustain structural integrity. We think this novel approach could spur a new conversation around the core of a field of study that is one of the most traditional one in the STEM panorama. We expect all the consequences of a notable scientific advancement and the beneficiary expects to capitalize on the result of FORSEES for paving his academic career and consolidate the subject on the international environment. The main reason why the perspective introduced in FORSEES is promising for impactful applications, is the development in CV diagnostic tools for man applied sciences fields. It is in fact a major trend to use imagery data for large scenario analyses and global assessments. We successfully replicated the results of RSV methods for seismic vulnerability using street level imagery and Deep Learning classifications. In defining such framework, we used a novel hierarchical binary cascade classification that proved more reliable and richer in information than the state of the art. One of the most important information we derived, was the quantitative investigation of the human factor in risk and safety perception of structural arrangements. We used quantitative metrics and compared results with trained engineers to assess it. Humanity requires more and better diagnosticians of the building stock, exploiting on the interfaces between human perception and machine intelligence rather than operate with alienating automatic frameworks. FORSEES results are powerful tools to explore this interface and to lay fertile ground for inclusive applications. The number of methodologies produced are scalable and compatible with the challenges of the modern world in fast risk assessment, image analysis and AI interactive devices.

real time flux forcess in buildings by visual analysis