Final Report Summary - VARCITY (Variation & the City)
The VarCity project succeeded in lifting 3D city models to a novel level of realism and liveliness. On the
one hand, the 3D models of the static parts could be improved via an understanding of what they
actually are. So rather than having a (triangulated) soup of 3D points, the goal was to treat shapes as
representing buildings, facades, vegetation, road, etc. Their visualization can then be diversified
accordingly, with e.g. tree canopies being fine-grained and volumetric, or windows reflective. In
addition, the project wanted to extract the typical traffic flows found in the different streets, from video samples available for a limited number of locations and times. The model can then be populated with similar virtual traffic, showing it as lively as the city really is, rather than as a deserted ghost town.
The VarCity project results are useful for a range of applications, like city design, architectural planning, the preparation and support of responses to catastrophic events, gaming, climate related studies, automated driving, etc. For many of these, it is indeed crucial that the model can be interacted with realistically, i.e. that the model is aware of the meaning and functionality of its parts.
As the goal was to build superior such models with affordable effort for entire cities, it is crucial that
the project’s approaches are automated, powerful, and efficient. Much of the work has been based on so-called inverse procedural modeling, i.e. the description of buildings as resulting from a series of constructive steps, with direct reference to the meaning of the parts. Hence, rather than focusing on 3D points or triangles, the approach handles meaningful parts, like windows or balconies. Moreover, the procedural descriptions were generated from images of the buildings as the sole input. The project has been working on massive implementations of these principles, such that the modeling of a complete city runs fully automatically in a matter of days. The input to the system are images taken along the streets (think of Google street view). The advantage of the resulting procedural models is that they are compact while allowing for more realistic renderings of the model as well as for the aforementioned possibility to interact with the model.
The dynamic part of the project aimed at understanding the traffic flows in the city. The challenge was to
extract those flows, using the knowledge about the city's 3D structure, from piecemeal observations,
like an occasional video taken by a pedestrian or a location-fixed surveillance camera. Such analysis
should allow for the instantaneous inference of city-wide traffic patterns, thereby creating more
helpful navigation systems or guides to avoid regions with heavy exhaust pollution. In order to get at
the necessary input data, we needed to detect cars, people, bikes, etc. in the videos and be able to
determine their motion patterns.
As an additional service, it was made possible to see historic or consumer video and photo footage
linked to the geographic sites where it was originally taken. This will allow visitors to see the places
at different times of the day, during different times of the year, or from another era. In order to
accommodate the viewers, one of the goals was to let the computer automatically build summaries of
such footage within the time budget they foresee.
Overall, VarCity has brought about a unique blend of cutting edge computer vision research, covering object recognition, tracking, and 3D extraction, as well as their integration.
one hand, the 3D models of the static parts could be improved via an understanding of what they
actually are. So rather than having a (triangulated) soup of 3D points, the goal was to treat shapes as
representing buildings, facades, vegetation, road, etc. Their visualization can then be diversified
accordingly, with e.g. tree canopies being fine-grained and volumetric, or windows reflective. In
addition, the project wanted to extract the typical traffic flows found in the different streets, from video samples available for a limited number of locations and times. The model can then be populated with similar virtual traffic, showing it as lively as the city really is, rather than as a deserted ghost town.
The VarCity project results are useful for a range of applications, like city design, architectural planning, the preparation and support of responses to catastrophic events, gaming, climate related studies, automated driving, etc. For many of these, it is indeed crucial that the model can be interacted with realistically, i.e. that the model is aware of the meaning and functionality of its parts.
As the goal was to build superior such models with affordable effort for entire cities, it is crucial that
the project’s approaches are automated, powerful, and efficient. Much of the work has been based on so-called inverse procedural modeling, i.e. the description of buildings as resulting from a series of constructive steps, with direct reference to the meaning of the parts. Hence, rather than focusing on 3D points or triangles, the approach handles meaningful parts, like windows or balconies. Moreover, the procedural descriptions were generated from images of the buildings as the sole input. The project has been working on massive implementations of these principles, such that the modeling of a complete city runs fully automatically in a matter of days. The input to the system are images taken along the streets (think of Google street view). The advantage of the resulting procedural models is that they are compact while allowing for more realistic renderings of the model as well as for the aforementioned possibility to interact with the model.
The dynamic part of the project aimed at understanding the traffic flows in the city. The challenge was to
extract those flows, using the knowledge about the city's 3D structure, from piecemeal observations,
like an occasional video taken by a pedestrian or a location-fixed surveillance camera. Such analysis
should allow for the instantaneous inference of city-wide traffic patterns, thereby creating more
helpful navigation systems or guides to avoid regions with heavy exhaust pollution. In order to get at
the necessary input data, we needed to detect cars, people, bikes, etc. in the videos and be able to
determine their motion patterns.
As an additional service, it was made possible to see historic or consumer video and photo footage
linked to the geographic sites where it was originally taken. This will allow visitors to see the places
at different times of the day, during different times of the year, or from another era. In order to
accommodate the viewers, one of the goals was to let the computer automatically build summaries of
such footage within the time budget they foresee.
Overall, VarCity has brought about a unique blend of cutting edge computer vision research, covering object recognition, tracking, and 3D extraction, as well as their integration.