Figure 1. Video from a fire exercise, filmed by two flight-synchronized drones from two different viewing angles, and with the video streams time-synchronized. This scenario has been used in our development project for 4D modelling.
Digital 3D models from images have been established as a standard product with many applications. However, our environment is filled with events and movements from vehicles, people, animals, water, fire and smoke. At I-CONIC, we are developing software that generates 4D models, i.e. time-dependent 3D models. The models are generated in real-time to be used when an event, such as a forest fire, takes place. The technology makes it possible to consider a process both stereoscopically and with the aid of a constantly changing point cloud. GPU programming is a crucial component for the extremely fast calculations.
Photogrammetry assumes that each part of the area to be mapped is visible in at least two images, taken from different viewpoints. The target must not have moved or changed the image between the exposures. However, there are many applications where moving objects and events that occur are of primary importance (Fig. 1). In addition to mapping, we can also include sports, film, scientific experiments, etc.
We want to achieve a user experience similar to that often found in computer games; while we look at and can move around in a digital 3D world, events occur in this world; vehicles and people move, explosions occur, etc. (Fig. 2). In this way, an intervention leader gets a “first-person viewer” perspective of the event without having to move. In addition to the increased understanding provided by this tool, measurements can be made to determine heights, volumes etc., also of moving objects.
Figure 2. A snapshot from a computer game, Emergency20, where you interact in an ever-changing 3D world.
We also want to be able to freeze a moment and then get a more detailed 3D model of the event at this particular moment. For example, Intel or 4DReplay have attractive solutions for this, which however, requires upwards of 160 cameras and is therefore expensive, complex and inflexible (Fig. 3). We will not, with our technology reach the same degree of realism, but our solution is far more practical, not least in time-critical efforts.
Figure 3. 4DReplay’s solution with a long row of system cameras, all filming from slightly different angles.
To be able to generate 4D models, we use at least two drones, which each acquire and link a video stream to the ground. The drones are synchronized so that they have a constant base between them, much like they were connected with a long stick. The base distance is continuously calculated during a mission and thus does not need to be well predetermined. The area on the ground must of course be seen simultaneously by both drones. The drones can either fly a predefined route, be flown manually, or hover, standing still over an event.
Figure 4. Overview of our concept with two or more drones filming an event and producing a continuous stream of time-dependent 3D models, presented to the user in real-time.
The video streams are synchronized, so that for each frame in a video sequence we know the frame that was taken exactly at the same time in the other video film, at least with an accuracy corresponding to the video’s recording rate, e.g. 50 frames/second, corresponding to 0.02 sec accuracy. In this way we get a stream with simultaneously registered image pairs. These image pairs are matched against each other so that their relative orientation can be determined.
The next step is another type of matching takes place to generate dense 3D point clouds, and from the dense point clouds a so-called textured mesh is created. This a now 3D model simulating the real world, it is updated at video-rate and moving object are also modelled.
We believe we are the first in the world to generate 3D models of moving objects from two or more moving drones, and we have applied for patents.