Artificial Intelligence & Life Art

Auszeichnung - Award of Distinction

VFRAME: Visual Forensics and Metadata Extraction

Adam Harvey (US)




URL:
https://vframe.io/research/synthetic-datasets/

VFRAME is a research project exploring how computer vision can be applied to human rights research. VFRAME started in 2017 as a prototype demonstrating how object detection can be used to locate illegal munitions in the Syrian conflict. During 2018, VFRAME worked with the Syrian Archive (syrianarchive.org) to develop a large-scale visual search engine and custom algorithms for processing million-scale video datasets. Then, a new problem emerged: it was only possible to create object detection models for objects with thousands of examples, yet most objects in conflict zones only appear in a limited number of videos, often under extreme circumstances and with low-resolution cameras. To train new object detection models relevant for human rights researchers working with footage from conflict zones a new approach was needed.

During 2019 VFRAME is developing synthetic data modeling techniques that can be used to create specialized datasets. Synthetic training data is 3D-modeled photorealistic imagery based on a set of real, verified images. Often only a few real images are needed to recreate the object as a 3D model. The 3D model can then be textured and placed into a 3D environment to match the real images. This 3D environment can then be customized to generate lighting, weather, camera, and scene variations that are exported to still frames and used as image training data.

The first prototype shows how this method can be applied to an illegal cluster munition, the AO-2.5RT. Although 3D modeling the objects and scenes is still a labor-intensive process, it has the potential to be scalable and provide training data for objects that would otherwise not exist in the quantity and quality needed to train object detection models. This new approach has the potential to vastly increase the potential applications of computer vision for human rights researchers and will be carried out through a collaboration with the Syrian and Yemeni Archive projects based in Berlin.

Biography:

Adam Harvey (US)

Adam Harvey (US) is an American artist and researcher based in Berlin, who focuses on computer vision, privacy, and surveillance technologies. He is a graduate of New York University's Interactive Telecommunications Program and previously studied engineering and photojournalism at the Pennsylvania State University. Harvey's previous work includes CV Dazzle, the Anti-Drone Burqa, SkyLift, and MegaPixels. He is currently a researcher in residence at Karlsruhe HfG with a focus on machine learning image datasets.

Credits:
Founder: Adam Harvey
Co-developer: Jules LaPlace
3D designer: Josh Labouve
Researchers: Hadi Al-Khatib and Jeff Deutch at Syrian Archive
Support received from: PrototypeFund.de