Leveraging 2D data to learn textured 3D mesh generation

P.M. Henderson, V. Tsiminaki, C. Lampert, in:, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVF, 2020, pp. 7498–7507.

Download
OA paper.pdf 10.26 MB
Conference Paper | Published | English
Department
Abstract
Numerous methods have been proposed for probabilistic generative modelling of 3D objects. However, none of these is able to produce textured objects, which renders them of limited use for practical tasks. In this work, we present the first generative model of textured 3D meshes. Training such a model would traditionally require a large dataset of textured meshes, but unfortunately, existing datasets of meshes lack detailed textures. We instead propose a new training methodology that allows learning from collections of 2D images without any 3D information. To do so, we train our model to explain a distribution of images by modelling each image as a 3D foreground object placed in front of a 2D background. Thus, it learns to generate meshes that when rendered, produce images similar to those in its training set. A well-known problem when generating meshes with deep networks is the emergence of self-intersections, which are problematic for many use-cases. As a second contribution we therefore introduce a new generation process for 3D meshes that guarantees no self-intersections arise, based on the physical intuition that faces should push one another out of the way as they move. We conduct extensive experiments on our approach, reporting quantitative and qualitative results on both synthetic data and natural images. These show our method successfully learns to generate plausible and diverse textured 3D samples for five challenging object classes.
Publishing Year
Date Published
2020-07-01
Proceedings Title
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Page
7498-7507
Conference
CVPR: Conference on Computer Vision and Pattern Recognition
Conference Location
Virtual
Conference Date
2020-06-14 – 2020-06-19
IST-REx-ID

Cite this

Henderson PM, Tsiminaki V, Lampert C. Leveraging 2D data to learn textured 3D mesh generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. CVF; 2020:7498-7507.
Henderson, P. M., Tsiminaki, V., & Lampert, C. (2020). Leveraging 2D data to learn textured 3D mesh generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 7498–7507). Virtual: CVF.
Henderson, Paul M, Vagia Tsiminaki, and Christoph Lampert. “Leveraging 2D Data to Learn Textured 3D Mesh Generation.” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7498–7507. CVF, 2020.
P. M. Henderson, V. Tsiminaki, and C. Lampert, “Leveraging 2D data to learn textured 3D mesh generation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Virtual, 2020, pp. 7498–7507.
Henderson PM, Tsiminaki V, Lampert C. 2020. Leveraging 2D data to learn textured 3D mesh generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. CVPR: Conference on Computer Vision and Pattern Recognition 7498–7507.
Henderson, Paul M., et al. “Leveraging 2D Data to Learn Textured 3D Mesh Generation.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVF, 2020, pp. 7498–507.
All files available under the following license(s):
Copyright Statement:
This Item is protected by copyright and/or related rights. [...]
Main File(s)
File Name
paper.pdf 10.26 MB
Access Level
OA Open Access
Date Uploaded
2020-07-31


Link(s) to Main File(s)
Access Level
OA Open Access

Export

Marked Publications

Open Data IST Research Explorer

Sources

arXiv 2004.04180

Search this title in

Google Scholar