--- _id: '14488' abstract: - lang: eng text: 'Portrait viewpoint and illumination editing is an important problem with several applications in VR/AR, movies, and photography. Comprehensive knowledge of geometry and illumination is critical for obtaining photorealistic results. Current methods are unable to explicitly model in 3D while handling both viewpoint and illumination editing from a single image. In this paper, we propose VoRF, a novel approach that can take even a single portrait image as input and relight human heads under novel illuminations that can be viewed from arbitrary viewpoints. VoRF represents a human head as a continuous volumetric field and learns a prior model of human heads using a coordinate-based MLP with individual latent spaces for identity and illumination. The prior model is learned in an auto-decoder manner over a diverse class of head shapes and appearances, allowing VoRF to generalize to novel test identities from a single input image. Additionally, VoRF has a reflectance MLP that uses the intermediate features of the prior model for rendering One-Light-at-A-Time (OLAT) images under novel views. We synthesize novel illuminations by combining these OLAT images with target environment maps. Qualitative and quantitative evaluations demonstrate the effectiveness of VoRF for relighting and novel view synthesis, even when applied to unseen subjects under uncontrolled illumination. This work is an extension of Rao et al. (VoRF: Volumetric Relightable Faces 2022). We provide extensive evaluation and ablative studies of our model and also provide an application, where any face can be relighted using textual input.' acknowledgement: Open Access funding enabled and organized by Projekt DEAL. article_processing_charge: Yes (via OA deal) article_type: original author: - first_name: Pramod full_name: Rao, Pramod last_name: Rao - first_name: B. R. full_name: Mallikarjun, B. R. last_name: Mallikarjun - first_name: Gereon full_name: Fox, Gereon last_name: Fox - first_name: Tim full_name: Weyrich, Tim last_name: Weyrich - first_name: Bernd full_name: Bickel, Bernd id: 49876194-F248-11E8-B48F-1D18A9856A87 last_name: Bickel orcid: 0000-0001-6511-9385 - first_name: Hanspeter full_name: Pfister, Hanspeter last_name: Pfister - first_name: Wojciech full_name: Matusik, Wojciech last_name: Matusik - first_name: Fangneng full_name: Zhan, Fangneng last_name: Zhan - first_name: Ayush full_name: Tewari, Ayush last_name: Tewari - first_name: Christian full_name: Theobalt, Christian last_name: Theobalt - first_name: Mohamed full_name: Elgharib, Mohamed last_name: Elgharib citation: ama: Rao P, Mallikarjun BR, Fox G, et al. A deeper analysis of volumetric relightiable faces. International Journal of Computer Vision. 2023. doi:10.1007/s11263-023-01899-3 apa: Rao, P., Mallikarjun, B. R., Fox, G., Weyrich, T., Bickel, B., Pfister, H., … Elgharib, M. (2023). A deeper analysis of volumetric relightiable faces. International Journal of Computer Vision. Springer Nature. https://doi.org/10.1007/s11263-023-01899-3 chicago: Rao, Pramod, B. R. Mallikarjun, Gereon Fox, Tim Weyrich, Bernd Bickel, Hanspeter Pfister, Wojciech Matusik, et al. “A Deeper Analysis of Volumetric Relightiable Faces.” International Journal of Computer Vision. Springer Nature, 2023. https://doi.org/10.1007/s11263-023-01899-3. ieee: P. Rao et al., “A deeper analysis of volumetric relightiable faces,” International Journal of Computer Vision. Springer Nature, 2023. ista: Rao P, Mallikarjun BR, Fox G, Weyrich T, Bickel B, Pfister H, Matusik W, Zhan F, Tewari A, Theobalt C, Elgharib M. 2023. A deeper analysis of volumetric relightiable faces. International Journal of Computer Vision. mla: Rao, Pramod, et al. “A Deeper Analysis of Volumetric Relightiable Faces.” International Journal of Computer Vision, Springer Nature, 2023, doi:10.1007/s11263-023-01899-3. short: P. Rao, B.R. Mallikarjun, G. Fox, T. Weyrich, B. Bickel, H. Pfister, W. Matusik, F. Zhan, A. Tewari, C. Theobalt, M. Elgharib, International Journal of Computer Vision (2023). date_created: 2023-11-05T23:00:54Z date_published: 2023-10-31T00:00:00Z date_updated: 2023-11-06T08:52:30Z day: '31' department: - _id: BeBi doi: 10.1007/s11263-023-01899-3 language: - iso: eng main_file_link: - open_access: '1' url: https://doi.org/10.1007/s11263-023-01899-3 month: '10' oa: 1 oa_version: Published Version publication: International Journal of Computer Vision publication_identifier: eissn: - 1573-1405 issn: - 0920-5691 publication_status: epub_ahead publisher: Springer Nature quality_controlled: '1' scopus_import: '1' status: public title: A deeper analysis of volumetric relightiable faces type: journal_article user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87 year: '2023' ... --- _id: '6952' abstract: - lang: eng text: 'We present a unified framework tackling two problems: class-specific 3D reconstruction from a single image, and generation of new 3D shape samples. These tasks have received considerable attention recently; however, most existing approaches rely on 3D supervision, annotation of 2D images with keypoints or poses, and/or training with multiple views of each object instance. Our framework is very general: it can be trained in similar settings to existing approaches, while also supporting weaker supervision. Importantly, it can be trained purely from 2D images, without pose annotations, and with only a single view per instance. We employ meshes as an output representation, instead of voxels used in most prior work. This allows us to reason over lighting parameters and exploit shading information during training, which previous 2D-supervised methods cannot. Thus, our method can learn to generate and reconstruct concave object classes. We evaluate our approach in various settings, showing that: (i) it learns to disentangle shape from pose and lighting; (ii) using shading in the loss improves performance compared to just silhouettes; (iii) when using a standard single white light, our model outperforms state-of-the-art 2D-supervised methods, both with and without pose supervision, thanks to exploiting shading cues; (iv) performance improves further when using multiple coloured lights, even approaching that of state-of-the-art 3D-supervised methods; (v) shapes produced by our model capture smooth surfaces and fine details better than voxel-based approaches; and (vi) our approach supports concave classes such as bathtubs and sofas, which methods based on silhouettes cannot learn.' acknowledgement: Open access funding provided by Institute of Science and Technology (IST Austria). article_processing_charge: Yes (via OA deal) article_type: original author: - first_name: Paul M full_name: Henderson, Paul M id: 13C09E74-18D9-11E9-8878-32CFE5697425 last_name: Henderson orcid: 0000-0002-5198-7445 - first_name: Vittorio full_name: Ferrari, Vittorio last_name: Ferrari citation: ama: Henderson PM, Ferrari V. Learning single-image 3D reconstruction by generative modelling of shape, pose and shading. International Journal of Computer Vision. 2020;128:835-854. doi:10.1007/s11263-019-01219-8 apa: Henderson, P. M., & Ferrari, V. (2020). Learning single-image 3D reconstruction by generative modelling of shape, pose and shading. International Journal of Computer Vision. Springer Nature. https://doi.org/10.1007/s11263-019-01219-8 chicago: Henderson, Paul M, and Vittorio Ferrari. “Learning Single-Image 3D Reconstruction by Generative Modelling of Shape, Pose and Shading.” International Journal of Computer Vision. Springer Nature, 2020. https://doi.org/10.1007/s11263-019-01219-8. ieee: P. M. Henderson and V. Ferrari, “Learning single-image 3D reconstruction by generative modelling of shape, pose and shading,” International Journal of Computer Vision, vol. 128. Springer Nature, pp. 835–854, 2020. ista: Henderson PM, Ferrari V. 2020. Learning single-image 3D reconstruction by generative modelling of shape, pose and shading. International Journal of Computer Vision. 128, 835–854. mla: Henderson, Paul M., and Vittorio Ferrari. “Learning Single-Image 3D Reconstruction by Generative Modelling of Shape, Pose and Shading.” International Journal of Computer Vision, vol. 128, Springer Nature, 2020, pp. 835–54, doi:10.1007/s11263-019-01219-8. short: P.M. Henderson, V. Ferrari, International Journal of Computer Vision 128 (2020) 835–854. date_created: 2019-10-17T13:38:20Z date_published: 2020-04-01T00:00:00Z date_updated: 2023-08-17T14:01:16Z day: '01' ddc: - '004' department: - _id: ChLa doi: 10.1007/s11263-019-01219-8 external_id: arxiv: - '1901.06447' isi: - '000491042100002' file: - access_level: open_access checksum: a0f05dd4f5f64e4f713d8d9d4b5b1e3f content_type: application/pdf creator: dernst date_created: 2019-10-25T10:28:29Z date_updated: 2020-07-14T12:47:46Z file_id: '6973' file_name: 2019_CompVision_Henderson.pdf file_size: 2243134 relation: main_file file_date_updated: 2020-07-14T12:47:46Z has_accepted_license: '1' intvolume: ' 128' isi: 1 language: - iso: eng month: '04' oa: 1 oa_version: Published Version page: 835-854 project: - _id: B67AFEDC-15C9-11EA-A837-991A96BB2854 name: IST Austria Open Access Fund publication: International Journal of Computer Vision publication_identifier: eissn: - 1573-1405 issn: - 0920-5691 publication_status: published publisher: Springer Nature quality_controlled: '1' scopus_import: '1' status: public title: Learning single-image 3D reconstruction by generative modelling of shape, pose and shading tmp: image: /images/cc_by.png legal_code_url: https://creativecommons.org/licenses/by/4.0/legalcode name: Creative Commons Attribution 4.0 International Public License (CC-BY 4.0) short: CC BY (4.0) type: journal_article user_id: 4359f0d1-fa6c-11eb-b949-802e58b17ae8 volume: 128 year: '2020' ... --- _id: '6944' abstract: - lang: eng text: 'We study the problem of automatically detecting if a given multi-class classifier operates outside of its specifications (out-of-specs), i.e. on input data from a different distribution than what it was trained for. This is an important problem to solve on the road towards creating reliable computer vision systems for real-world applications, because the quality of a classifier’s predictions cannot be guaranteed if it operates out-of-specs. Previously proposed methods for out-of-specs detection make decisions on the level of single inputs. This, however, is insufficient to achieve low false positive rate and high false negative rates at the same time. In this work, we describe a new procedure named KS(conf), based on statistical reasoning. Its main component is a classical Kolmogorov–Smirnov test that is applied to the set of predicted confidence values for batches of samples. Working with batches instead of single samples allows increasing the true positive rate without negatively affecting the false positive rate, thereby overcoming a crucial limitation of single sample tests. We show by extensive experiments using a variety of convolutional network architectures and datasets that KS(conf) reliably detects out-of-specs situations even under conditions where other tests fail. It furthermore has a number of properties that make it an excellent candidate for practical deployment: it is easy to implement, adds almost no overhead to the system, works with any classifier that outputs confidence scores, and requires no a priori knowledge about how the data distribution could change.' article_processing_charge: Yes (via OA deal) article_type: original author: - first_name: Rémy full_name: Sun, Rémy last_name: Sun - first_name: Christoph full_name: Lampert, Christoph id: 40C20FD2-F248-11E8-B48F-1D18A9856A87 last_name: Lampert orcid: 0000-0001-8622-7887 citation: ama: 'Sun R, Lampert C. KS(conf): A light-weight test if a multiclass classifier operates outside of its specifications. International Journal of Computer Vision. 2020;128(4):970-995. doi:10.1007/s11263-019-01232-x' apa: 'Sun, R., & Lampert, C. (2020). KS(conf): A light-weight test if a multiclass classifier operates outside of its specifications. International Journal of Computer Vision. Springer Nature. https://doi.org/10.1007/s11263-019-01232-x' chicago: 'Sun, Rémy, and Christoph Lampert. “KS(Conf): A Light-Weight Test If a Multiclass Classifier Operates Outside of Its Specifications.” International Journal of Computer Vision. Springer Nature, 2020. https://doi.org/10.1007/s11263-019-01232-x.' ieee: 'R. Sun and C. Lampert, “KS(conf): A light-weight test if a multiclass classifier operates outside of its specifications,” International Journal of Computer Vision, vol. 128, no. 4. Springer Nature, pp. 970–995, 2020.' ista: 'Sun R, Lampert C. 2020. KS(conf): A light-weight test if a multiclass classifier operates outside of its specifications. International Journal of Computer Vision. 128(4), 970–995.' mla: 'Sun, Rémy, and Christoph Lampert. “KS(Conf): A Light-Weight Test If a Multiclass Classifier Operates Outside of Its Specifications.” International Journal of Computer Vision, vol. 128, no. 4, Springer Nature, 2020, pp. 970–95, doi:10.1007/s11263-019-01232-x.' short: R. Sun, C. Lampert, International Journal of Computer Vision 128 (2020) 970–995. date_created: 2019-10-14T09:14:28Z date_published: 2020-04-01T00:00:00Z date_updated: 2024-02-22T14:57:30Z day: '01' ddc: - '004' department: - _id: ChLa doi: 10.1007/s11263-019-01232-x ec_funded: 1 external_id: isi: - '000494406800001' file: - access_level: open_access checksum: 155e63edf664dcacb3bdc1c2223e606f content_type: application/pdf creator: dernst date_created: 2019-11-26T10:30:02Z date_updated: 2020-07-14T12:47:45Z file_id: '7110' file_name: 2019_IJCV_Sun.pdf file_size: 1715072 relation: main_file file_date_updated: 2020-07-14T12:47:45Z has_accepted_license: '1' intvolume: ' 128' isi: 1 issue: '4' language: - iso: eng month: '04' oa: 1 oa_version: Published Version page: 970-995 project: - _id: 2532554C-B435-11E9-9278-68D0E5697425 call_identifier: FP7 grant_number: '308036' name: Lifelong Learning of Visual Scene Understanding - _id: B67AFEDC-15C9-11EA-A837-991A96BB2854 name: IST Austria Open Access Fund publication: International Journal of Computer Vision publication_identifier: eissn: - 1573-1405 issn: - 0920-5691 publication_status: published publisher: Springer Nature quality_controlled: '1' related_material: link: - relation: erratum url: https://doi.org/10.1007/s11263-019-01262-5 record: - id: '6482' relation: earlier_version status: public scopus_import: '1' status: public title: 'KS(conf): A light-weight test if a multiclass classifier operates outside of its specifications' tmp: image: /images/cc_by.png legal_code_url: https://creativecommons.org/licenses/by/4.0/legalcode name: Creative Commons Attribution 4.0 International Public License (CC-BY 4.0) short: CC BY (4.0) type: journal_article user_id: 3E5EF7F0-F248-11E8-B48F-1D18A9856A87 volume: 128 year: '2020' ...