Microsoft at CVPR Twenty Twenty-Four: Pushing the Boundaries of Computer Vision

Redmond, Washington – June fifteenth, Twenty Twenty-Four – Microsoft is psyched to be a sponsor of the forty-first annual Conference on Computer Vision and Pattern Recognition (CVPR Twenty Twenty-Four), happening from June seventeenth to June twenty-first. As a major event in the field, CVPR brings together the sharpest minds to explore groundbreaking advancements in computer vision and pattern recognition. This year’s conference covers a diverse range of topics, including three-D reconstruction and modeling, action and motion analysis, video and image processing, synthetic data generation, and neural networks, among many others.

This year, Microsoft researchers are presenting sixty-three accepted papers, with six chosen for the big leagues – oral presentations. These contributions highlight the impactful work being done at Microsoft across various facets of computer vision.

Interdisciplinary Research for Real-World Impact

The breadth of research projects presented by Microsoft at CVPR Twenty Twenty-Four showcases the company’s commitment to interdisciplinary collaboration. From developing techniques for precisely reconstructing three-D human figures in augmented reality to leveraging synthetic data for enhanced real-world scenario replication, Microsoft researchers are pushing the boundaries of what’s possible.

One notable area of focus is the integration of machine learning with natural language processing and structured data. This fusion enables the development of models that not only perceive the visual world but can also interact with it in meaningful ways. Ultimately, these projects aim to elevate machine perception capabilities, leading to more accurate and responsive interactions between humans and machines.

Highlights from Microsoft’s Oral Presentations

This year, six Microsoft research papers have been selected for oral presentations, a testament to their quality and potential impact:

TreeOfLife-10M & BioCLIP: Revolutionizing Biological Image Analysis

This research introduces TreeOfLife-10M, the largest and most diverse dataset of biology images designed for machine learning, and BioCLIP, a foundation model specifically trained for biological sciences. Utilizing the vast array of organism images and structured knowledge within TreeOfLife-10M, BioCLIP excels in fine-grained biological classification, outperforming existing models and demonstrating superior generalization capabilities. This work paves the way for transformative advancements in fields like biodiversity monitoring and ecological research.

EgoGen: Generating Realistic Egocentric Views in Augmented Reality

Creating realistic anatomical movements for authentic egocentric views in augmented reality is a major challenge. EgoGen, a sophisticated synthetic data generator, addresses this by improving the accuracy of training data for egocentric tasks and refining the integration of motion and perception. This practical solution for generating realistic egocentric training data will significantly benefit egocentric computer vision research.

Florence-Two: A Unified Vision Foundation Model for Diverse Tasks

Florence-Two is a unified, prompt-based vision foundation model capable of handling a wide range of tasks, from image captioning to object detection and segmentation. Trained on the massive FLD-5B dataset, comprising billions of annotations on millions of images, Florence-Two interprets text prompts as instructions and generates accurate outputs across various vision and vision-language tasks.

Reasoning Segmentation: Towards More Sophisticated Image Understanding

This research introduces “reasoning segmentation,” a new task that utilizes complex query texts to generate segmentation masks. The researchers have also created a benchmark dataset with intricate reasoning and world knowledge challenges. Their proposed tool, Large Language Instructed Segmentation Assistant (LISA), combines the strengths of large language models with segmentation capabilities, enabling it to handle complex queries and exhibit impressive zero-shot learning abilities.

MultiPly: Reconstructing Multiple People in 3D from Single Videos

MultiPly introduces a novel framework for reconstructing multiple people in 3D from single-camera videos captured in natural settings. This technique employs a layered neural representation refined through differentiable volume rendering. By combining self-supervised 3D and promptable 2D techniques, MultiPly achieves reliable instance segmentation even in scenarios with close human interaction.

SceneFun3D: Advancing Functionality Understanding in 3D Scenes

Traditional 3D scene understanding methods primarily focus on semantic and instance segmentation. SceneFun3D pushes boundaries by enabling interaction with functional elements like handles and buttons. This robust dataset contains over fourteen thousand, eight hundred precise interaction annotations across seven hundred ten high-resolution real-world 3D indoor scenes. SceneFun3D is poised to accelerate research in functionality segmentation, task-driven affordance grounding, and 3D motion estimation.

Microsoft’s Vision for the Future of Computer Vision

Microsoft’s presence at CVPR Twenty Twenty-Four underscores its dedication to advancing the field of computer vision. The company’s research contributions, particularly in areas like three-D reconstruction, synthetic data generation, and the integration of language and vision, have the potential to revolutionize industries ranging from healthcare and retail to gaming and accessibility.

By openly sharing its research, datasets, and tools, Microsoft aims to foster collaboration and accelerate innovation within the computer vision community. The company is committed to developing responsible AI solutions that benefit society and address real-world challenges.

Beyond CVPR: Microsoft’s Commitment to Responsible AI

Beyond CVPR, Microsoft remains dedicated to advancing responsible AI research and practice. From identifying gender bias in languages like Hindi to uncovering AI-related risks for workers, Microsoft researchers are actively contributing to the development and deployment of ethical and inclusive AI systems.

Learn More

To learn more about Microsoft’s participation at CVPR Twenty Twenty-Four, including the full list of publications and session details, please visit the dedicated conference webpage (link to webpage if available).