Google AI Presenting MELON, A Technique That Can Determine Object-Centric Camera

  • Editor
  • March 19, 2024

In the world of digital imaging and computer vision, a significant breakthrough has been announced by Google AI, marking a monumental step forward in the field.

Dubbed MELON, this innovative technique addresses a longstanding challenge: reconstructing the 3D shape of an object from a limited set of 2D images. This development could revolutionize various industries, including e-commerce 3D model creation and autonomous vehicle navigation, by enabling precise object-centric camera pose determination.

Understanding objects in three dimensions from sparse visual data has puzzled researchers for years. A crux of this challenge is the inference of camera poses—the specific viewpoints from which images are captured. Known camera poses can leverage existing methods like neural radiance fields (NeRF) or 3D Gaussian Splatting for 3D reconstruction.

However, unknown poses introduce a “chicken and egg” problem: determining poses requires a 3D object model, yet the model’s reconstruction is contingent upon known poses. This dilemma is further complicated by objects’ pseudo-symmetries, where certain shapes appear identical from various angles, complicating the differentiation of viewpoints.

In an official blog post, Google AI said, “We leverage two key techniques to aid the convergence of this ill-posed problem. The first is a very lightweight, dynamically trained convolutional neural network (CNN) encoder that regresses camera poses from training images.”

“We pass a downscaled training image to a four-layer CNN that infers the camera pose. This CNN is initialized from noise and requires no pre-training. Its capacity is so small that it forces similar-looking images to similar poses, providing an implicit regularization and greatly aiding convergence.”


MELON innovates by employing a dynamically trained convolutional neural network (CNN) to infer camera poses directly from images, coupled with a modulo loss technique that accounts for objects’ pseudo-symmetries.

This approach allows for the accurate prediction of camera viewpoints, even without pre-determined poses, by rendering the object from a set of fixed perspectives and selecting the view that best aligns with each image. This method achieves rapid convergence to optimal camera poses, a critical step for effective 3D reconstruction.

These two techniques are integrated into standard NeRF training, except that instead of fixed camera poses, poses are inferred by the CNN and duplicated by the modulo loss.

Photometric gradients back-propagate through the best-fitting cameras into the CNN. We observe that cameras generally converge quickly to globally optimal poses (see animation below). After training of the neural field, MELON can synthesize novel views using standard NeRF rendering methods.


The technique was tested using the NeRF-Synthetic dataset, a benchmark for NeRF research, simplifying the problem only to require inference of the camera’s polar coordinates. This test further demonstrates MELON’s potential to accurately reconstruct 3D objects from limited visual information, providing a robust solution to a complex challenge that has implications for various practical applications.

A further announcement was made through Google AI’s official X account:

MELON’s development signifies a significant leap in the quest to bridge the gap between 2D imaging and 3D object understanding, offering a powerful tool for a range of applications.

From enhancing online shopping experiences with 3D product views to improving navigation systems for autonomous vehicles, MELON’s impact on technology and daily life promises to be far-reaching.

People seemed to be excited about this:

As this technique continues to evolve, it stands as a testament to the innovative spirit driving advancements in AI and computer vision, promising new horizons in digital imaging and beyond.

For more such news, visit our AI news at

Was this article helpful?
Generic placeholder image

Dave Andre


Digital marketing enthusiast by day, nature wanderer by dusk. Dave Andre blends two decades of AI and SaaS expertise into impactful strategies for SMEs. His weekends? Lost in books on tech trends and rejuvenating on scenic trails.

Related Articles

Leave a Reply

Your email address will not be published. Required fields are marked *