Replacing camera projection with AI

Mark filmed and directed the mood film for installation artist Fanni Hegyi and the work she has done on a newly built office in Budapest. Her patination style adds a unique look to an otherwise modern office space. The process of patination includes many different steps, often with many days apart, so picking a day for shooting was difficult on its own. The budget was low, so the only equipment that was used was a handheld Fujifilm XT-3, which meant a lot of the camera movements needed to be fixed in post in DaVinci. However, there were scenes like this example, where adding nice smooth floating camera movements were essential to tell the story Mark wanted to tell. He used a mix of AI tools to speed up the process of an otherwise slow and time-consuming task like projecting a 3D scene in a software like Nuke.

First, a reference frame was used in RunwayML that generated that panning movement using the below prompts:

A wide shot of an industrial garage space. The camera slowly panns towards the background wall.

An AI generated depth map was also used that gave more controll over depth of field settings and grading elements.

Using Nuke for camera tracking this footage, made it easy to put other elements into the scene. Mark needed brass textures to be comped in on the left side to showcase how these metal works looked before patination. Again, AI generated image was a quick and easy way to find these textures, that did not only have the same perspective, but also some of the lighting and reflections of the real scene.

Some particle elements were made using traditional tecnhiques within Nuke and for masking out the main character from another shot, a copycat model was trained to quickly do the roto tasks.