3d Video: Zipling

Use a Python script with Depth Pro (fast, high-res):

import torch
from depth_pro import DepthPro
model = DepthPro.from_pretrained("depthpro/checkpoint.pth")
model = model.cuda()
for i in range(num_frames):
img = load_image(f"frames/frame_i:04d.jpg")
depth = model.infer(img)  # returns depth map
save_depth(depth, f"depth/depth_i:04d.png")

Depth values range from near (white) to far (black).

Strengths:

Limitations:

Comparison to prior art: Zipline is faster than D-NeRF, higher quality than single-depth, and more practical than 32-camera domes.

Medical students currently learn anatomy from cadavers or 2D videos. Zipling captures a live surgery from 6 different angles simultaneously. The student can then replay the surgery in 3D, moving their viewpoint to see exactly how the surgeon’s hands moved relative to the retractor. It is the closest thing to standing in the OR without the liability.

Zipling (a portmanteau of zipper + mapping + linking) is an emerging DIY technique where: zipling 3d video

Unlike native 3D filmed with two cameras, zipling synthesizes 3D from a single stream.

Combine left and right frames horizontally:

ffmpeg -framerate 30 -i left_frames/l_%04d.png -i right_frames/r_%04d.png -filter_complex "[0]scale=960:1080[l];[1]scale=960:1080[r];[l][r]hstack" output_sbs.mp4