Postgraduate research project · PPKE-ITK

Geometry-Guided Neural Compositing

A local, GPU-accelerated vision pipeline that reconstructs the 3D geometry of an ordinary single-camera video and inserts synthetic elements so they sit convincingly in the scene.

PyTorch · CUDA Local-first Reproducible License-clean

Nemes Péter · Supervisor: Dr. Zsedrovits Tamás · Applied Artificial Intelligence postgraduate specialisation, Faculty of Information Technology and Bionics, Pázmány Péter Catholic University (PPKE-ITK) · defended 2026

Abstract

Making synthetic elements belong in real footage

Convincingly adding a synthetic element to existing video is hard.

Thesis

Research context

Programme: Applied Artificial Intelligence — postgraduate specialisation, PPKE-ITK
Author: Nemes Péter
Supervisor: Dr. Zsedrovits Tamás
Defended: 23 June 2026

Research question

Can a geometry-first, license-clean model stack composite synthetic elements into single-camera video convincingly on local consumer hardware?

Contributions

A geometry-first pipeline that inserts synthetic elements into monocular video, driven by reconstructed 3D structure.
A license-clean, reproducible multi-model stack that runs end to end on a single consumer GPU.
A disciplined, AI-assisted, human-in-the-loop development methodology — a reproducible environment, per-domain rule files, and a developer-plus-critic review loop.

Keywords monocular depth estimation neural compositing geometry-aware placement reproducible ML pipelines license-clean model selection human-in-the-loop AI development

10 neural models in the active pipeline

24 models evaluated & license-screened

0 non-commercial weights shipped

32-bit scene-referred linear working space

Capability map

Five vision problems, one coordinated system

Each domain is a substantial computer-vision problem in its own right.

01 1 model

Depth estimation

Temporally consistent per-pixel depth from a single moving camera.

02 2 models

Camera & scene solve

Recovering camera motion, intrinsics and sparse 3D structure.

03 1 model

Segmentation & occlusion

Video-consistent masks that decide what occludes what.

04 4 models

Generative synthesis

Diffusion-based image and motion generation, scene-conditioned.

05 2 models

Physically-based compositing

Shadows, matte, colour, grain and blur combined in linear light.

06 rule-based

Orchestration

A queue-backed worker system running reproducible, resumable jobs.

Per-stage output

Every stage of a single shot, side by side

The pipeline's per-stage outputs for one shot, from depth to the final frame.

pipeline stages · 2026

Limitations. A prototype, not a production system, produced under a hardware constraint; testing, scaling, a security audit and a quantitative accuracy evaluation remain future work.

Engineering principles

The decisions that make it trustworthy

Beyond the models, the project is an exercise in disciplined systems engineering.

License-clean model stack

Only permissively-licensed weights (Apache-2.0 / MIT / BSD) ever enter the pipeline.

Reproducible by construction

Deterministic manifests record pinned model versions and fixed seeds.

VRAM-aware orchestration

Large models are hot-swapped in and out of VRAM to fit one consumer GPU.

Scene-referred linear compositing

All compositing happens in 32-bit float linear light; gamma only at the edges.

Non-destructive by default

The original footage and audio are preserved; the result is additive.

Local-first & private

Runs entirely on local hardware — no frames sent to any third-party API.

Technology

Stack

A GPU compute core in Python, wrapped in a typed job-orchestration layer.

PyTorch 2.x · CUDA 12.1 Python 3.11 OpenCV Open3D OpenEXR · 16-bit depth Diffusion models · SDXL-class TypeScript · worker orchestration Queue-backed jobs · resumable

Deliberately not shown

Scope of this page

This page presents the capabilities and engineering approach of the project.