0.735
mAP50
Best configuration, measured end to end on the panorama. Real data alone reaches 0.706.
visionmlvisionsegmentationsynthetic-data
HILTI: Construction-Specific Instance Segmentation
Instance segmentation and tracking of architectural components in 360° construction video, with a procedural Blender pipeline that manufactures the training data the domain lacks.
ETH Zürich, 3D Vision course (HILTI collaboration) · Feb 2026 – Jun 2026
Overview
Construction sites are filmed with 360° cameras for progress tracking and quality control. The footage arrives as equirectangular video, and the job is to segment and track nine architectural classes per frame, three of which form confusable pairs: column against wall column, inner against outer window, door against dismounted door. Two things make this hard. The projection stretches non-uniformly with latitude and wraps at the seam, so any object crossing it appears as two fragments at opposite image edges. And the entire labelled corpus is 955 frames of construction interior that looks nothing like COCO.
The geometry problem yields to re-projection: each frame becomes six perspective cube faces, segmented by COCO-pretrained models in their native domain, then merged back into the panorama by a seam-aware stitch. Those two steps together take RF-DETR-Seg from 0.161 to 0.706 mAP50, the projection carrying most of it. The data problem needed a different answer, a procedural Blender pipeline that builds, renders, and labels construction scenes in one pass, which lifted the result to 0.735 with the largest gains on the rarest classes.
What was built
1. Cubemap projection and seam-aware stitching
Re-projecting into six perspective faces lets pretrained models work in their native geometry, at the cost of five new seams where there was one.
- Each panoramic frame becomes six 90° faces, segmented independently, then painted back into equirectangular coordinates by nearest-neighbour mapping.
- Fragments merge when their dilated masks touch, regardless of predicted class, because a sliver of an object on one face is routinely misclassified. The merged label is a vote weighted by score times area on face.
- Projection is the single largest gain in the pipeline: RF-DETR-Seg goes from 0.161 to 0.591 mAP50 before stitching, and 0.706 after.
2. Backbone comparison and learning-rate sweep
Three per-face backbones spanning CNN single-stage, query-based transformer, and real-time DETR, each fine-tuned on cube-face crops for 50 epochs with early stopping.
- RF-DETR-Seg won, and not narrowly: 0.706 mAP50 against 0.577 for YOLO11-Seg and 0.239 for Mask2Former.
- The cubemap is not universally free. It roughly doubles Mask2Former and lifts RF-DETR-Seg more than fourfold, but YOLO11-Seg is marginally better left on the raw panorama, 0.587 against 0.577.
- The learning-rate sweep put 5e-5 first on cube faces at 0.7618 and the 1e-4 baseline third at 0.7414. After re-projection the order inverts: 1e-4 holds 0.706 while 5e-5 falls to 0.605.
3. Procedural synthetic construction scenes
955 labelled frames cannot carry a nine-class taxonomy, so the rest of the training data is manufactured: Blender assembles a site, renders it, and annotates it in the same pass.
- Around 30 sampled parameters (room dimensions, column grid, opening counts, materials, lighting) drive constructive solid geometry over box primitives. Walls are solid blocks with doors and windows subtracted; columns and staircase steps sit on randomised grids.
- Seven scenario types override random sampling to guarantee content, so rare classes stop being rare. One of them, empty structural background, renders a finished interior with zero annotated instances as a hard negative against false positives on bare walls and floors.
- Three cameras per scene render at 800×800 and exactly 90° field of view, matching the cube faces from stage 1 so synthetic frames carry no systematic scale bias against real ones.
- A second render pass records the generating object per pixel, giving per-instance COCO masks at zero labelling cost. A shared logical instance id unions the twelve to twenty step boxes of a staircase into one annotation, matching the real-data convention.
- Mixing 300 generated scenes into training lifted overall mAP50 from 0.706 to 0.735. The two rarest classes moved most, elevator by 0.135 and staircase by 0.052, and no class went backwards.
4. Judging synthetic data without training on it
Training a detector on a fresh batch takes around 24 hours, far too slow to steer generation. A dashboard closes the loop in about five minutes instead.
- Pixel and CIELAB statistics run over real and synthetic sets in parallel: brightness, saturation, edge density, per-class mask-area distributions, reported as per-metric deltas and an aggregate Bhattacharyya distance.
- Pixel statistics can match while the scenes contain the wrong things, so DINOv2 features at 768 dimensions per image add a centroid cosine distance, Kernel Inception Distance, and a nearest-real-neighbour distribution.
- Together the two catch the right colours, wrong things failure mode that neither finds alone.
5. Temporal tracking
Per-frame masks become persistent identities across the 1 FPS stream, reasoned on the sphere rather than in flat pixels.
- Each mask centroid converts to longitude and latitude, so wrapping at the seam is handled natively and a per-track Kalman filter can run a constant-velocity model directly on the sphere.
- Inter-frame camera rotation is estimated and applied to each track before matching, so identities survive the large motion between frames at one frame per second.
- Hungarian assignment over mask IoU, Mahalanobis centroid distance, shape and colour descriptors links detections to tracks. Class is a running vote, and tracks unmatched for five frames are dropped.
- Tracking was assessed qualitatively on held-out sequences. Every number quoted on this page measures per-frame segmentation, not identity persistence.
Technical details
Six perspective cube faces per panoramic frame, an instance-segmentation model on each, a seam-aware merge back to equirectangular, and an online tracker over the result.
- Input: 360° equirectangular video at 2880×1440, sampled at 1 FPS
- Output: per-instance masks with class labels and identities that persist across frames
- Projection: cubemap, six perspective faces at 90° field of view
- Taxonomy: nine classes (column, wall column, door, dismounted door, door frame, inner window, outer window, elevator, staircase), three of them confusable pairs
- Real data: 955 frames across three sequences, with pixel-accurate instance masks supplied by HILTI. Split 70/15/15 on contiguous blocks within each sequence to avoid leakage between near-duplicate frames
- Metric: mAP50, mean average precision at 0.5 mask overlap. Reported on the stitched panorama unless named as face-space, which scores the cube faces before re-projection
- Synthetic data: 300 additional procedural scenes rendered with BlenderProc and Cycles at 800×800
- Training: 50 epochs with early stopping, horizontal flip, hue/saturation/brightness jitter, copy-paste oversampling of rare classes, random erasing
- Best model: RF-DETR-Seg (DINOv2 backbone) at learning rate 1e-4
- Tracking: spherical Kalman filter in longitude/latitude with Hungarian assignment, ego-motion compensated
Key technical decisions
- Cubemap re-projection over spherical CNNs: Re-projecting to perspective faces reuses COCO-pretrained checkpoints unchanged. Spherical convolutions are more principled but need custom operators and forfeit the pretraining, which is the one thing a 955-frame dataset cannot afford to lose.
- Merge across classes, not within them: An object sliced by a face boundary reprojects to fragments with zero IoU and often disagreeing labels. Merging on dilated-mask adjacency regardless of class, then voting by score times area, recovers the object where an IoU-and-class merge drops it.
- Tune on the metric you ship: Learning rate 5e-5 wins on cube faces at 0.7618 mAP50 and loses in equirectangular space at 0.605, where 1e-4 holds 0.706. Per-face accuracy is not the deliverable, so the per-face ranking was the wrong one to trust.
- Hard negatives in the synthetic mix: A quarter of generated scenes are fully built interiors with zero annotated instances. Without them the detector fires on bare walls and floors, which the added synthetic geometry makes more frequent rather than less.
- Statistics over renders, not more renders: A 24-hour training run per batch made generation unsteerable. A five-minute dashboard over pixel and DINOv2 feature distances made it possible to reject a bad batch before paying to train on it.
Results
+0.545
mAP50, pipeline vs direct inference
0.161 running the same model straight on panoramic frames, 0.706 through projection and stitching.
+0.135
Elevator AP50 from synthetic data
| Class | Real | Real + synthetic | Δ |
|---|---|---|---|
| Elevator | 0.577 | 0.712 | +0.135 |
| Staircase | 0.368 | 0.420 | +0.052 |
| Outer window | 0.530 | 0.560 | +0.030 |
| Inner window | 0.505 | 0.523 | +0.018 |
| Column | 0.479 | 0.494 | +0.015 |
| Wall column | 0.585 | 0.600 | +0.015 |
| Door frame | 0.192 | 0.202 | +0.010 |
| Door | 0.444 | 0.444 | 0.000 |
| mAP50 | 0.706 | 0.735 | +0.029 |
Challenges & tradeoffs
- Stitching costs part of what projection buys: Measured on cube faces the best model reaches 0.786 mAP50; after re-projection and merging into the panorama it reads 0.735. Roughly five points are lost in the stitch, and objects spanning a face border are still occasionally missed outright.
- Synthetic renders are not photoreal: Against real footage they carry roughly 4× lower edge density and 2× lower colour saturation. They still improve every class, but that gap is the ceiling on how much more they can give without a learned refiner.
- Class imbalance survives the split: Columns appear in nearly every frame of an open floor plan while elevators, staircases, and dismounted doors sit in a handful. Door frame stays the hardest at 0.202 AP50 even after synthetic augmentation.
- Temporal correlation in the data: At 1 FPS adjacent frames are near-duplicates, so a random split leaks the test set into training. Splitting each sequence into contiguous blocks first costs distributional balance and buys an honest number.
- Objects on a face border can vanish entirely: Voting fixes a fragment that is misclassified, but not one the detector never proposed. The untried fix is to run inference over several cubemap rotations of the same frame, so every region sits away from a boundary on at least one of them.
What I learned
- Matching the pretraining domain is the highest-leverage move on a small dataset. Re-projection took the same backbone from 0.161 to 0.706 mAP50 without touching the architecture.
- Optimise against the metric you ship. Learning rate 5e-5 leads on cube faces at 0.7618 and trails in equirectangular space at 0.605, where 1e-4 holds 0.706.
- Synthetic data pays where labels are scarcest. Overall mAP50 moved 0.706 to 0.735, but elevator gained 0.135 and staircase 0.052, and no class regressed.
- Synthetic-data pipelines are systems problems, not rendering ones. A five-minute statistical dashboard steered generation better than higher render fidelity, because a 24-hour training run per batch makes iteration impossible.
Gallery
01 / 08The equirectangular seam in a harmless position. The left and right borders of the image are adjacent points on the sphere, so anything crossing them is split in two. Here the seam lands in open floor and costs nothing. 02 / 08The same scene rotated so the seam cuts through columns. Each now appears as two disconnected fragments at opposite image edges, which a detector reads as separate objects. Recovering these is what the seam-aware merge exists for. 03 / 08The full pipeline. Each panoramic frame is projected onto six 90° cube faces, segmented independently by one of three backbones, merged back into panorama coordinates, then linked across frames by a tracker holding Kalman state, appearance, and class votes per identity. 04 / 08Re-projection leaves one object as several fragments with disagreeing labels: wall column at 0.82 and 0.32, door frame at 0.30, inner window at 0.44. Merging on dilated-mask adjacency and voting by score times area recovers a single outer window at 0.96. 05 / 08The synthetic pipeline. Around 30 sampled parameters build a scene from box primitives, domain randomisation varies materials, lighting, and clutter, and a second render pass emits per-instance COCO masks. The samples below show the labels arriving for free. 06 / 08Backbone comparison in equirectangular space. RF-DETR-Seg gains most from the cubemap, 0.161 to 0.706 mAP50, and Mask2Former roughly doubles. YOLO11-Seg is the exception: at 0.577 against 0.587 it is marginally better left on the raw panorama. 07 / 08Where each post-processing stage lands. Face-space scores are always highest, unstitched re-projection always lowest, and seam-aware stitching recovers most of the difference. For RF-DETR-Seg with synthetic data: 0.786, 0.615, and 0.735 respectively. 08 / 08The learning-rate sweep, measured on cube faces. Read here, 5e-5 wins at 0.7618. Measured after re-projection the same run reads 0.605 against 0.706 for the 1e-4 baseline, which is why the baseline learning rate stayed where it was.