# MB-001 research: TouchDesigner → Apple Vision Pro live stereo video

**Date:** 2026-11-03 (Asia/Hong_Kong)  
**Scope:** Research only — how to stream live video from TouchDesigner into a native visionOS app with a separate frame for each eye.  
**Context:** MakerBot idea MB-001 (Vision Pro head-tracked camera rig).  
- Path A: two Insta360 Go Ultras  
- Path B: USB webcams via onboard computer (user owns at least one Quanrui Vision **USB48MP01-QFZ** UVC module)  
- Headset: **Wi-Fi 6 only** (802.11ax @5 GHz, max PHY ~1200 Mbps / 80 MHz / 2 spatial streams per Apple deployment guide)

Related (sibling, if/when present): `MB-001-avp-live-video.md` — general visionOS live-video APIs (non-TD). Do not overwrite that file; this doc is TD-focused.

---

## 1. TouchDesigner streaming outputs

Sources: Derivative docs for [WebRTC](https://docs.derivative.ca/WebRTC), [WebRTC DAT](https://docs.derivative.ca/WebRTC_DAT), [Video Stream Out TOP](https://docs.derivative.ca/Video_Stream_Out_TOP), [NDI Out TOP](https://docs.derivative.ca/NDI_Out_TOP), [NDI](https://docs.derivative.ca/NDI), [Syphon Spout Out TOP](https://docs.derivative.ca/Syphon_Spout_Out_TOP), [Touch Out TOP](https://docs.derivative.ca/Touch_Out_TOP), [Interoperability](https://docs.derivative.ca/Interoperability).

| Output | Where it runs | Leaves TD machine over Wi‑Fi to AVP? | Typical latency class | Notes |
| --- | --- | --- | --- | --- |
| **Syphon** (macOS) | Same Mac GPU shared memory | **No** — local only | ~1 frame / few ms | Via Syphon Spout Out TOP. 8-bit RGBA. Bridge needed: Syphon → encoder (NDI/WebRTC/SRT) → network. |
| **Spout** (Windows) | Same PC GPU shared memory | **No** — local only | ~1 frame / few ms | Nvidia/AMD GPU required (Intel not supported). Up to 32-bit float RGBA. Same bridge pattern as Syphon. |
| **NDI Out TOP** | LAN (Ethernet/Wi‑Fi) | **Yes** | **Lowest network** on LAN — often ~1–3 frames / ~16–100 ms glass-to-glass when healthy | SpeedHQ CPU codec, YUV 4:2:2, mDNS discovery. Higher bandwidth than H.264. Derivative: “very low latency vs RTSP/H.264, at the cost of higher network bandwidth.” visionOS receive needs NDI Apple SDK **6.2+**. |
| **WebRTC** (WebRTC DAT + Video Stream Out TOP) | LAN or WAN (P2P; STUN/TURN) | **Yes** | **Low** — often ~50–200+ ms on LAN with low-latency encoder presets; higher with jitter buffers / TURN | Official TD path. Needs signaling server (Palette `signalingServer` / custom). Video Stream Out quality presets include **Low-Latency Low/Med/High**. Max B-frames → more latency. **Hardware note:** Video Stream Out TOP’s NVIDIA HW encoder path is **Windows + Nvidia**; WebRTC mode rides that TOP. |
| **SRT** (Video Stream Out TOP) | LAN/WAN | **Yes** | **Low–medium** — often ~50–200 ms (tunable latency buffer) | Listener/caller URLs. H.264/H.265. Good for unreliable links; can carry per-frame metadata. visionOS needs a custom SRT client (no first-party SRT in AVFoundation). |
| **RTSP** (Video Stream Out TOP as server) | LAN | **Yes** | **Medium** — often ~100–500+ ms depending on client buffering | Nvidia HW encoder (Windows). Multiple clients. Unfriendly for head-tracked telepresence vs NDI/WebRTC. |
| **RTMP / Enhanced RTMP** | Usually to CDN/ingest | **Possible but wrong tool** | **High** — typically seconds | Broadcasting (Twitch/YouTube). Not for closed-loop head tracking. |
| **HLS / HTTP segment streaming** | Via external packager or browser workflows | Indirect | **High** — classic HLS 15–30 s; LL-HLS often ~2–5 s | Fine for “watch later,” not telepresence. AVPlayer on visionOS is excellent for HLS but latency is the killer. |
| **Touch Out / Touch In TOP** | TD ↔ TD over TCP/IP | Not useful for AVP | Tunable queue; not designed for Wi‑Fi/internet | Proprietary TD↔TD. Prefer NDI/WebRTC for AVP. |
| **Custom TOP / UDP / shared Metal** | Custom | Yes if you invent the protocol | Can be lowest if carefully written | Highest engineering cost. Possible with Custom OPs + a Metal/CompositorServices receiver. |

### What can leave the TD box toward the Vision Pro?

**Direct over Wi‑Fi:** NDI, WebRTC, SRT, RTSP, RTMP, (LL-)HLS.  
**Not over the network:** Syphon, Spout (same-machine GPU share only). Common bridge: **Mac TD → Syphon → OBS/NDI|WebRTC encoder**, or stay in TD and use **NDI Out** / **WebRTC** natively.

### Latency ranking for TD *outputs* (to a nearby AVP on the same 5 GHz Wi‑Fi 6 LAN)

1. **Syphon/Spout** (local only — then add a network hop)  
2. **NDI** (best *end-to-end* native TD→LAN option for production-style video)  
3. **WebRTC** with low-latency quality + zero B-frames + LAN-only ICE (no TURN)  
4. **SRT** with small latency window  
5. **RTSP**  
6. **LL-HLS / HLS / RTMP**

Caveat: measured glass-to-glass depends on encode settings, Wi‑Fi airtime, and how the visionOS app renders (RealityKit vs Metal CompositorServices). Head-tracked *servo* latency is a separate loop from video latency; video lag + physical lag stacks into motion sickness risk.

---

## 2. Receiving & rendering in a native visionOS app (per-eye)

User builds their **own** visionOS app (developer account already in use). Goal: left camera → left eye, right camera → right eye.

### 2.1 RealityKit paths

| API | Role | Stereo notes | Docs / refs |
| --- | --- | --- | --- |
| **VideoMaterial** | Plays video as a material on a mesh | `init(avPlayer:)` for AVPlayer; `init(videoRenderer:)` for custom frame push | [VideoMaterial](https://developer.apple.com/documentation/realitykit/videomaterial) |
| **VideoPlayerComponent** | Entity component for video | Works with AVPlayer or **AVSampleBufferVideoRenderer** | Apple sample: *Rendering stereoscopic video with RealityKit* |
| **AVSampleBufferVideoRenderer** | Enqueue `CMSampleBuffer`s yourself | Convert SBS → multi-layer tagged buffers (leftEye/rightEye) for true stereo | Same sample; tags via `CMTaggedDynamicBuffer` / `CMStereoViewComponents` |
| **LowLevelTexture** + **TextureResource** | Metal-writable RealityKit texture | Update every frame from decoded CVPixelBuffer/MTLTexture; pair with ShaderGraph **Camera Index Switch** | WWDC24 [Bring your iOS or iPadOS game to visionOS](https://developer.apple.com/videos/play/wwdc2024/10093/); Apple *Displaying a stereoscopic image* sample |
| **ShaderGraph Camera Index Switch** | Picks left vs right texture per eye | Feed `LeftTexture` / `RightTexture` parameters | [Displaying a stereoscopic image](https://developer.apple.com/documentation/visionos/displaying-a-stereoscopic-image) (Apple sample) |

**Recommended RealityKit stereo patterns for live streams:**

1. **Side-by-side (SBS) single stream** → split each frame → tag layers 0/1 as left/right → `AVSampleBufferVideoRenderer` + `VideoPlayerComponent` (matches Apple’s stereoscopic video sample).  
2. **Two independent textures** → two `LowLevelTexture`s (or one SBS texture with UV split in ShaderGraph) → Camera Index Switch. Best when you have two WebRTC/NDI tracks.  
3. **MV-HEVC / spatial video** — great for *file* playback and Apple Immersive; not a practical live encode path from TD/Pi today unless you build a custom MV-HEVC live encoder (hard).

### 2.2 CompositorServices / Metal (full ImmersiveSpace)

For lowest presentation latency and full control: `LayerRenderer` in an ImmersiveSpace, render into per-eye (or layered type2DArray) drawables each frame. See Apple: *Drawing fully immersive content using Metal* / WWDC23–25 Metal immersive sessions. You own decode → MTLTexture → per-eye blit. Highest code cost; best motion-to-photon once streaming works.

### 2.3 Transports into the app

| Transport | visionOS support | Fit for MB-001 |
| --- | --- | --- |
| **NDI SDK (Apple) 6.2+** | Official **visionOS** library (`libndi_visionos`); 6.3.2 fixed missing recv-advertiser symbols | **Strong starter** if TD machine is on same LAN. Declare Bonjour `_ndi._tcp`, `NSLocalNetworkUsageDescription`; multicast entitlement often needed for discovery. |
| **WebRTC (LiveKit client-sdk-swift, etc.)** | LiveKit Swift supports visionOS (recent changelogs; stereo RealityKit still DIY — see [issue #810](https://github.com/livekit/client-sdk-swift/issues/810)) | Strong if you want browser-parity, NAT traversal, or TD WebRTC DAT. Custom `VideoRenderer` → RealityKit. |
| **HLS / LL-HLS via AVPlayer** | First-class | Too slow for head-tracked telepresence. |
| **RTSP / SRT** | No turnkey Apple API — bring your own library | Possible; more DIY than NDI/WebRTC. |
| **Safari / WebXR** | Not the user’s preferred path (they want a native app) | Skip for production; maybe a smoke test. |

### 2.4 Stereo packing options

1. **Side-by-side (SBS)** — one encode, one NDI/WebRTC track; app splits. Simplest sync.  
2. **Two streams / two tracks** — left and right separately; must clock-sync; more bandwidth; clearer per-eye quality control.  
3. **Over/under** — same idea as SBS with different layout.  
4. **MV-HEVC** — Apple’s native 3D file format; live from TD is uncommon.

For MB-001 Path B (two USB cams): **SBS composite in TD (or on the Pi)** is the easiest first sync model; graduate to dual tracks if needed.

---

## 3. Open-source / verified projects

Verified existing (fetched 2026-11-03):

| Project | URL | What it shows |
| --- | --- | --- |
| **TD-WebRTC-LAN** | https://github.com/jshea2/TD-WebRTC-LAN | TouchDesigner → LAN browsers via WebRTC (low-latency). Good TD-side reference; AVP would use a native WebRTC client instead of (or in addition to) a browser page. |
| **TouchDesigner SignalingAPI** | https://github.com/TouchDesigner/SignalingAPI | Official signaling schemas for TD WebRTC palette. |
| **WebRTC Remote Panel Web Demo** | https://github.com/TouchDesigner/WebRTC-Remote-Panel-Web-Demo | Browser receiver for TD remote panels. |
| **iControl OSC & NDI** | https://github.com/gwangyu-lee/icontrol-osc-ndi | Vision Pro app: receive NDI + send OSC (incl. TD workflows). Companion write-ups on gwangyulee.com. |
| **iXR-OSC-NDI** | https://github.com/gwangyu-lee/iXR-OSC-NDI | AVP receives 360 NDI; hands → OSC. |
| **KlakNDI-visionOS-Demo** | https://github.com/gwangyu-lee/KlakNDI-visionOS-Demo | Unity KlakNDI visionOS demo path (pre/around official SDK support). |
| **KlakNDI visionOS discussion** | https://github.com/keijiro/KlakNDI/issues/203 | Community notes; NDI 6.2 added official visionOS SDK support. |
| **LiveKit Swift + stereo RealityKit** | https://github.com/livekit/client-sdk-swift/issues/810 | Guidance: custom VideoRenderer → VideoMaterial / sample buffers for SBS stereo. |
| **Speaker Deck: NDI → Vision Pro (incl. TD send)** | https://speakerdeck.com/jyouryuusui/ndi-de-apple-vision-pro-ni-riarutaimu-eizou-soushin | Practical checklist: NDI Apple SDK, Bonjour `_ndi._tcp`, multicast entitlement wording, **TouchDesigner NDI send → Vision Pro receive** PoC (Insta360 Ace Pro used in demo). |

No single “clone this and you’re done” open-source **stereo telepresence** TD→AVP app was found; the NDI AVP + TD send pattern above is the closest battle-tested path.

---

## 4. Platform limits (AVP + Wi‑Fi 6 + telepresence)

### Wi‑Fi 6 on Vision Pro

Per [Apple Platform Deployment — Wi‑Fi specs](https://support.apple.com/guide/deployment/wi-fi-ethernet-specifications-apple-devices-dep268652e6c/web) (Vision Pro / Vision Pro M5):

- **802.11ax @5 GHz:** max PHY **1200 Mbps**, **80 MHz**, 2 spatial streams  
- **ax @2.4 GHz:** 195 Mbps — avoid for video  
- Not Wi‑Fi 7 / not 6 GHz Wi‑Fi 6E on the headset  

Practical tips: dedicated 5 GHz SSID, TD/Pi host **wired to the AP** if possible, AVP on 5 GHz only, minimize other airtime. NDI High Bandwidth stereo 1080p60 can be multi-hundred Mbps; HX/WebRTC H.264 at 8–20 Mbps/eye is gentler on Wi‑Fi.

### Latency budget (head-tracked cameras + servos)

| Budget | Feel |
| --- | --- |
| **< ~20–40 ms** video motion-to-photon | Comfortable telepresence target |
| **~50–80 ms** | Often usable; notice lag on fast head turns |
| **> ~100 ms** + servo lag | Strong sickness risk when the world moves after your head |

Remember: **video latency + servo mechanical latency + pose-send RTT** stack. Prefer low-latency video *and* a fast local control link (UDP/BLE) for servos.

### App lifecycle / permissions

- Streaming receivers generally need the app **foreground**; ImmersiveSpace / full immersion for locked stereo view.  
- **Local Network** permission: `NSLocalNetworkUsageDescription`.  
- NDI discovery: `NSBonjourServices` → `_ndi._tcp` (and apply for **multicast entitlement** if required by your discovery mode).  
- Background networking for continuous camera preview is restricted; design as an active immersive experience.  
- Safari WebRTC may work for a quick test but is not the user’s intended product path.

---

## 5. Options ranked by likely end-to-end latency (TD → AVP stereo)

1. **TD NDI Out (SBS or dual sources) → native visionOS NDI SDK receiver → RealityKit LowLevelTexture / AVSampleBuffer stereo or Metal CompositorServices** — best LAN production path; proven TD→AVP by community PoCs.  
2. **TD WebRTC DAT + Video Stream Out (low-latency preset, 0 B-frames) → LiveKit or custom WebRTC Swift → custom VideoRenderer → stereo RealityKit** — slightly more moving parts; better WAN/NAT story.  
3. **Mac-local Syphon → same-Mac bridge encoder → (1) or (2)** — only if another Mac app must process frames before send; adds a hop.  
4. **TD SRT → custom visionOS SRT client** — solid contribution protocol; more DIY on device.  
5. **TD RTSP → third-party client** — workable prototype, higher buffer risk.  
6. **HLS / LL-HLS / RTMP** — reject for closed-loop head tracking.

**Spout/Syphon alone cannot reach the headset.**

---

## 6. Recommended starter stack (MB-001)

### Phase 0 — smoke test (hours)

- TouchDesigner on a **Windows+Nvidia** or Mac on the **same 5 GHz LAN** as the AVP.  
- Composite Path B cams (or test patterns) to **SBS 1920×1080 or 3840×1080 @30**.  
- **NDI Out TOP** source name e.g. `MB001-Stereo`.  
- On AVP: build a tiny native app with **NDI SDK 6.2+**, Bonjour + local network strings; draw SBS on a quad first (mono), then add Camera Index Switch / tagged stereo layers.  
- Reference implementations: gwangyu-lee NDI AVP apps + Speaker Deck TD→NDI→AVP notes.

### Phase 1 — stereo + pose

- Per-eye textures via ShaderGraph Camera Index Switch or AVSampleBuffer multi-layer.  
- Parallel: stream head pose from visionOS → ESP32/servos (UDP). Measure **pose RTT** and **video lag** separately (clap board / LED flash).

### Phase 2 — WebRTC fallback / Pi path

- If NDI bandwidth hurts on Wi‑Fi: switch TD (or the Pi) to **WebRTC H.264** low-latency.  
- Path B onboard computer: prefer a board with **hardware encode** if doing dual 1080p30; Pi 5 is CPU-encode only (see ideas.md caveats). USB48MP01-QFZ: use **1080p/4K MJPEG @30**, not 48MP stills; need a **second matched unit** for stereo.

### Explicit non-goals for v1

- Syphon/Spout as the AVP link  
- HLS/RTMP  
- Relying on Wi‑Fi 7 to the headset (headset is Wi‑Fi 6)

---

## 7. Key gotchas

1. **Syphon/Spout never cross the air** — always bridge.  
2. **Video Stream Out HW encode = Windows + Nvidia** in Derivative’s docs — plan machine choice accordingly for RTSP/SRT/RTMP/WebRTC-via-that-TOP.  
3. **NDI loves Ethernet**; on Wi‑Fi use conservative resolution/FPS or HX-style compressed paths / WebRTC.  
4. **Multicast entitlement + Bonjour** for NDI discovery on visionOS — budget Apple approval time.  
5. **Stereo sync** — SBS from one compositor is easier than two free-running USB cams.  
6. **Latency stacking** with head-tracked servos — optimize video *and* control loops.  
7. **LiveKit/WebRTC stereo** is DIY on RealityKit (no turnkey stereo component as of late 2025 issue discussion).  
8. **Foreground ImmersiveSpace** — don’t assume background streaming.  
9. User camera **USB48MP01-QFZ**: UVC, ~38 mm, prefer MJPEG 1080p/4K@30; confirm with `v4l2-ctl --list-formats-ext`; second unit required for Path B stereo.

---

## 8. Source index (primary)

- https://docs.derivative.ca/WebRTC  
- https://docs.derivative.ca/WebRTC_DAT  
- https://docs.derivative.ca/Video_Stream_Out_TOP  
- https://docs.derivative.ca/NDI_Out_TOP  
- https://docs.derivative.ca/NDI  
- https://docs.derivative.ca/Syphon_Spout_Out_TOP  
- https://docs.ndi.video/all/developing-with-ndi/sdk/release-notes (visionOS in 6.2; symbol fix 6.3.2)  
- https://developer.apple.com/documentation/realitykit/videomaterial  
- https://developer.apple.com/videos/play/wwdc2024/10093/  
- https://developer.apple.com/documentation/visionos/displaying-a-stereoscopic-image  
- https://support.apple.com/guide/deployment/wi-fi-ethernet-specifications-apple-devices-dep268652e6c/web  
- https://developer.apple.com/documentation/technotes/tn3179-understanding-local-network-privacy  
- https://github.com/jshea2/TD-WebRTC-LAN  
- https://github.com/TouchDesigner/SignalingAPI  
- https://github.com/gwangyu-lee/icontrol-osc-ndi  
- https://github.com/gwangyu-lee/iXR-OSC-NDI  
- https://github.com/livekit/client-sdk-swift/issues/810  
- https://speakerdeck.com/jyouryuusui/ndi-de-apple-vision-pro-ni-riarutaimu-eizou-soushin  

---

*Saved for MakerBot MB-001 build notes. Research-only; no installs or app builds performed in this pass.*