Stateless Multimodal Routing: The Ingestion Pipeline

WebRTC Video Routing via the LiveKit SFU

For visual telemetry, the orchestration layer deploys an embedded, local LiveKit Selective Forwarding Unit (SFU) directly on the bare-metal edge nodes. [61] Unlike centralized cloud video APIs — which compress video to H.264, ship it over the internet, and await server-side inference — the local SFU operates on raw, low-latency feeds. [61]

LiveKit serves as the real-time media backbone, transporting voice and video over WebRTC. [61] The SFU does not interpret, reason about, or analyze the video; its sole function is deterministic, latency-optimized routing. [61] It manages session parameters over WebSockets, transports the media securely via Datagram Transport Layer Security (DTLS) and the Secure Real-time Transport Protocol (SRTP), and forwards spatial video frames to the appropriate tenant vision models. [61]

• Synchronous observation bundling: to satisfy the requirements of robotics and spatial-awareness policy, outgoing video frames and state packets must arrive bundled. The livekit/portal implementation appends the sender's monotonic clock timestamp (for example, timestamp_us) as packet-trailer metadata on every outgoing frame. [63] This guarantees that multi-camera arrays produce perfectly synchronized observations per system tick, letting the backend vision models process aligned stereoscopic frames without jitter-induced hallucination.
• Frame decoding: video streams are decoded the moment they reach the NVIDIA L40S, using the GPU's three onboard NVDEC engines. [51] This bypasses CPU decoding overhead entirely.
• Zero-retention mechanism: once a spatial frame has been parsed into structured contextual data — entity bounding boxes, identification hashes, coordinate mapping — by the tenant's vision model, the raw frame buffer in GPU VRAM is overwritten. No uncompressed video frame persists longer than the inference duration.

Telephonic and Spatial Audio Forking via Asterisk PBX

Acoustic telemetry — spatial microphones and telephonic inputs — is ingested through a localized Asterisk Private Branch Exchange (PBX). Traditional audio integration relies on application-layer polling such as AGI or EAGI, which operate in blocking modes with limited audio access. [64] The orchestration layer replaces this with Asterisk's AudioSocket protocol and the Asterisk REST Interface (ARI) ExternalMedia channels. [64]

Dialplan and Stasis initiation: when an inbound audio event reaches the PBX, Asterisk answers it and routes it to a Stasis application via the dialplan (extensions.conf), handing control of the channel to the orchestration layer's ARI client. [65]

Snoop channel instantiation: the ARI client creates a mixing bridge and attaches a Snoop channel to passively fork the raw audio, letting the agent monitor the session bidirectionally without disrupting it. [67]

ExternalMedia routing: an ExternalMedia channel is instantiated; the client queries the UNICASTRTP_LOCAL_ADDRESS and UNICASTRTP_LOCAL_PORT variables to point the stream at a localized UDP port on the loopback interface (127.0.0.1). [65]

The channel is configured through a strict JSON payload injected via the ARI REST endpoint. [69]

{
  "channelId": "SI_EM_AUDIO_01",
  "app": "software_integrator",
  "external_host": "127.0.0.1:10000",
  "encapsulation": "rtp",
  "transport": "udp",
  "connection_type": "client",
  "format": "slin16",
  "direction": "both"
}

Payload determinism: the audio format is bound to slin16 (16 kHz, 16-bit signed linear PCM). [65] Converting to slin16 avoids the degradation introduced by telephony codecs such as μ-law or A-law and matches the native sample rate expected by modern speech-to-text models. [69]

RTP framing mechanics: the slin16 audio is framed at precise 20-millisecond intervals to prevent buffer bloat. [65] At a 16,000 Hz sample rate a 20 ms frame yields exactly 320 samples; at 16-bit depth (2 bytes per sample) every RTP payload is exactly 640 bytes. [65] This deterministic packet size aligns with memory-allocation limits, eliminating fragmentation and ensuring that memory boundaries are respected during DMA transfers.

Ephemeral Ring Buffers and Streaming Whisper Processing

The 640-byte audio payloads are depacketized — RTP headers stripped to isolate the raw PCM — and written into volatile tmpfs ring buffers mounted in /dev/shm (shared memory). [65] This forces the operating system to allocate the buffer strictly in RAM, preventing any block-level disk I/O or swap-file caching. [70]

These continuous payloads stream directly into an optimized whisper.cpp instance running locally in the GPU execution space. [72] Whisper processes the ambient audio in real time, using server-side Voice Activity Detection (VAD) to trigger inference boundaries and executing speech-to-text (STT) and diarization to produce structured JSON (timestamp, speaker ID, text). [65]

The core of the stateless mandate is enforced here: the instant the STT model yields its structured string, the /dev/shm ring-buffer pointer is advanced, dropping the raw audio payload. The raw biometric voice data ceases to exist within milliseconds of its creation. The resulting structured JSON is handed off to the tenant's isolated data lake. In this way the orchestration layer extracts the semantic reality of a room while cryptographically guaranteeing the destruction of the underlying raw biometric telemetry.