# Bee camera system Notes from poking at the Hivemapper Bee dashcam. The camera path doesn't use V4L2 at all — frames live in `/tmp/recording/pics/` and `/data/recording/cached_observations/`, written by a DepthAI pipeline running on the on-die Myriad X VPU. All access goes through XLink, not `/dev/video*`. ## hardware SoC is Intel Keem Bay (RVC2 — Robotics Vision Core 2): 4× Cortex-A53 @ 1.5GHz, integrated Myriad X VPU with 16 SHAVE cores and a Neural Compute Engine, 10nm. 4GB LPDDR4 on the board, ~3.5GB usable. CMA reserves ~1.34GB for VPU/camera DMA, swap is 2GB. ``` MemTotal: 3,584,000 kB SwapTotal: 2,097,148 kB CmaTotal: 1,408,000 kB # VPU/camera DMA reservation ``` The camera is a Luxonis OAK-1-compatible module — Sony IMX378 (or equivalent 12MP), 4056×3040 native, downscaled to 2028×1024 by the pipeline. ~30 FPS. MIPI CSI-2 into the VPU's ISP; the ARM side never touches it directly. Bus layout: ``` ┌─────────────────────────────────────────────────────┐ │ Intel Keem Bay SoC │ │ ┌──────────┐ ┌──────────┐ ┌────────────────┐ │ │ │ A53 ×4 │ │ Myriad X │ │ Neural Compute │ │ │ └────┬─────┘ └────┬─────┘ └────────┬───────┘ │ │ └────────┬────┴─────────────────┘ │ │ Internal AXI/NoC │ │ ┌─────┬─────┼──────┬───────────┐ │ │ PCIe USB SDIO MIPI CSI │ │ └─────┼─────┼─────┼──────┼───────────┘ │ │ │ │ │ Marvell Telit eMMC IMX378 88W8997 LE910C4 WiFi/BT LTE ``` ## kernel + V4L2 Custom Yocto build with Intel VPU drivers. `kmb_cam` / `kmb_imx412` may be loaded for the sensor itself, plus the usual `videodev` + `v4l2_fwnode`. None of it is reachable via `/dev/video*` during normal operation — the VPU owns the camera hardware exclusively and the host talks to it over XLink (PCIe transport on Keem Bay; USB on desktop OAK devices). VPU is controlled via sysfs at `/sys/class/vpu/`. Firmware is loaded by writing the filename to the `fwname` attribute. Two firmwares are present: - `luxonis_vpu.bin` — DepthAI firmware (what we want) - `vpu_nvr_b0.bin` — Intel HDDL firmware (conflicts, see below) If you want frames without going through the existing stack you have three options: use the depthai_gate / odc-api stack as-is, stop depthai_gate and run your own DepthAI pipeline, or reverse-engineer XLink and roll custom VPU firmware. The first is by far the easiest. ## depthai_gate Python+Flask, listens on localhost:11492, ~158 threads, ~200MB RSS. Lives at `/opt/depthai_gate/` (estimated — not confirmed on-device yet). Service unit looks like: ```ini [Unit] Description=DepthAI Camera Gate After=network.target [Service] Type=simple User=root ExecStart=/opt/depthai_gate/run.py Restart=always ``` What it does: loads `luxonis_vpu.bin` into the VPU, opens the XLink connection, configures the DepthAI pipeline (ColorCamera → ImageManip → XLinkOut, plus optional NeuralNetwork node), captures frames, and drops them into `/tmp/recording/pics/`. Pipeline config probably lives at `/opt/depthai_gate/pipeline.json` or `/data/camera_config.json`, possibly hardcoded. XLink status values seen in logs: ``` xlink_device_status=2 # connected, healthy =1 # connecting / error =0 # disconnected ``` ### the VPU conflict `deviceservice.service` (Intel HDDL / OpenVINO) ships enabled and races depthai_gate for the VPU: 1. HDDL starts at boot, loads `vpu_nvr_b0.bin` 2. depthai_gate starts, overwrites with `luxonis_vpu.bin` 3. HDDL retries XLink every 2s forever, can't talk to the now-Luxonis firmware 4. If depthai_gate restarts after HDDL is already running, HDDL grabs the VPU first and the camera goes dead 5. `secure-wdtclient` watchdog crash-loops on the dead VPU → memory pressure → OOM Fix: ```bash systemctl disable --now deviceservice systemctl mask deviceservice # survives OTA ``` ## map-ai Reads frames from depthai_gate, runs detection on the VPU's NCE, blurs faces and plates, writes results to SQLite (`/data/recording/odc-api.db`) and blurred frames to disk. ```ini [Unit] Description=Map AI Processing After=depthai_gate.service [Service] Type=simple User=root ExecStart=/opt/map-ai/run.py Restart=always ``` Pipeline: ``` frame from depthai_gate │ ▼ ML inference (on VPU NCE) - road sign classifier - face detector - license plate detector │ ▼ privacy blur - Gaussian on faces/plates - cv2.imwrite blurred copy │ ├──► Redis (status keys, not detections) └──► SQLite (observations, landmarks, frames) ``` Models: | Model | Path | Purpose | |-------|------|---------| | Road signs | `/opt/object-detection/model.blob` or `/data/models/` | classification | | Privacy | `/opt/odc-api/python/` or `/data/models/` | face/plate detection | | PVC | `/data/recording/models/pvc.onnx` | unknown — 227 bytes, probably an index file | Privacy model hash gets baked into FrameKm metadata for verification. Redis only carries readiness flags: ``` GET MAP_AI_READY → "True" GET EXTERNAL_MODEL_CLASSIFIER_READY → "True" ``` Detections go to SQLite, not Redis ZSETs. ## frame storage | Path | FS | Purpose | Persists? | |------|----|---------|-----------| | `/tmp/recording/pics/` | tmpfs | live frames | no | | `/tmp/recording/preview/` | tmpfs | preview mode | no | | `/data/recording/cached_observations/` | ext4 | landmark observations | yes | | `/data/recording/framekm/` | ext4 | upload bundles | yes | | `/tmp/rgb/` | tmpfs | frame list files | no | Frames are JPEG at 2028×1024, ~85% quality, ~150-200KB each. Naming: ``` # live /tmp/recording/pics/{system_time_ms}_{frame_id}_{sequence}.jpg e.g. 1709920000123_0001_0042.jpg # cached observations /data/recording/cached_observations/{timestamp}_{subsecond}_{frame_number}.jpg e.g. 1746377552_043000_2945056.jpg ``` `folder_purger` keeps disk under control — when `/tmp/recording/pics/` crosses 400MB, oldest frames go: ``` folder-purger /tmp/recording/pic 400000000 /mnt/data/gps 2000000000 ... ``` SQLite schemas (simplified): ```sql CREATE TABLE frames ( system_time INTEGER PRIMARY KEY, image_name TEXT ); CREATE TABLE observations ( id INTEGER PRIMARY KEY, landmark_id INTEGER, image_name TEXT, x1 REAL, y1 REAL, x2 REAL, y2 REAL, ts INTEGER ); ``` DB lives at `/data/recording/odc-api.db` (also seen as `data-logger.v2.0.0.db`). ## video-processor + FrameKm `video-processor` isn't documented in the firmware I've looked at. Based on naming it bundles frames+metadata into FrameKm tarballs, handles any H.264/H.265 encoding for preview, and orders frames for upload. It doesn't produce raw frames — it only packages already-blurred ones — so it's not useful for camera-access work. A FrameKm is ~1km of driving data: ``` framekm-2024-03-08-12-34-56-abc123.tar ├── manifest.json ├── frame_0001.jpg ├── frame_0002.jpg ├── ... ├── gnss_auth_buffer.bin └── gnss_auth_signature.bin ``` Manifest: ```json { "name": "framekm-2024-03-08-12-34-56-abc123", "numFrames": 150, "deviceId": "fvhL2I-iCT", "firmwareVersion": "0.0.1", "privacyModelHash": "sha256:abc123...", "gnssAuthBuffer": "base64...", "gnssAuthSignature": "base64...", "gnssAuthPublicKey": "base64...", "createdAt": 1709920000000 } ``` ## odc-api REST API at `http://192.168.0.10:5000/api/1/`. Binds to the AP interface (`wlp1s0f0`) only — not reachable from home LAN without going through the Bee's AP. Preview endpoints: | Endpoint | Method | Notes | |----------|--------|-------| | `/preview/start` | GET | 120s timeout, then auto-stop | | `/preview/stop` | GET | | | `/preview/status` | GET | | | `/preview/metadata` | GET | latest frame metadata | Preview works by writing a new config and bouncing camera-bridge: ```typescript export const startPreview = async () => { await execSync('mkdir /tmp/recording/preview'); writeFileSync(IMAGER_CONFIG_PATH, JSON.stringify(getPreviewConfig())); await execSync(CMD.STOP_CAMERA); await sleep(1000); await execSync(CMD.START_CAMERA); }; ``` The 120s auto-stop is there to protect 4K recording quality. Landmark endpoints (where the cached observation images come out): | Endpoint | Method | Notes | |----------|--------|-------| | `/landmarks/images/:id` | GET | image paths for a landmark | | `/landmarks/:id/chips` | GET | list of chip endpoints | | `/landmarks/:id/chips/:chip_id` | GET | cropped observation JPEG | | `/landmarks/boundingBox/:id` | GET | bbox coords | | `/landmarks/upload` | PUT | upload landmark image | Image retrieval flow: ``` GET /landmarks/images/123 → ["/data/recording/cached_observations/1746377552_043000_2945056.jpg"] GET /landmarks/123/chips/456 → cropped JPEG of the bbox region ``` Camera bridge config: `/opt/camera-bridge/config.json`. Control commands from `bee.ts`: ```typescript export const CMD = { RESTART_CAMERA: 'systemctl restart camera-bridge', START_CAMERA: 'systemctl start camera-bridge', STOP_CAMERA: 'systemctl stop camera-bridge', START_PREVIEW: 'systemctl start camera-preview', STOP_PREVIEW: 'systemctl stop camera-preview', }; ``` There is no `/camera/frame` endpoint. To get a frame you either use preview mode and read `/tmp/recording/preview/`, walk the landmarks API for chips, or SSH in and read `/tmp/recording/pics/` directly. ## full data flow ``` IMX378 → MIPI CSI-2 → VPU ISP → DepthAI pipeline │ depthai_gate (:11492) writes /tmp/recording/pics/ │ ┌───────────────────┴───────────────────┐ ▼ ▼ /tmp/recording/pics/ map-ai (VPU NCE) (raw, purged >400MB) - sign classifier - privacy blur │ ┌────────────────────────────────────┤ ▼ ▼ /data/recording/cached_observations/ odc-api.db (SQLite) (blurred, persistent) landmarks/observations/frames │ │ └────────────────┬───────────────────┘ ▼ odc-api (:5000) /preview/*, /landmarks/* │ ▼ hivemapper-data-logger → FrameKm bundles in /data/recording/framekm/ │ ▼ odc-api → mitmdump :8888 → Cloudflare Workers → HERE OLP ``` Single detection trace: ``` 1. IMX378 → MIPI → VPU ISP → depthai_gate 2. /tmp/recording/pics/1709920000123_0001_0042.jpg 3. map-ai picks it up, runs road sign classifier on the NCE 4. hit (e.g. speed limit 35), faces/plates blurred 5. observations row written + image saved to cached_observations/ 6. landmarks row created/updated (class_label, lat, lon, confidence) 7. /landmarks/last/5 surfaces it; /landmarks/{id}/chips/{chip_id} returns the crop ``` ## replacement notes ### getting a frame without odc-api Direct file read (simplest, race-prone, no metadata): ```bash ssh -p 2222 root@localhost # via Lucy tunnel ls -lt /tmp/recording/pics/ | head -10 # poor man's stream while true; do cp $(ls -t /tmp/recording/pics/*.jpg | head -1) /tmp/current.jpg sleep 0.033 done ``` Redis pub/sub would be cleaner — `r.pubsub().subscribe('frame_ready')` — but I haven't confirmed depthai_gate publishes anything like that. Worth a `redis-cli MONITOR` while recording is live. ### getting a frame without depthai_gate Don't, unless you really mean it. You'd be replacing the whole VPU pipeline: ```python import depthai as dai pipeline = dai.Pipeline() cam = pipeline.create(dai.node.ColorCamera) cam.setResolution(dai.ColorCameraProperties.SensorResolution.THE_4_K) cam.setIspScale(1, 2) # → 2028×1024 xout = pipeline.create(dai.node.XLinkOut) xout.setStreamName("video") cam.video.link(xout.input) with dai.Device(pipeline) as device: q = device.getOutputQueue("video") while True: cv2.imwrite("/tmp/frame.jpg", q.get().getCvFrame()) ``` Breaks everything Hivemapper depends on — map-ai, landmarks, FrameKm. Only useful if the goal is full liberation, not augmentation. ### fastest frame grabs With the stack running: ```bash # direct ssh -p 2222 root@localhost 'ls -t /tmp/recording/pics/*.jpg | head -1 | xargs cat' > frame.jpg # via API (needs preview mode) curl http://192.168.0.10:5000/api/1/preview/start sleep 2 ssh -p 2222 root@localhost 'ls -t /tmp/recording/preview/*.jpg | head -1 | xargs cat' > frame.jpg curl http://192.168.0.10:5000/api/1/preview/stop ``` ### proposed odc-api extension Two new routes — single frame + MJPEG stream: ```typescript router.get('/frame', async (req, res) => { const frames = readdirSync('/tmp/recording/pics') .filter(f => f.endsWith('.jpg')) .sort().reverse(); if (!frames.length) return res.status(404).send('No frames available'); res.sendFile(join('/tmp/recording/pics', frames[0])); }); router.get('/stream', async (req, res) => { res.writeHead(200, { 'Content-Type': 'multipart/x-mixed-replace; boundary=frame', 'Cache-Control': 'no-cache', }); const interval = setInterval(() => { const frames = readdirSync('/tmp/recording/pics') .filter(f => f.endsWith('.jpg')).sort().reverse(); if (frames.length) { const data = readFileSync(join('/tmp/recording/pics', frames[0])); res.write('--frame\r\n'); res.write('Content-Type: image/jpeg\r\n'); res.write(`Content-Length: ${data.length}\r\n\r\n`); res.write(data); res.write('\r\n'); } }, 33); // ~30 FPS req.on('close', () => clearInterval(interval)); }); ``` Long-term replacement shape — leave depthai_gate alone, add a separate watcher service that inotify-tails `/tmp/recording/pics/` and serves frames over HTTP. Doesn't fight the VPU, doesn't break the upload chain. ## things still to confirm - exact depthai_gate pipeline config (find files under `/opt/`) - does depthai_gate publish frame events to Redis at all? (`redis-cli MONITOR`) - camera-bridge vs depthai_gate — what's the actual dependency? (systemd deps, strace) - preview config format — read `getPreviewConfig()` - ML model exact locations (`find /opt /data -name '*.blob' -o -name '*.onnx'`) - frame timestamp accuracy vs GNSS time ## appendix — file paths ``` /tmp/recording/pics/ live frames /tmp/recording/preview/ preview frames /data/recording/cached_observations/ landmark images /data/recording/framekm/ upload bundles /data/recording/odc-api.db SQLite DB /opt/camera-bridge/config.json camera config /opt/depthai_gate/ DepthAI service (estimated) /opt/odc-api/ Node API service /sys/class/vpu/ VPU sysfs ``` ## appendix — boot order ``` multi-user.target ├── redis.service t+2s ├── depthai_gate.service t+8s loads luxonis_vpu.bin ├── map-ai.service t+12s needs depthai_gate ├── hivemapper-data-logger.service t+15s └── odc-api.service t+18s ``` ## appendix — ports ``` 22 sshd TCP AP-only (socket) 5000 odc-api HTTP AP iface 6379 redis TCP localhost 8888 mitmdump HTTP localhost 11492 depthai_gate HTTP/Flask localhost ```