varroa/docs/BEE-CAMERA.md

Bee camera system

Notes from poking at the Hivemapper Bee dashcam. The camera path doesn't use V4L2 at all — frames live in /tmp/recording/pics/ and /data/recording/cached_observations/, written by a DepthAI pipeline running on the on-die Myriad X VPU. All access goes through XLink, not /dev/video*.

hardware

SoC is Intel Keem Bay (RVC2 — Robotics Vision Core 2): 4× Cortex-A53 @ 1.5GHz, integrated Myriad X VPU with 16 SHAVE cores and a Neural Compute Engine, 10nm. 4GB LPDDR4 on the board, ~3.5GB usable. CMA reserves ~1.34GB for VPU/camera DMA, swap is 2GB.

MemTotal:    3,584,000 kB
SwapTotal:   2,097,148 kB
CmaTotal:    1,408,000 kB   # VPU/camera DMA reservation

The camera is a Luxonis OAK-1-compatible module — Sony IMX378 (or equivalent 12MP), 4056×3040 native, downscaled to 2028×1024 by the pipeline. ~30 FPS. MIPI CSI-2 into the VPU's ISP; the ARM side never touches it directly.

Bus layout:

┌─────────────────────────────────────────────────────┐
│                  Intel Keem Bay SoC                 │
│  ┌──────────┐  ┌──────────┐  ┌────────────────┐    │
│  │ A53 ×4   │  │ Myriad X │  │ Neural Compute │    │
│  └────┬─────┘  └────┬─────┘  └────────┬───────┘    │
│       └────────┬────┴─────────────────┘            │
│           Internal AXI/NoC                          │
│     ┌─────┬─────┼──────┬───────────┐                │
│   PCIe   USB  SDIO  MIPI CSI       │                │
└─────┼─────┼─────┼──────┼───────────┘                │
      │     │     │      │
 Marvell  Telit  eMMC  IMX378
 88W8997  LE910C4
 WiFi/BT  LTE

kernel + V4L2

Custom Yocto build with Intel VPU drivers. kmb_cam / kmb_imx412 may be loaded for the sensor itself, plus the usual videodev + v4l2_fwnode. None of it is reachable via /dev/video* during normal operation — the VPU owns the camera hardware exclusively and the host talks to it over XLink (PCIe transport on Keem Bay; USB on desktop OAK devices).

VPU is controlled via sysfs at /sys/class/vpu/. Firmware is loaded by writing the filename to the fwname attribute. Two firmwares are present:

• luxonis_vpu.bin — DepthAI firmware (what we want)
• vpu_nvr_b0.bin — Intel HDDL firmware (conflicts, see below)

If you want frames without going through the existing stack you have three options: use the depthai_gate / odc-api stack as-is, stop depthai_gate and run your own DepthAI pipeline, or reverse-engineer XLink and roll custom VPU firmware. The first is by far the easiest.

depthai_gate

Python+Flask, listens on localhost:11492, ~158 threads, ~200MB RSS. Lives at /opt/depthai_gate/ (estimated — not confirmed on-device yet). Service unit looks like:

[Unit]
Description=DepthAI Camera Gate
After=network.target

[Service]
Type=simple
User=root
ExecStart=/opt/depthai_gate/run.py
Restart=always

What it does: loads luxonis_vpu.bin into the VPU, opens the XLink connection, configures the DepthAI pipeline (ColorCamera → ImageManip → XLinkOut, plus optional NeuralNetwork node), captures frames, and drops them into /tmp/recording/pics/. Pipeline config probably lives at /opt/depthai_gate/pipeline.json or /data/camera_config.json, possibly hardcoded.

XLink status values seen in logs:

xlink_device_status=2   # connected, healthy
                  =1   # connecting / error
                  =0   # disconnected

the VPU conflict

deviceservice.service (Intel HDDL / OpenVINO) ships enabled and races depthai_gate for the VPU:

1. HDDL starts at boot, loads vpu_nvr_b0.bin
2. depthai_gate starts, overwrites with luxonis_vpu.bin
3. HDDL retries XLink every 2s forever, can't talk to the now-Luxonis firmware
4. If depthai_gate restarts after HDDL is already running, HDDL grabs the VPU first and the camera goes dead
5. secure-wdtclient watchdog crash-loops on the dead VPU → memory pressure → OOM

Fix:

systemctl disable --now deviceservice
systemctl mask deviceservice   # survives OTA

map-ai

Reads frames from depthai_gate, runs detection on the VPU's NCE, blurs faces and plates, writes results to SQLite (/data/recording/odc-api.db) and blurred frames to disk.

[Unit]
Description=Map AI Processing
After=depthai_gate.service

[Service]
Type=simple
User=root
ExecStart=/opt/map-ai/run.py
Restart=always

Pipeline:

frame from depthai_gate
        │
        ▼
  ML inference (on VPU NCE)
  - road sign classifier
  - face detector
  - license plate detector
        │
        ▼
  privacy blur
  - Gaussian on faces/plates
  - cv2.imwrite blurred copy
        │
        ├──► Redis  (status keys, not detections)
        └──► SQLite (observations, landmarks, frames)

Models:

Model	Path	Purpose
Road signs	/opt/object-detection/model.blob or /data/models/	classification
Privacy	/opt/odc-api/python/ or /data/models/	face/plate detection
PVC	/data/recording/models/pvc.onnx	unknown — 227 bytes, probably an index file

Privacy model hash gets baked into FrameKm metadata for verification.

Redis only carries readiness flags:

GET MAP_AI_READY                     → "True"
GET EXTERNAL_MODEL_CLASSIFIER_READY  → "True"

Detections go to SQLite, not Redis ZSETs.

frame storage

Path	FS	Purpose	Persists?
/tmp/recording/pics/	tmpfs	live frames	no
/tmp/recording/preview/	tmpfs	preview mode	no
/data/recording/cached_observations/	ext4	landmark observations	yes
/data/recording/framekm/	ext4	upload bundles	yes
/tmp/rgb/	tmpfs	frame list files	no

Frames are JPEG at 2028×1024, ~85% quality, ~150-200KB each.

Naming:

# live
/tmp/recording/pics/{system_time_ms}_{frame_id}_{sequence}.jpg
e.g. 1709920000123_0001_0042.jpg

# cached observations
/data/recording/cached_observations/{timestamp}_{subsecond}_{frame_number}.jpg
e.g. 1746377552_043000_2945056.jpg

folder_purger keeps disk under control — when /tmp/recording/pics/ crosses 400MB, oldest frames go:

folder-purger /tmp/recording/pic 400000000 /mnt/data/gps 2000000000 ...

SQLite schemas (simplified):

CREATE TABLE frames (
    system_time INTEGER PRIMARY KEY,
    image_name TEXT
);

CREATE TABLE observations (
    id INTEGER PRIMARY KEY,
    landmark_id INTEGER,
    image_name TEXT,
    x1 REAL, y1 REAL, x2 REAL, y2 REAL,
    ts INTEGER
);

DB lives at /data/recording/odc-api.db (also seen as data-logger.v2.0.0.db).

video-processor + FrameKm

video-processor isn't documented in the firmware I've looked at. Based on naming it bundles frames+metadata into FrameKm tarballs, handles any H.264/H.265 encoding for preview, and orders frames for upload. It doesn't produce raw frames — it only packages already-blurred ones — so it's not useful for camera-access work.

A FrameKm is ~1km of driving data:

framekm-2024-03-08-12-34-56-abc123.tar
├── manifest.json
├── frame_0001.jpg
├── frame_0002.jpg
├── ...
├── gnss_auth_buffer.bin
└── gnss_auth_signature.bin

Manifest:

{
  "name": "framekm-2024-03-08-12-34-56-abc123",
  "numFrames": 150,
  "deviceId": "fvhL2I-iCT",
  "firmwareVersion": "0.0.1",
  "privacyModelHash": "sha256:abc123...",
  "gnssAuthBuffer": "base64...",
  "gnssAuthSignature": "base64...",
  "gnssAuthPublicKey": "base64...",
  "createdAt": 1709920000000
}

odc-api

REST API at http://192.168.0.10:5000/api/1/. Binds to the AP interface (wlp1s0f0) only — not reachable from home LAN without going through the Bee's AP.

Preview endpoints:

Endpoint	Method	Notes
/preview/start	GET	120s timeout, then auto-stop
/preview/stop	GET
/preview/status	GET
/preview/metadata	GET	latest frame metadata

Preview works by writing a new config and bouncing camera-bridge:

export const startPreview = async () => {
  await execSync('mkdir /tmp/recording/preview');
  writeFileSync(IMAGER_CONFIG_PATH, JSON.stringify(getPreviewConfig()));
  await execSync(CMD.STOP_CAMERA);
  await sleep(1000);
  await execSync(CMD.START_CAMERA);
};

The 120s auto-stop is there to protect 4K recording quality.

Landmark endpoints (where the cached observation images come out):

Endpoint	Method	Notes
/landmarks/images/:id	GET	image paths for a landmark
/landmarks/:id/chips	GET	list of chip endpoints
/landmarks/:id/chips/:chip_id	GET	cropped observation JPEG
/landmarks/boundingBox/:id	GET	bbox coords
/landmarks/upload	PUT	upload landmark image

Image retrieval flow:

GET /landmarks/images/123
  → ["/data/recording/cached_observations/1746377552_043000_2945056.jpg"]

GET /landmarks/123/chips/456
  → cropped JPEG of the bbox region

Camera bridge config: /opt/camera-bridge/config.json. Control commands from bee.ts:

export const CMD = {
  RESTART_CAMERA: 'systemctl restart camera-bridge',
  START_CAMERA:   'systemctl start   camera-bridge',
  STOP_CAMERA:    'systemctl stop    camera-bridge',
  START_PREVIEW:  'systemctl start   camera-preview',
  STOP_PREVIEW:   'systemctl stop    camera-preview',
};

There is no /camera/frame endpoint. To get a frame you either use preview mode and read /tmp/recording/preview/, walk the landmarks API for chips, or SSH in and read /tmp/recording/pics/ directly.

full data flow

IMX378 → MIPI CSI-2 → VPU ISP → DepthAI pipeline
                                      │
                              depthai_gate (:11492)
                              writes /tmp/recording/pics/
                                      │
                  ┌───────────────────┴───────────────────┐
                  ▼                                       ▼
        /tmp/recording/pics/                     map-ai (VPU NCE)
        (raw, purged >400MB)                     - sign classifier
                                                 - privacy blur
                                                       │
                  ┌────────────────────────────────────┤
                  ▼                                    ▼
    /data/recording/cached_observations/   odc-api.db (SQLite)
    (blurred, persistent)                  landmarks/observations/frames
                  │                                    │
                  └────────────────┬───────────────────┘
                                   ▼
                            odc-api (:5000)
                            /preview/*, /landmarks/*
                                   │
                                   ▼
                          hivemapper-data-logger
                          → FrameKm bundles in /data/recording/framekm/
                                   │
                                   ▼
                  odc-api → mitmdump :8888 → Cloudflare Workers → HERE OLP

Single detection trace:

1. IMX378 → MIPI → VPU ISP → depthai_gate
2. /tmp/recording/pics/1709920000123_0001_0042.jpg
3. map-ai picks it up, runs road sign classifier on the NCE
4. hit (e.g. speed limit 35), faces/plates blurred
5. observations row written + image saved to cached_observations/
6. landmarks row created/updated (class_label, lat, lon, confidence)
7. /landmarks/last/5 surfaces it; /landmarks/{id}/chips/{chip_id} returns the crop

replacement notes

getting a frame without odc-api

Direct file read (simplest, race-prone, no metadata):

ssh -p 2222 root@localhost   # via Lucy tunnel
ls -lt /tmp/recording/pics/ | head -10

# poor man's stream
while true; do
  cp $(ls -t /tmp/recording/pics/*.jpg | head -1) /tmp/current.jpg
  sleep 0.033
done

Redis pub/sub would be cleaner — r.pubsub().subscribe('frame_ready') — but I haven't confirmed depthai_gate publishes anything like that. Worth a redis-cli MONITOR while recording is live.

getting a frame without depthai_gate

Don't, unless you really mean it. You'd be replacing the whole VPU pipeline:

import depthai as dai

pipeline = dai.Pipeline()
cam = pipeline.create(dai.node.ColorCamera)
cam.setResolution(dai.ColorCameraProperties.SensorResolution.THE_4_K)
cam.setIspScale(1, 2)              # → 2028×1024

xout = pipeline.create(dai.node.XLinkOut)
xout.setStreamName("video")
cam.video.link(xout.input)

with dai.Device(pipeline) as device:
    q = device.getOutputQueue("video")
    while True:
        cv2.imwrite("/tmp/frame.jpg", q.get().getCvFrame())

Breaks everything Hivemapper depends on — map-ai, landmarks, FrameKm. Only useful if the goal is full liberation, not augmentation.

fastest frame grabs

With the stack running:

# direct
ssh -p 2222 root@localhost 'ls -t /tmp/recording/pics/*.jpg | head -1 | xargs cat' > frame.jpg

# via API (needs preview mode)
curl http://192.168.0.10:5000/api/1/preview/start
sleep 2
ssh -p 2222 root@localhost 'ls -t /tmp/recording/preview/*.jpg | head -1 | xargs cat' > frame.jpg
curl http://192.168.0.10:5000/api/1/preview/stop

proposed odc-api extension

Two new routes — single frame + MJPEG stream:

router.get('/frame', async (req, res) => {
  const frames = readdirSync('/tmp/recording/pics')
    .filter(f => f.endsWith('.jpg'))
    .sort().reverse();
  if (!frames.length) return res.status(404).send('No frames available');
  res.sendFile(join('/tmp/recording/pics', frames[0]));
});

router.get('/stream', async (req, res) => {
  res.writeHead(200, {
    'Content-Type': 'multipart/x-mixed-replace; boundary=frame',
    'Cache-Control': 'no-cache',
  });
  const interval = setInterval(() => {
    const frames = readdirSync('/tmp/recording/pics')
      .filter(f => f.endsWith('.jpg')).sort().reverse();
    if (frames.length) {
      const data = readFileSync(join('/tmp/recording/pics', frames[0]));
      res.write('--frame\r\n');
      res.write('Content-Type: image/jpeg\r\n');
      res.write(`Content-Length: ${data.length}\r\n\r\n`);
      res.write(data);
      res.write('\r\n');
    }
  }, 33);   // ~30 FPS
  req.on('close', () => clearInterval(interval));
});

Long-term replacement shape — leave depthai_gate alone, add a separate watcher service that inotify-tails /tmp/recording/pics/ and serves frames over HTTP. Doesn't fight the VPU, doesn't break the upload chain.

things still to confirm

• exact depthai_gate pipeline config (find files under /opt/)
• does depthai_gate publish frame events to Redis at all? (redis-cli MONITOR)
• camera-bridge vs depthai_gate — what's the actual dependency? (systemd deps, strace)
• preview config format — read getPreviewConfig()
• ML model exact locations (find /opt /data -name '*.blob' -o -name '*.onnx')
• frame timestamp accuracy vs GNSS time

appendix — file paths

/tmp/recording/pics/                       live frames
/tmp/recording/preview/                    preview frames
/data/recording/cached_observations/       landmark images
/data/recording/framekm/                   upload bundles
/data/recording/odc-api.db                 SQLite DB
/opt/camera-bridge/config.json             camera config
/opt/depthai_gate/                         DepthAI service (estimated)
/opt/odc-api/                              Node API service
/sys/class/vpu/                            VPU sysfs

appendix — boot order

multi-user.target
  ├── redis.service                    t+2s
  ├── depthai_gate.service              t+8s    loads luxonis_vpu.bin
  ├── map-ai.service                    t+12s   needs depthai_gate
  ├── hivemapper-data-logger.service    t+15s
  └── odc-api.service                   t+18s

appendix — ports

22      sshd          TCP       AP-only (socket)
5000    odc-api       HTTP      AP iface
6379    redis         TCP       localhost
8888    mitmdump      HTTP      localhost
11492   depthai_gate  HTTP/Flask localhost