varroa/docs/BEE-CAMERA.md

# Bee camera system

Notes from poking at the Hivemapper Bee dashcam. The camera path doesn't use V4L2 at all — frames live in `/tmp/recording/pics/` and `/data/recording/cached_observations/`, written by a DepthAI pipeline running on the on-die Myriad X VPU. All access goes through XLink, not `/dev/video*`.

## hardware

SoC is Intel Keem Bay (RVC2 — Robotics Vision Core 2): 4× Cortex-A53 @ 1.5GHz, integrated Myriad X VPU with 16 SHAVE cores and a Neural Compute Engine, 10nm. 4GB LPDDR4 on the board, ~3.5GB usable. CMA reserves ~1.34GB for VPU/camera DMA, swap is 2GB.

```
MemTotal:    3,584,000 kB
SwapTotal:   2,097,148 kB
CmaTotal:    1,408,000 kB   # VPU/camera DMA reservation
```

The camera is a Luxonis OAK-1-compatible module — Sony IMX378 (or equivalent 12MP), 4056×3040 native, downscaled to 2028×1024 by the pipeline. ~30 FPS. MIPI CSI-2 into the VPU's ISP; the ARM side never touches it directly.

Bus layout:

```
┌─────────────────────────────────────────────────────┐
│                  Intel Keem Bay SoC                 │
│  ┌──────────┐  ┌──────────┐  ┌────────────────┐    │
│  │ A53 ×4   │  │ Myriad X │  │ Neural Compute │    │
│  └────┬─────┘  └────┬─────┘  └────────┬───────┘    │
│       └────────┬────┴─────────────────┘            │
│           Internal AXI/NoC                          │
│     ┌─────┬─────┼──────┬───────────┐                │
│   PCIe   USB  SDIO  MIPI CSI       │                │
└─────┼─────┼─────┼──────┼───────────┘                │
      │     │     │      │
 Marvell  Telit  eMMC  IMX378
 88W8997  LE910C4
 WiFi/BT  LTE
```

## kernel + V4L2

Custom Yocto build with Intel VPU drivers. `kmb_cam` / `kmb_imx412` may be loaded for the sensor itself, plus the usual `videodev` + `v4l2_fwnode`. None of it is reachable via `/dev/video*` during normal operation — the VPU owns the camera hardware exclusively and the host talks to it over XLink (PCIe transport on Keem Bay; USB on desktop OAK devices).

VPU is controlled via sysfs at `/sys/class/vpu/`. Firmware is loaded by writing the filename to the `fwname` attribute. Two firmwares are present:

- `luxonis_vpu.bin` — DepthAI firmware (what we want)
- `vpu_nvr_b0.bin` — Intel HDDL firmware (conflicts, see below)

If you want frames without going through the existing stack you have three options: use the depthai_gate / odc-api stack as-is, stop depthai_gate and run your own DepthAI pipeline, or reverse-engineer XLink and roll custom VPU firmware. The first is by far the easiest.

## depthai_gate

Python+Flask, listens on localhost:11492, ~158 threads, ~200MB RSS. Lives at `/opt/depthai_gate/` (estimated — not confirmed on-device yet). Service unit looks like:

```ini
[Unit]
Description=DepthAI Camera Gate
After=network.target

[Service]
Type=simple
User=root
ExecStart=/opt/depthai_gate/run.py
Restart=always
```

What it does: loads `luxonis_vpu.bin` into the VPU, opens the XLink connection, configures the DepthAI pipeline (ColorCamera → ImageManip → XLinkOut, plus optional NeuralNetwork node), captures frames, and drops them into `/tmp/recording/pics/`. Pipeline config probably lives at `/opt/depthai_gate/pipeline.json` or `/data/camera_config.json`, possibly hardcoded.

XLink status values seen in logs:

```
xlink_device_status=2   # connected, healthy
                  =1   # connecting / error
                  =0   # disconnected
```

### the VPU conflict

`deviceservice.service` (Intel HDDL / OpenVINO) ships enabled and races depthai_gate for the VPU:

1. HDDL starts at boot, loads `vpu_nvr_b0.bin`
2. depthai_gate starts, overwrites with `luxonis_vpu.bin`
3. HDDL retries XLink every 2s forever, can't talk to the now-Luxonis firmware
4. If depthai_gate restarts after HDDL is already running, HDDL grabs the VPU first and the camera goes dead
5. `secure-wdtclient` watchdog crash-loops on the dead VPU → memory pressure → OOM

Fix:

```bash
systemctl disable --now deviceservice
systemctl mask deviceservice   # survives OTA
```

## map-ai

Reads frames from depthai_gate, runs detection on the VPU's NCE, blurs faces and plates, writes results to SQLite (`/data/recording/odc-api.db`) and blurred frames to disk.

```ini
[Unit]
Description=Map AI Processing
After=depthai_gate.service

[Service]
Type=simple
User=root
ExecStart=/opt/map-ai/run.py
Restart=always
```

Pipeline:

```
frame from depthai_gate
        │
        ▼
  ML inference (on VPU NCE)
  - road sign classifier
  - face detector
  - license plate detector
        │
        ▼
  privacy blur
  - Gaussian on faces/plates
  - cv2.imwrite blurred copy
        │
        ├──► Redis  (status keys, not detections)
        └──► SQLite (observations, landmarks, frames)
```

Models:

| Model | Path | Purpose |
|-------|------|---------|
| Road signs | `/opt/object-detection/model.blob` or `/data/models/` | classification |
| Privacy | `/opt/odc-api/python/` or `/data/models/` | face/plate detection |
| PVC | `/data/recording/models/pvc.onnx` | unknown — 227 bytes, probably an index file |

Privacy model hash gets baked into FrameKm metadata for verification.

Redis only carries readiness flags:

```
GET MAP_AI_READY                     → "True"
GET EXTERNAL_MODEL_CLASSIFIER_READY  → "True"
```

Detections go to SQLite, not Redis ZSETs.

## frame storage

| Path | FS | Purpose | Persists? |
|------|----|---------|-----------|
| `/tmp/recording/pics/` | tmpfs | live frames | no |
| `/tmp/recording/preview/` | tmpfs | preview mode | no |
| `/data/recording/cached_observations/` | ext4 | landmark observations | yes |
| `/data/recording/framekm/` | ext4 | upload bundles | yes |
| `/tmp/rgb/` | tmpfs | frame list files | no |

Frames are JPEG at 2028×1024, ~85% quality, ~150-200KB each.

Naming:

```
# live
/tmp/recording/pics/{system_time_ms}_{frame_id}_{sequence}.jpg
e.g. 1709920000123_0001_0042.jpg

# cached observations
/data/recording/cached_observations/{timestamp}_{subsecond}_{frame_number}.jpg
e.g. 1746377552_043000_2945056.jpg
```

`folder_purger` keeps disk under control — when `/tmp/recording/pics/` crosses 400MB, oldest frames go:

```
folder-purger /tmp/recording/pic 400000000 /mnt/data/gps 2000000000 ...
```

SQLite schemas (simplified):

```sql
CREATE TABLE frames (
    system_time INTEGER PRIMARY KEY,
    image_name TEXT
);

CREATE TABLE observations (
    id INTEGER PRIMARY KEY,
    landmark_id INTEGER,
    image_name TEXT,
    x1 REAL, y1 REAL, x2 REAL, y2 REAL,
    ts INTEGER
);
```

DB lives at `/data/recording/odc-api.db` (also seen as `data-logger.v2.0.0.db`).

## video-processor + FrameKm

`video-processor` isn't documented in the firmware I've looked at. Based on naming it bundles frames+metadata into FrameKm tarballs, handles any H.264/H.265 encoding for preview, and orders frames for upload. It doesn't produce raw frames — it only packages already-blurred ones — so it's not useful for camera-access work.

A FrameKm is ~1km of driving data:

```
framekm-2024-03-08-12-34-56-abc123.tar
├── manifest.json
├── frame_0001.jpg
├── frame_0002.jpg
├── ...
├── gnss_auth_buffer.bin
└── gnss_auth_signature.bin
```

Manifest:

```json
{
  "name": "framekm-2024-03-08-12-34-56-abc123",
  "numFrames": 150,
  "deviceId": "fvhL2I-iCT",
  "firmwareVersion": "0.0.1",
  "privacyModelHash": "sha256:abc123...",
  "gnssAuthBuffer": "base64...",
  "gnssAuthSignature": "base64...",
  "gnssAuthPublicKey": "base64...",
  "createdAt": 1709920000000
}
```

## odc-api

REST API at `http://192.168.0.10:5000/api/1/`. Binds to the AP interface (`wlp1s0f0`) only — not reachable from home LAN without going through the Bee's AP.

Preview endpoints:

| Endpoint | Method | Notes |
|----------|--------|-------|
| `/preview/start` | GET | 120s timeout, then auto-stop |
| `/preview/stop` | GET | |
| `/preview/status` | GET | |
| `/preview/metadata` | GET | latest frame metadata |

Preview works by writing a new config and bouncing camera-bridge:

```typescript
export const startPreview = async () => {
  await execSync('mkdir /tmp/recording/preview');
  writeFileSync(IMAGER_CONFIG_PATH, JSON.stringify(getPreviewConfig()));
  await execSync(CMD.STOP_CAMERA);
  await sleep(1000);
  await execSync(CMD.START_CAMERA);
};
```

The 120s auto-stop is there to protect 4K recording quality.

Landmark endpoints (where the cached observation images come out):

| Endpoint | Method | Notes |
|----------|--------|-------|
| `/landmarks/images/:id` | GET | image paths for a landmark |
| `/landmarks/:id/chips` | GET | list of chip endpoints |
| `/landmarks/:id/chips/:chip_id` | GET | cropped observation JPEG |
| `/landmarks/boundingBox/:id` | GET | bbox coords |
| `/landmarks/upload` | PUT | upload landmark image |

Image retrieval flow:

```
GET /landmarks/images/123
  → ["/data/recording/cached_observations/1746377552_043000_2945056.jpg"]

GET /landmarks/123/chips/456
  → cropped JPEG of the bbox region
```

Camera bridge config: `/opt/camera-bridge/config.json`. Control commands from `bee.ts`:

```typescript
export const CMD = {
  RESTART_CAMERA: 'systemctl restart camera-bridge',
  START_CAMERA:   'systemctl start   camera-bridge',
  STOP_CAMERA:    'systemctl stop    camera-bridge',
  START_PREVIEW:  'systemctl start   camera-preview',
  STOP_PREVIEW:   'systemctl stop    camera-preview',
};
```

There is no `/camera/frame` endpoint. To get a frame you either use preview mode and read `/tmp/recording/preview/`, walk the landmarks API for chips, or SSH in and read `/tmp/recording/pics/` directly.

## full data flow

```
IMX378 → MIPI CSI-2 → VPU ISP → DepthAI pipeline
                                      │
                              depthai_gate (:11492)
                              writes /tmp/recording/pics/
                                      │
                  ┌───────────────────┴───────────────────┐
                  ▼                                       ▼
        /tmp/recording/pics/                     map-ai (VPU NCE)
        (raw, purged >400MB)                     - sign classifier
                                                 - privacy blur
                                                       │
                  ┌────────────────────────────────────┤
                  ▼                                    ▼
    /data/recording/cached_observations/   odc-api.db (SQLite)
    (blurred, persistent)                  landmarks/observations/frames
                  │                                    │
                  └────────────────┬───────────────────┘
                                   ▼
                            odc-api (:5000)
                            /preview/*, /landmarks/*
                                   │
                                   ▼
                          hivemapper-data-logger
                          → FrameKm bundles in /data/recording/framekm/
                                   │
                                   ▼
                  odc-api → mitmdump :8888 → Cloudflare Workers → HERE OLP
```

Single detection trace:

```
1. IMX378 → MIPI → VPU ISP → depthai_gate
2. /tmp/recording/pics/1709920000123_0001_0042.jpg
3. map-ai picks it up, runs road sign classifier on the NCE
4. hit (e.g. speed limit 35), faces/plates blurred
5. observations row written + image saved to cached_observations/
6. landmarks row created/updated (class_label, lat, lon, confidence)
7. /landmarks/last/5 surfaces it; /landmarks/{id}/chips/{chip_id} returns the crop
```

## replacement notes

### getting a frame without odc-api

Direct file read (simplest, race-prone, no metadata):

```bash
ssh -p 2222 root@localhost   # via Lucy tunnel
ls -lt /tmp/recording/pics/ | head -10

# poor man's stream
while true; do
  cp $(ls -t /tmp/recording/pics/*.jpg | head -1) /tmp/current.jpg
  sleep 0.033
done
```

Redis pub/sub would be cleaner — `r.pubsub().subscribe('frame_ready')` — but I haven't confirmed depthai_gate publishes anything like that. Worth a `redis-cli MONITOR` while recording is live.

### getting a frame without depthai_gate

Don't, unless you really mean it. You'd be replacing the whole VPU pipeline:

```python
import depthai as dai

pipeline = dai.Pipeline()
cam = pipeline.create(dai.node.ColorCamera)
cam.setResolution(dai.ColorCameraProperties.SensorResolution.THE_4_K)
cam.setIspScale(1, 2)              # → 2028×1024

xout = pipeline.create(dai.node.XLinkOut)
xout.setStreamName("video")
cam.video.link(xout.input)

with dai.Device(pipeline) as device:
    q = device.getOutputQueue("video")
    while True:
        cv2.imwrite("/tmp/frame.jpg", q.get().getCvFrame())
```

Breaks everything Hivemapper depends on — map-ai, landmarks, FrameKm. Only useful if the goal is full liberation, not augmentation.

### fastest frame grabs

With the stack running:

```bash
# direct
ssh -p 2222 root@localhost 'ls -t /tmp/recording/pics/*.jpg | head -1 | xargs cat' > frame.jpg

# via API (needs preview mode)
curl http://192.168.0.10:5000/api/1/preview/start
sleep 2
ssh -p 2222 root@localhost 'ls -t /tmp/recording/preview/*.jpg | head -1 | xargs cat' > frame.jpg
curl http://192.168.0.10:5000/api/1/preview/stop
```

### proposed odc-api extension

Two new routes — single frame + MJPEG stream:

```typescript
router.get('/frame', async (req, res) => {
  const frames = readdirSync('/tmp/recording/pics')
    .filter(f => f.endsWith('.jpg'))
    .sort().reverse();
  if (!frames.length) return res.status(404).send('No frames available');
  res.sendFile(join('/tmp/recording/pics', frames[0]));
});

router.get('/stream', async (req, res) => {
  res.writeHead(200, {
    'Content-Type': 'multipart/x-mixed-replace; boundary=frame',
    'Cache-Control': 'no-cache',
  });
  const interval = setInterval(() => {
    const frames = readdirSync('/tmp/recording/pics')
      .filter(f => f.endsWith('.jpg')).sort().reverse();
    if (frames.length) {
      const data = readFileSync(join('/tmp/recording/pics', frames[0]));
      res.write('--frame\r\n');
      res.write('Content-Type: image/jpeg\r\n');
      res.write(`Content-Length: ${data.length}\r\n\r\n`);
      res.write(data);
      res.write('\r\n');
    }
  }, 33);   // ~30 FPS
  req.on('close', () => clearInterval(interval));
});
```

Long-term replacement shape — leave depthai_gate alone, add a separate watcher service that inotify-tails `/tmp/recording/pics/` and serves frames over HTTP. Doesn't fight the VPU, doesn't break the upload chain.

## things still to confirm

- exact depthai_gate pipeline config (find files under `/opt/`)
- does depthai_gate publish frame events to Redis at all? (`redis-cli MONITOR`)
- camera-bridge vs depthai_gate — what's the actual dependency? (systemd deps, strace)
- preview config format — read `getPreviewConfig()`
- ML model exact locations (`find /opt /data -name '*.blob' -o -name '*.onnx'`)
- frame timestamp accuracy vs GNSS time

## appendix — file paths

```
/tmp/recording/pics/                       live frames
/tmp/recording/preview/                    preview frames
/data/recording/cached_observations/       landmark images
/data/recording/framekm/                   upload bundles
/data/recording/odc-api.db                 SQLite DB
/opt/camera-bridge/config.json             camera config
/opt/depthai_gate/                         DepthAI service (estimated)
/opt/odc-api/                              Node API service
/sys/class/vpu/                            VPU sysfs
```

## appendix — boot order

```
multi-user.target
  ├── redis.service                    t+2s
  ├── depthai_gate.service              t+8s    loads luxonis_vpu.bin
  ├── map-ai.service                    t+12s   needs depthai_gate
  ├── hivemapper-data-logger.service    t+15s
  └── odc-api.service                   t+18s
```

## appendix — ports

```
22      sshd          TCP       AP-only (socket)
5000    odc-api       HTTP      AP iface
6379    redis         TCP       localhost
8888    mitmdump      HTTP      localhost
11492   depthai_gate  HTTP/Flask localhost
```