adacam/PROJECT_STATUS.md

# AdaCam / Hivemapper Bee — Project Status
_Last updated: 2026-03-16_

---

## Goal
Convert Hivemapper Bee dashcams (HDC-S, Intel Keem Bay platform) into "AdaCam" units:
- Block Hivemapper telemetry / OTA
- Install adacam services (GPS forwarding, Wigle logging, etc.)
- Regain persistent root SSH access
- Long-term: custom firmware image, signed USB recovery, own update infrastructure

---

## Hardware

| Item | Value |
|------|-------|
| Platform | Intel Keem Bay (keembay) |
| SoC | Movidius MV0212, Yocto Linux |
| Storage | eMMC, 11 partitions |
| Update mechanism | Hivemapper USB updater (usb-updater.service) + custom Mender fork |
| SSH | dropbear or openssh on port 22 |
| AP interface | wlp1s0f0 (192.168.0.10 factory) |
| WiFi client | wlp1s0f1 |
| LTE | LE910C4-NF (wwan0) |

### Partition Layout

| Partition | Size | Purpose |
|-----------|------|---------|
| mmcblk1p1 | 32MB | ? |
| mmcblk1p2 | 512K | ? |
| mmcblk1p3 | 512K | ? |
| mmcblk1p4 | 256MB | Kernel A (ext4, FIT image) |
| mmcblk1p5 | 3GB | Rootfs A (ext4) |
| mmcblk1p6 | 128MB | dm-verity hashtree A |
| mmcblk1p7 | 256MB | Kernel B |
| mmcblk1p8 | 3GB | Rootfs B (ext4) |
| mmcblk1p9 | 128MB | dm-verity hashtree B |
| mmcblk1p10 | 64MB | /factory |
| mmcblk1p11 | ~52GB | /data (rw, persistent) |

### Filesystem Mounts (critical)

| Mount | Source | Type | Notes |
|-------|--------|------|-------|
| / | mmcblk1p5 or p8 | ext4 | **READ-ONLY** |
| /etc | overlayfs | overlay | lower=rootfs/etc, upper=/data/overlay/current/, workdir=/data/overlay/workdir/ |
| /data | mmcblk1p11 | ext4 | rw, **persists across ALL firmware flashes** |
| /home | /data/home | bind | persistent |
| /var | /data/var | bind | persistent |
| /uboot | mmcblk1p4 | ext4 | U-Boot env partition |

**KEY INSIGHT:** Only `/etc` is overlaid. `/usr`, `/bin`, `/usr/bin` etc. are directly on the read-only rootfs. The overlay upper dir lives on `/data` which survives firmware flashes.

---

## Firmware / Update System

### Mender Fork
- Version: `f9a29241` (custom Hivemapper fork, not standard Mender)
- **Does NOT call state scripts** (confirmed via v4 test — no USB log file written)
- Artifact format (must match exactly):
  ```json
  {"payloads":[{"type":"dm-verity-update"}],"artifact_provides":{"artifact_name":"NAME"},"artifact_depends":{"device_type":["keembay"]}}
  ```
- NO `rootfs-image.version` in provides (breaks install if included)
- NO state scripts
- NO meta-data file needed

### dm-verity-update Module
- Path: `/usr/share/mender/modules/v3/dm-verity-update`
- Streams (in order): system.img → passive_rootfs, syshash.img → passive_hash, boot.img → passive_kernel
- **Blindly cats streams** — does NOT recompute hash tree
- dm-verity is **NOT enforced at runtime** (`verity=0` in /proc/cmdline)
- syshash is only used by usb-updater for "already up to date" comparison check

### usb-updater Script
- Path: `/usr/bin/usb-updater`
- Triggered by `usb-updater.service` (enabled, not masked by liberate.sh)
- Flow:
  1. Create swapfile at /data/swap if not exists (2GB)
  2. Find .mender file in `/mnt/usb/hivemapper_update/`
  3. Extract outer tar to USB tmp dir
  4. Extract `syshash.img` from data/0000.tar.gz
  5. Compare USB syshash with active hash partition (mmcblk1p6 or p9)
  6. If same → "OS up to date", exit
  7. If different → `mender --install artifact.mender`
  8. If success → `mender --commit` → `reboot`
  9. If fail → "Update failed", exit
- USB dir cleaned up after syshash compare (before mender runs)
- `mender-client.service` being masked does NOT affect CLI `mender --install`

### A/B Slot Logic

| U-Boot var | Meaning |
|-----------|---------|
| mender_boot_part=5 | Slot A active |
| mender_boot_part=8 | Slot B active |
| upgrade_available=1 | Update pending commit |
| upgrade_available=0 | Committed / stable |

**Post-install rollback trap:** usb-updater calls `mender --commit` before reboot. After reboot, `ArtifactVerifyReboot` in the mender state machine checks `upgrade_available=1` — but commit already set it to 0 → exits 1 → mender triggers rollback procedure. However ArtifactRollback condition also fails (upgrade_available != 1) so boot_part doesn't change. Device ends up on new firmware but mender thinks it rolled back. Harmless in practice.

### dm-verity Parameters (for veritysetup)
```
Hash algorithm: sha384
Salt: 3f3e0633b0a2cbf5066cd50af1a178ff8f50f660382f6c7508f58566cec64142
Data blocks: 524288
Block size: 4096
```

---

## Device Status

### Unit 1 — Original Bee (LIBERATED ✅)
- IP: 192.168.0.155 (from Lucy's perspective, via SSH tunnel)
- SSH: works, root access
- Tunnel: bee-tunnel.service → Lucy (when powered on)
- Status: Fully liberated, adacam services running
- mender_boot_part: 8 (Slot B active)
- Artifact: Release-20260309193836

### Unit 2 — Replacement Bee (LOCKED OUT ⚠️)
- IP: 192.168.0.10 (factory AP)
- SSH: key-only, no authorized keys → Permission denied (publickey)
- Cause: liberate.sh (old version, before v0.4) ran Phase 4 SSH hardening (PasswordAuthentication no) then failed at `mkdir /root/.ssh` (read-only rootfs). Hardened sshd_config was written to /data/overlay/current/ssh/sshd_config before the failure.
- Overlay file `/data/overlay/current/ssh/sshd_config` has hardened config and survives every flash
- usb-updater.service: enabled, NOT masked
- mender-client.service: masked (by Phase 1 of liberate.sh — harmless for USB updates)

---

## Artifacts Built

| File | Size | Description | Result |
|------|------|-------------|--------|
| ssh-recovery.mender | small | State script only, no firmware | State scripts not called → failed |
| hivemapper-forced.mender | 431MB | Factory firmware, random syshash | Infinite USB loop (hash mismatch) |
| adacam-ssh-fix.mender | 443MB | Modified sshd_config in system.img, correct syshash, no state scripts | Overlay overrides system.img changes → still locked |
| adacam-ssh-fix-v2.mender | 444MB | State script + wrong header (rootfs-image.version) | mender --install likely failed |
| adacam-ssh-fix-v3.mender | 444MB | State script + correct header format | State scripts not called by this mender fork |
| adacam-ssh-fix-v4.mender | 444MB | State script + USB log debug | Confirmed: NO log file = state scripts never run |
| **adacam-ssh-fix-v5.mender** | **403MB** | **Modified usb-updater in system.img + correct syshash** | **PENDING TEST** |

All artifacts at: `/mnt/cache/appdata/openclaw/config/workspace/projects/adacam/recovery/`

### v5 Approach (current)
- Modified `/usr/bin/usb-updater` in system.img to prepend SSH recovery code
- `/usr/bin` is on rootfs, NOT in /etc overlay → change survives flash
- On next USB insert after reboot into v5 firmware, usb-updater:
  1. Writes open sshd_config to /etc/ssh/sshd_config (via overlay to /data/overlay/current/ssh/sshd_config)
  2. Writes authorized_keys to /home/root/.ssh/authorized_keys
  3. Restarts sshd
  4. Continues normal usb-updater behavior
- SHA256: `acfbd16db9620f23785f8b103ffaeff6aed780f383273a61a23c8002f2bf0980`
- Served via nginx on Lucy: `http://192.168.0.5:9999/adacam-ssh-fix-v5.mender`

---

## Build Infrastructure

### Files on Lucy
```
/mnt/cache/appdata/openclaw/config/workspace/
  projects/
    adacam/
      firmware/
        extracted/
          core.mender                          # original factory firmware
          mender/data_extracted/
            system.img                         # original 2GB rootfs
            syshash.img                        # original 33MB hash tree
            boot.img                           # original 256MB kernel (ext4 + FIT image)
      recovery/
        ssh-recovery.mender
        hivemapper-forced.mender
        adacam-ssh-fix.mender
        adacam-ssh-fix-v2.mender
        adacam-ssh-fix-v3.mender
        adacam-ssh-fix-v4.mender
        adacam-ssh-fix-v5.mender              # CURRENT
    build_v2.py                                # artifact builder (reuses existing data tar)
    build_v5.sh                                # system.img patcher + artifact builder
    patch-mender-artifact.py
    build-mender-artifact.py

/tmp/bee_v5_work/
  system.img                                   # patched (usb-updater prepended)
  syshash.img                                  # recomputed for patched system.img
  usb-updater-patched                          # the patched script (reference copy)
```

### Build Commands
```bash
# SSH to Lucy
ssh -i ~/.openclaw/id_ed25519_unraid -o StrictHostKeyChecking=no root@192.168.0.5

# Run Python via Docker (python3 not available on Lucy bare metal)
docker run --rm \
  -v /mnt/cache/appdata/openclaw/config/workspace:/workspace \
  -v /tmp/bee_v5_work:/v5work \
  mingc/android-build-box python3 /v5work/build.py

# Compute syshash
veritysetup format \
  --hash=sha384 \
  --salt=3f3e0633b0a2cbf5066cd50af1a178ff8f50f660382f6c7508f58566cec64142 \
  --data-block-size=4096 \
  --hash-block-size=4096 \
  system.img syshash.img

# Serve artifact
docker restart adacam-http   # nginx on Lucy:9999
```

### Keys (authorized_keys)
```
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIK87jxvlXvo60pxwdtyJsXeFsb4KsAiFx4FnyXz81kh7 cobb@adacam
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOQxwJU91TCxds34P18D3xRbu7rxlrgTUoml/H8nxeDK kayos@openclaw
```

---

## Lessons Learned

1. **Never harden SSH before confirming key auth works.** Write keys first, test, then restrict.
2. **The /etc overlay survives firmware flashes.** Any config change via /etc goes to /data and persists. Flashing new sshd_config in system.img does nothing if the overlay has a conflicting file.
3. **This Mender fork does NOT call state scripts.** Don't use state scripts with dm-verity-update type.
4. **rootfs-image.version in artifact_provides breaks mender --install.** Match the exact original header format.
5. **usb-updater cleans up its tmp dir AFTER the syshash comparison.** Seeing tmp on the USB mid-process is normal; it doesn't indicate failure.
6. **/usr/bin is NOT overlaid.** Changes to files in system.img outside of /etc are not overridden and take effect after flash.
7. **Build data tars to disk, not BytesIO.** Lucy has ~761MB free RAM — compressing 2GB in memory OOMs.

---

## Next Steps (when resuming)

### Immediate
1. Test `adacam-ssh-fix-v5.mender` on Unit 2
   - Flash via USB
   - After reboot: plug in any USB drive
   - Check if sshd_config overlay is cleared + SSH opens
   - If SSH opens: run liberate.sh v0.4

### If v5 fails
- UART access as last resort (physical header on PCB, unknown pinout — no public schematics found)
- Consider Hivemapper community forums / Discord for UART pinout info

### After SSH is restored (Unit 2)
- Run `liberate.sh v0.4` (safe — /data-only writes, usb-updater preserved, no SSH hardening)
- Verify adacam services install correctly

### Long-term (Path A — Custom Firmware Image)
- Mount clean system.img
- Bake in: adacam services, correct sshd_config, authorized_keys, modified usb-updater
- Generate Mender artifact signing keypair
- Bake `artifact_verify_key` into custom image
- Sign all future artifacts with our key (locks out Hivemapper OTA permanently)
- Retire liberate.sh

---

## Repository
- Gitea: `http://192.168.0.5:3001/Sulkta-Coop/adacam`
- liberate.sh current version: v0.4 (commit bddc1507)
- API token: `33a9eb57b58c262f4434c12028bc3a30b1ff7021`