Sulkta-Coop/adacam
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
adacam/docs/BRICK-BEE-RECOVERY-RESEARCH.md

7.5 KiB
Raw Permalink Blame History

Brick Bee Recovery Research

Date: 2026-03-22
Status: ONGOING
Problem: Unit 2 Bee SSH locked out, USB Mender flashes not working

The Problem

What Happened

  1. Liberation script v0.3 ran on Unit 2
  2. Script wrote PasswordAuthentication no to /data/overlay/current/ssh/sshd_config
  3. Script failed BEFORE writing SSH authorized_keys
  4. Result: SSH requires key auth, but no keys authorized → locked out

Why Mender Can't Fix It

  • Mender only swaps rootfs A/B partitions
  • /data partition is NEVER touched
  • Bad sshd_config lives in /data/overlay/current/ssh/sshd_config
  • Overlay survives ALL firmware flashes
  • Even successful flash → bad overlay still applied → still locked out

USB Updater Behavior

What We Observe

  • USB activity when plugging in USB drive with artifacts
  • Device boots, AP comes up (dashcam-81B2B81681545109)
  • But nothing "lands and sticks" — no change in behavior

How usb-updater Works

  1. Looks for /mnt/usb/hivemapper_update/*.mender
  2. Extracts syshash.img from artifact
  3. Compares against inactive partition's hash
  4. If different → runs mender --install <file>
  5. If same → exits (thinks "already up to date")

Diagnostic Findings (from Truck Bee - same firmware)

Mender Config (/etc/mender/mender.conf)

{
    "InventoryPollIntervalSeconds": 28800,
    "RetryPollIntervalSeconds": 300,
    "ServerURL": "https://docker.mender.io",
    "TenantToken": "dummy",
    "UpdatePollIntervalSeconds": 1800
}

NO ArtifactVerifyKey — signature verification is NOT enforced.

Device Type

  • Original Hivemapper firmware: device_type: ["keembay"]
  • Our recovery artifacts: device_type: ["keembay"]
  • Match confirmed — not the issue

Artifact Header (our ssh-recovery.mender)

{
  "payloads": [{"type": "dm-verity-update"}],
  "artifact_provides": {"artifact_name": "ssh-recovery-v1"},
  "artifact_depends": {"device_type": ["keembay"]}
}

Theories for Why Flash Isn't Working

Theory 1: Hash Comparison Exit (LIKELY)

If artifact's syshash matches inactive partition, usb-updater exits without flashing. This would happen if:

  • We're using same/similar rootfs as what's already on device
  • The inactive partition already has similar content

Test: Need to verify syshash is actually different.

Theory 2: dm-verity Failure After Flash

If flash succeeds but syshash.img doesn't match the written system.img:

  1. Device boots new partition
  2. dm-verity fails hash check
  3. Device falls back to old partition
  4. Appears as "nothing changed"

Test: Can't verify without serial console.

Theory 3: Mender Install Fails Silently

mender --install might be failing due to:

  • Corrupt artifact format
  • Wrong Mender client version
  • dm-verity module issues

Test: Can't verify without logs from bricked device.

Theory 4: Flash Works But We Don't Notice

Flash might succeed but:

  • Device boots new partition with same overlay
  • SSH still broken
  • We think flash failed

Test: Would need to check mender_boot_part - can't without shell.

Solution Approaches

Approach A: ArtifactInstall_Leave State Script

Add a state script to the artifact that fixes the overlay BEFORE reboot:

#!/bin/sh
# ArtifactInstall_Leave — runs after rootfs written, before reboot
# Still on OLD system, has full /data access

# Remove bad sshd_config from overlay
rm -f /data/overlay/current/ssh/sshd_config

# Write authorized_keys to overlay
mkdir -p /data/overlay/current/root/.ssh
chmod 700 /data/overlay/current/root/.ssh
cat > /data/overlay/current/root/.ssh/authorized_keys << 'EOF'
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIK87jxvlXvo60pxwdtyJsXeFsb4KsAiFx4FnyXz81kh7 cobb@adacam
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOQxwJU91TCxds34P18D3xRbu7rxlrgTUoml/H8nxeDK kayos@openclaw
EOF
chmod 600 /data/overlay/current/root/.ssh/authorized_keys

exit 0

Problem: Hivemapper's Mender fork does NOT call state scripts (confirmed from previous testing).

Approach B: Patched usb-updater in Rootfs (v5+ approach)

Modify /usr/bin/usb-updater in system.img to run recovery code.

Status: v5-v8 tried this, no evidence of success.

Possible issues:

  1. Flash not happening (hash match or failure)
  2. Flash happening but dm-verity failing
  3. Flash happening but recovery code not executing

Approach C: First-Boot Recovery Service

Add systemd service to rootfs that runs once on first boot:

[Unit]
Description=Fix overlay on first boot
ConditionPathExists=/data/.overlay-recovery-needed

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/fix-overlay.sh

[Install]
WantedBy=multi-user.target

Create trigger file via patched usb-updater.

Problem: Requires flash to succeed first.

Approach D: UART Serial Console (Hardware)

Intel Keem Bay exposes UART at 0x20180000.

Steps:

  1. Open Bee enclosure (need screwdrivers)
  2. Find UART pads on PCB
  3. Connect USB-to-serial adapter
  4. Get bootloader/root shell
  5. Manually fix /data/overlay/current/ssh/sshd_config

Status: Cobb looking for screwdrivers.

Approach E: Contact Hivemapper

Request signed recovery artifact or factory reset procedure.

Status: Not attempted.

What We've Tried (v1-v8)

Version Approach Result
v1 State script only State scripts not called
v2 State script + fixed header State scripts not called
v3 State script + header fix State scripts not called
v4 State script + USB debug log Confirmed: state scripts never run
v5 Patched usb-updater No evidence of success
v6 Boot service + patched usb-updater No evidence of success
v7 Refinements No evidence of success
v8 Refinements No evidence of success

Current Blockers

  1. Can't verify if flash is happening — no shell access
  2. Can't check logs — no shell access
  3. Can't check which partition booted — no shell access
  4. State scripts don't work — Hivemapper Mender fork ignores them

Next Steps

  1. UART access — Cobb finding screwdrivers to open Bee
  2. Verify artifact syshash differs — compare against known partition hashes
  3. Try nuclear approach — artifact that just wipes entire /data/overlay
  4. Contact Hivemapper — request recovery assistance

eMMC Partition Table

Part Label Type Notes
p1 capsule - Firmware capsule
p2 env-main - U-Boot env (primary)
p3 env-redund - U-Boot env (backup)
p4 boot_a ext4 Kernel slot A
p5 system_a ext4 Rootfs slot A
p6 syshash_a dm-verity Hash for A
p7 boot_b ext4 Kernel slot B
p8 system_b ext4 Rootfs slot B (ACTIVE)
p9 syshash_b dm-verity Hash for B
p10 factory - Factory calibration
p11 data ext4 Persistent /data (52GB)

U-Boot env partitions (p2/p3): Contains mender_boot_part. UART access would let us modify this.

Key Files

  • Artifacts: /root/.openclaw/workspace/projects/adacam/recovery/
  • Recon data: /root/.openclaw/workspace/recon/
  • Deep recon: /root/.openclaw/workspace/recon/deep2/
  • Liberation scripts: /root/.openclaw/workspace/projects/adacam/
  • This doc: /root/.openclaw/workspace/docs/BRICK-BEE-RECOVERY-RESEARCH.md

Unit Reference

Unit SSID Status
Truck Bee (Unit 1) dashcam-4A928016A02C1046 Working, READ-ONLY
Brick Bee (Unit 2) dashcam-81B2B81681545109 SSH locked out

Last updated: 2026-03-22 14:03 PDT