Sulkta-Coop/adacam
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
adacam/docs/research/adacam-recovery-plan-2026-03-30.md

8.9 KiB
Raw Blame History

adacam Recovery Plan — 2026-03-30

Device: Hivemapper Bee (fvhL2I-iCT) | Hostname: keembay Status: Camera in truck, hostapd broken, Pi at home

1. Access Recovery

Physical options — pick one:

  • A (recommended): Go to truck with a laptop + USB-ETH adapter
  • B: Bring camera home so Pi can reach it
  • C: Bring Pi to truck

Primary: Ethernet Tether

Hardware needed:

  • USB-A to Gigabit Ethernet (RJ45) adapter — look for ASIX AX88179 or Realtek RTL8153 chipset (Plugable, Anker, Cable Matters). These are the chipsets most likely present in a 5.4.x Yocto kernel.
    • ⚠️ UNCERTAIN: no guarantee these modules are compiled into this Yocto build. The existing br0 bridge at 192.168.197.55/28 suggests USB networking was anticipated — encouraging but not confirmed.
  • Standard ethernet cable
  • Router/switch with DHCP (your home router if camera is home, or laptop internet sharing)

How it works:

  1. Plug USB-ETH adapter into camera USB port
  2. Connect ethernet cable to router/switch
  3. Check router DHCP table for new lease — look for hostname keembay or unknown MAC
  4. SSH to that IP as root from any machine on the same network
  5. If pam_nologin blocks: wait 2-3 min after power-on. If it persists longer, system is stuck in failed boot state — triage will show why.

Fallback: Pi + AP

Only viable if Pi is physically near the camera. AP is broadcasting but rejecting associations — this may be the wifi-client.service issue (see Fix #1). Static IP profile dashcam-hivemapper (192.168.0.11/24) is already on the Pi and bypasses DHCP. But if hostapd is broken at layer 2, association will still fail regardless of IP method.

Bottom line: ethernet tether is your real entry point.

2. Triage — Read ONLY, no changes

Run these in order once SSH is in:

# 1. Is system fully booted?
systemctl is-system-running
systemctl list-units --failed
journalctl -b --no-pager -p err

# 2. Is pam_nologin file still present?
ls -la /etc/nologin /run/nologin 2>/dev/null

# 3. Why is hostapd rejecting connections?
systemctl status hostapd --no-pager -l
journalctl -u hostapd -b --no-pager -n 100
cat /etc/hostapd/hostapd.conf

# 4. Is wifi-client.service still there?
systemctl status wifi-client.service --no-pager
cat /etc/systemd/system/wifi-client.service 2>/dev/null
systemctl is-enabled wifi-client.service 2>/dev/null

# 5. Network interface state
ip addr show
ip link show
ip route show

# 6. Is wpa_supplicant running on the WRONG interface?
ps aux | grep -E 'wpa_supplicant|hostapd'

# 7. Was resolv.conf reverted?
ls -la /etc/resolv.conf
cat /etc/resolv.conf

# 8. Persistence script — baseline read
cat /data/persist/install.sh

# 9. Disk space
df -h /

# 10. Compare boot logs (previous vs current)
journalctl -b -1 --no-pager -p warning -n 50
journalctl -b --no-pager -p warning -n 50

3. Recovery Steps (in order)

Fix #1 — wifi-client.service (LIKELY ROOT CAUSE)

What: hostapd rejecting WiFi associations at layer 2 (SSID visible but connection fails)

Why: wifi-client.service was created today and may still be enabled. If it's running wpa_supplicant or ip commands against wlp1s0f0 (the AP interface), it takes the interface away from hostapd. hostapd appears running but isn't actually processing auth. This matches the symptom exactly.

Confirm first: Triage steps 4 and 6 — is wifi-client.service enabled, and is wpa_supplicant on wlp1s0f0?

Fix:

  • Stop and disable wifi-client.service
  • Remove /etc/systemd/system/wifi-client.service
  • systemctl daemon-reload
  • Restart hostapd
  • Restart dnsmasq

Persist? No — you're removing the file, removal is permanent.

Fix #2 — hostapd config integrity

What: Verify hostapd.conf wasn't altered

Why: Unknown — precautionary

Fix: Check key fields from triage step 3:

  • interface=wlp1s0f0 ✓
  • ssid= correct ✓
  • wpa_passphrase= correct ✓
  • channel= and country_code= present

If all match expected values and Fix #1 resolved the interface conflict, no config changes needed.

Persist? No.

Fix #3 — /etc/resolv.conf

What: May be a plain file with 8.8.8.8 instead of the systemd-resolved symlink

Confirm: Triage step 7 — is it a symlink or a plain file?

Fix (only if plain file):

  • Delete the plain file
  • Create symlink to /run/systemd/resolve/stub-resolv.conf
    • ⚠️ Verify the exact target first: check /run/systemd/resolve/ for what files exist
  • Restart systemd-resolved

Persist? Optionally add a check to /data/persist/install.sh to recreate the symlink if OTA ever resets it.

Fix #4 — pam_nologin (if still blocking after Fix #1)

What: /etc/nologin or /run/nologin blocking SSH

Why: System stuck before reaching multi-user.target — likely caused by wifi-client.service failing hard

Fix:

  • Fix #1 should resolve this on next clean boot
  • If /etc/nologin exists (persistent, not tmpfs): remove it manually
  • /run/nologin is tmpfs and clears on reboot automatically

Persist? No.

Fix #5 — WiFi routing conflict (pre-existing, do after AP is working)

What: wlp1s0f0 (AP) and wlp1s0f1 (client) both on 192.168.0.0/24 — camera has no internet via wlp1s0f1

Fix (4 commands, must be added to /data/persist/install.sh to survive reboot):

ip route add 192.168.0.1/32 dev wlp1s0f1 2>/dev/null
ip rule add from 192.168.0.155 lookup 100 priority 100 2>/dev/null
ip route add default via 192.168.0.1 dev wlp1s0f1 table 100 2>/dev/null
ip route add 192.168.0.0/24 dev wlp1s0f1 table 100 2>/dev/null

Persist? YES — add all 4 to /data/persist/install.sh

Fix #6 — /etc/fstab duplicate (low priority)

What: Duplicate /factory entry causing systemd warnings

Fix: Remove one of the two identical lines for /dev/mmcblk1p10 in /etc/fstab

Persist? No — fstab is on root partition.

Fix #7 — Root partition cleanup (low priority)

What: / at 86%, 255MB free

Fix:

  • Check du -sh /var/log/*
  • Truncate or delete /var/log/daeman.log (was 5.1MB)
  • Configure logrotate or rsyslog size limits

4. Pi Cleanup

After camera is stable:

  1. Re-enable zerocool autoconnect: sudo nmcli connection modify zerocool connection.autoconnect yes
  2. Remove duplicate dashcam profile if it was auto-created: sudo nmcli connection delete dashcam-4A928016A02C1046 (keep dashcam-hivemapper)
  3. Restore dashcam-hivemapper to DHCP (remove static IP): sudo nmcli connection modify 'dashcam-hivemapper' ipv4.method auto ipv4.addresses "" ipv4.gateway ""
  4. Verify home WiFi reconnects: sudo nmcli connection up zerocool

5. Verification

  1. Phone connects to dashcam AP (dashcam-4A928016A02C1046) without error ✓
  2. ssh root@192.168.0.10 works from Pi ✓
  3. ping -I wlp1s0f1 -c 3 8.8.8.8 succeeds from camera ✓
  4. curl -s http://localhost:5500/api/1/health returns {"status": "ok"} ✓
  5. curl -s http://localhost:5500/api/1/forwarder/status shows forwarding active ✓
  6. redis-cli get MAP_AI_READY returns True ✓
  7. Power cycle → verify AP comes back up and SSH works without intervention ✓

6. If SSH is Completely Unrecoverable

Options in order of difficulty:

  1. UART serial console — Intel Keem Bay dev boards expose UART on a header or pads. Need to research the exact PCB layout for the Hivemapper Bee specifically. A CP2102 or CH340 USB-UART adapter (~$6 on Amazon) would work once pads are found. This gives full boot console access regardless of SSH/PAM state.

  2. eMMC flash via USB boot — Keem Bay supports USB boot mode. Would require finding the USB boot pin/switch on the PCB and flashing a recovery image. Risky without the original firmware image.

  3. RAUC recovery partition — The device has an A/B partition scheme. If there's a known-good partition B, RAUC could roll back — but Mender/RAUC is masked, so this path is unclear.

Realistic path if all else fails: UART. Research Hivemapper Bee board layout specifically (not generic Keem Bay devkit). The Bee teardown community may have UART pad locations documented.

Bee #2 — Locked Out (SSH Key Auth Only, Key Never Deployed)

Separate device. Separate problem. Camera hardware is fine.

What happened: SSH was configured for key-auth only. The authorized key was never added to the device. Service crashed before setup completed. Now completely locked out — password auth is disabled.

Recovery options:

  1. Ethernet tether — same as above. Get in via ethernet. Once in, add the SSH key to ~/.ssh/authorized_keys and verify.
  2. If ethernet also fails: UART console (same path as above — physical access, serial adapter)
  3. If password auth was never explicitly disabled: try the default Hivemapper root password (unknown — would need research)

This is fixable once you have console access. The camera isn't bricked — it's locked.

Plan generated 2026-03-30. Do not execute anything without Cobb's go-ahead.