adacam/docs/research/adacam-bug-report-2026-03-29.md

Adacam Bug Report — 2026-03-29

Device: Hivemapper Bee (keembay)
Device ID: dashcam-4A928016A02C1046
Diagnostic Time: 2026-03-30 05:15 UTC
System Uptime: Since ~Mar 28 15:24 UTC (approx 38 hours)

---

Critical Issues

1. WiFi Client Interface Has No Internet Connectivity

Severity: CRITICAL
Status: Broken — no outbound traffic from camera to internet

Symptoms:

• DNS resolution fails completely: api.adamaps.org: resolve call failed: All attempts to contact name servers or networks failed
• Ping to 8.8.8.8: 100% packet loss
• Ping to gateway (192.168.0.1): 100% packet loss
• Ping to other LAN hosts (192.168.0.5, 192.168.0.184): 100% packet loss

Network State:

• WiFi association: ✅ Connected to zerocool (b4:75:0e:5a:07:ff)
• wpa_state: COMPLETED
• IP assigned: 192.168.0.155/24 on wlp1s0f1
• Default route: via 192.168.0.1 dev wlp1s0f1 ✅
• ARP for gateway: 00:25:90:26:3c:29 (populated)

Root Cause: Unknown — WiFi appears connected at L2 but L3 traffic fails. Possibly:

• Router firewall blocking the camera's MAC
• IP conflict with 192.168.0.155 on another device
• wlp1s0f0 (AP interface at 192.168.0.10) causing routing confusion (same /24 on both interfaces)

Potential Fix:

1. Check router DHCP leases and firewall for 192.168.0.155
2. Consider removing the 192.168.0.0/24 route from wlp1s0f0 when WiFi client is active
3. Investigate if hostapd/dnsmasq on wlp1s0f0 conflicts with client connectivity

---

2. bee-tunnel.service Failed (14,000+ restarts)

Severity: CRITICAL (but blocked by #1)
Status: Service file removed, but still attempting restarts from systemd memory

Error: ssh: connect to host 192.168.0.5 port 22: No route to host

Details:

• Restart counter: 14,712+
• The service tries to create a reverse SSH tunnel to Lucy (192.168.0.5)
• Service file /etc/systemd/system/bee-tunnel.service no longer exists
• This is because adamaps-persist.service failed (see below), which should create this file

Root Cause: WiFi client connectivity broken → cannot reach Lucy → service fails → but service file wasn't recreated on boot

Fix (once WiFi works):

1. Re-run /data/persist/install.sh to recreate bee-tunnel.service
2. Or reboot to trigger adamaps-persist again

---

Warning Issues

3. adamaps-persist.service Failed

Severity: WARNING
Status: Failed at boot

Details:

• Service runs /data/persist/install.sh
• Failed at system startup (timestamp shows Jan 1 1970 — RTC not set)
• This script creates bee-tunnel.service and bee-collector.service

Root Cause: Unknown — may have failed due to network dependency or timing issue at boot

Fix:

1. Manually run: sh /data/persist/install.sh
2. Then: systemctl daemon-reload && systemctl restart bee-tunnel bee-collector

---

4. Duplicate /etc/fstab Entry for /factory

Severity: WARNING
Status: Non-blocking but causes systemd warnings

Error: Failed to create unit file /run/systemd/generator/factory.mount, as it already exists

Evidence:

/dev/mmcblk1p10      /factory             auto       defaults,nofail,ro    0  0
/dev/mmcblk1p10      /factory             auto       defaults,nofail,ro    0  0

Root Cause: Duplicate line in /etc/fstab

Fix: Remove one of the duplicate lines from /etc/fstab

---

5. factory.mount Failed

Severity: WARNING
Status: Mount failed

Details:

• Trying to mount /dev/mmcblk1p10 to /factory
• Failed with exit-code at boot

Root Cause: Likely related to duplicate fstab entry OR the partition is corrupted/missing

Fix:

1. Fix fstab duplicate
2. Check if /dev/mmcblk1p10 exists: ls -la /dev/mmcblk1p10
3. Try manual mount: mount /dev/mmcblk1p10 /factory

---

6. usb-updater.service Failed

Severity: WARNING
Status: Exit code 1

Details: Service "USB-updaer" (typo in description) failed. Likely non-critical.

Fix: Check journal for details: journalctl -u usb-updater --no-pager

---

7. Root Partition Disk Usage at 86%

Severity: WARNING
Status: / at 255MB free

Details:

/dev/mmcblk1p8   1.8G  1.5G  255M  86% /
/dev/mmcblk1p11   51G  8.2G   41G  17% /data

Root Cause: Root filesystem filling up with logs or temp files

Fix:

1. Check for large files in /var/log: du -sh /var/log/*
2. Rotate or truncate oversized logs
3. Check /tmp usage

---

8. rsyslogd File Size Limit Errors

Severity: WARNING
Status: Recurring error

Error: file size limit cmd for file '/var/log/daeman.log' failed with code -2125

Details:

• daeman.log is 5.1MB
• rsyslog configured with file size limits that are hitting constraints

Fix: Configure proper log rotation in rsyslog.conf or logrotate

---

9. LTE Service Cycling Every 10 Minutes

Severity: INFO
Status: Expected behavior?

Details:

• Restart counter at 339+
• Succeeds → runs ~10 minutes → restarts

Root Cause: May be intentional watchdog behavior or network keep-alive

Assessment: Likely by design; no action needed unless LTE is needed

---

Healthy Components

Component	Status	Notes
adacam-odc.service	✅ Running	API on port 5500 (not 5000)
hivemapper-data-logger	✅ Running	GNSS/IMU logging active
map-ai.service	✅ Running	ML pipeline functional
Redis	✅ Running	MAP_AI_READY=True
/api/1/health	✅ OK	{"status": "ok", "version": "adacam-odc-2.0.0"}
/api/1/forwarder/status	✅ OK	7814 detections forwarded, 2921 images
/api/1/ml/status	✅ OK	ML ready
GNSS frequency	✅ 4 Hz	GnssFreqHz = 4.000067
IMU frequency	✅ 197 Hz	ImuFreqHz = 197.000000
WiFi association	✅ Connected	zerocool, channel 11, WPA2-PSK
/data partition	✅ 17% used	41GB free
SSH access via AP	✅ Working	wlp1s0f0 at 192.168.0.10

---

Summary

Primary Issue: Camera is connected to WiFi but has no outbound IP connectivity. DNS and ping both fail. This blocks:

• ADAMaps detection forwarding
• bee-tunnel reverse SSH
• Any cloud connectivity

Recommended Priority:

1. Debug WiFi client connectivity (CRITICAL)
2. Fix /etc/fstab duplicate (EASY)
3. Re-run adamaps-persist installer once network works
4. Clean up root partition logs (PREVENTIVE)

---

Commands for Manual Investigation

# Check routing table conflict
ip route show
ip rule show

# Force reconnect WiFi
wpa_cli -i wlp1s0f1 reassociate

# Test from client interface specifically
ping -I wlp1s0f1 -c 3 192.168.0.1

# Check for IP conflicts
arping -I wlp1s0f1 -D 192.168.0.155

# Check router logs for this MAC
# Camera MAC on wlp1s0f1: 20:ba:36:e0:5c:16

---

Report generated by Kayos (read-only diagnostic — no changes made)