adacam Diagnostic - 2026-03-29

Summary

Root Cause: DNS resolution is completely broken on the camera. The forwarder cannot reach api.adamaps.org.

Current State: Service is running, Flask API is up, forwarder cursor is caught up to max ID, but forwarder thread is not actively cycling due to inability to resolve external hostnames.

Findings

1. Database State ✅

DB exists: Yes, at /data/recording/odc-api.db
Tables present: landmarks, frames, framekms, config, and many others
Landmarks count: 2,488 rows
Latest 5 detections: IDs 229783-229787, all roadwork-cone class
Latest timestamp: 1774367731312 ms = Tue Mar 24 15:55:31 UTC 2026

2. Forwarder Cursor State

{
  "last_detection_id": 229787,
  "last_image_id": 222654,
  "total_forwarded": 7814,
  "total_images": 2921,
  "last_run": 1774216692.601199
}

Cursor position: 229787 (matches max ID in landmarks - no backlog)
last_run timestamp: Sun Mar 22 21:58:12 UTC 2026

3. Forwarder Log Analysis

Log file: /data/adacam/forwarder.log
Last modified: 2026-03-23 03:09:06 UTC
Last entries: Successfully uploaded images, then log stopped
Errors observed (Mar 23):
- 02:53:59 - HTTPSConnectionPool read timeout
- 02:56:39 - Read timeout
- 02:58:15 - Connection aborted, timeout
- 03:08:46 - HTTP 408
- All followed by successful uploads (recoverable timeouts)
No fatal error logged - log just stops

4. Service Status

Service: adacam-odc.service active (running) since Mar 26 14:19:27 UTC
PID 626: python3 /data/adacam/adacam_odc.py
Memory: 38.9MB
Cleanup still working: Log shows age-based framekm cleanup on Mar 28

5. Network State ⚠️ THE PROBLEM

WiFi is UP:

wlp1s0f1: 192.168.0.155/24 - state UP
Default route via 192.168.0.1 dev wlp1s0f1

Internet connectivity OK:

PING 8.8.8.8: 2 packets transmitted, 2 received, 0% loss
RTT: 4.5-9.3ms

DNS is BROKEN:

ping: bad address 'api.adamaps.org'
resolvectl query api.adamaps.org: "All attempts to contact name servers or networks failed"

systemd-resolved:

Running since boot (timestamp shows 1970-01-01 - RTC issue)
Has fallback DNS configured (1.1.1.1, 8.8.8.8)
But NO link-specific DNS configuration visible
Resolution failing despite having internet connectivity

6. map-ai Status

Running: PID 729, 9.1% CPU, using 280+ minutes of CPU time
Writing detections: 248 detections written since Mar 24
All forwarded: Cursor at max ID 229787

Root Cause Analysis

Timeline:

Mar 22 21:58 UTC - Forwarder last_run timestamp
Mar 23 ~03:00 UTC - Multiple timeouts to api.adamaps.org (likely DNS starting to fail)
Mar 23 03:09 UTC - Last log entry, forwarder thread stops
Mar 24 - map-ai continues writing detections locally
Mar 26 14:19 UTC - Service restarted (but forwarder likely fails immediately due to DNS)
Mar 29 - Current state: Flask up, forwarder dead, DNS broken

Why forwarding stopped: The forwarder thread crashed or exited cleanly when it couldn't resolve api.adamaps.org. Service restart on Mar 26 likely failed to bring forwarder back because DNS was still broken.

Why DNS is broken: systemd-resolved has no configured upstream DNS for the WiFi links. The fallback servers exist but aren't being used. Possible causes:

NetworkManager/netplan/systemd-networkd misconfiguration
DHCP not providing DNS
Link-specific DNS configuration missing

Current Data Gap

Forwarded through: Mar 22-23 (based on last_run and log)
Pending forwarding: 0 detections (cursor at max)
map-ai writing: Yes, but most recent detections are from Mar 24 (car may not have driven since)

Recommended Fix (DO NOT APPLY - READ ONLY)

To restore forwarding:

Fix DNS resolution on the camera
Restart adacam-odc service
Forwarder should resume cycling and forward any new detections

Possible DNS fixes:

Add static DNS to systemd-resolved link config
Configure DHCP to provide DNS
Add DNS=8.8.8.8 to network config

4.1 KiB Raw Blame History