Capture logs on an embedded Linux device that freezes

How can I tell a device freeze from a display fault?

If the touch dies at the same moment the unit stops answering the network, the display is a victim and the fault is in the controller: the same freeze hit units that ship with no screen at all. Check it for free: ping the unit and open your backend or device portal. The fleet here shipped two products on one platform, a controller with a display and a controller without, running the same software. The freeze appeared on both, which moves the fault off the screen and into the OS.

When the screen misbehaves but the device answers ping and SSH, you have a display-chain problem instead: work through troubleshooting an embedded controller video display. A picture that freezes while the OS keeps responding points at playback, and that is an HMI video playback freeze to chase.

One split worth making on display units: plug in a USB mouse while the unit hangs. If the mouse drives the page and the touch panel stays dead, the browser and UI stack are alive and the touch driver is a fault of its own. Two faults can share a device; keep the tickets separate.

Why does journalctl show no entries when the device freezes?

No entries means that unit wrote nothing to the journal since boot, and if /var/log/journal is missing the whole journal lives in RAM, so the reboot destroys what it did record. Check which case you are in with ls /var/log/journal. With the journald default Storage=auto, a missing directory puts the journal under /run, where the next boot wipes it. The journald documentation describes the fix, and it is the one in the next section: create the directory and set persistent storage.

The service side is its own story. On the frozen unit here, the container stack and the update agent had written nothing to the journal since boot, so there was nothing to lose. Kernel messages were never in those logs: they sit in the ring buffer, which lives until the power goes. That is why the one good capture in this fleet was a dmesg dump taken over SSH while the unit hung: 40.3 KB of kernel messages, gone ten seconds later when the power was cut.

How do I make the system logs survive a reboot?

Create /var/log/journal, set Storage=persistent in /etc/systemd/journald.conf, restart the journal service, then reboot once on purpose: journalctl --list-boots must show the boot you just ended as boot -1. The steps:

1. sudo mkdir -p /var/log/journal
2. In /etc/systemd/journald.conf, set Storage=persistent.
3. sudo systemctl restart systemd-journald
4. Reboot once on purpose.
5. Check that journalctl --list-boots lists two boots. Read the one that ended with journalctl -b -1 -e, errors only with journalctl -b -1 -p err.

Bake steps 1 and 2 into the image so every unit you ship from now on keeps its history. After the next freeze, the tail of the frozen boot is waiting in boot -1 even if nobody reached the unit in time.

What do I run on the device while it is still frozen?

Run dmesg first and journalctl -b second, and save both to files named with the date, because this window closes the moment someone power-cycles the unit and it never comes back. The capture set:

sudo dmesg > dmesg-freeze-20260410-1422.txt
sudo journalctl -b --no-pager > journal-freeze-20260410-1422.txt
sudo journalctl -b -p err --no-pager
free -h
df -h
systemctl --failed

Name each file with the capture date and time, as above, so two captures from the same unit never overwrite each other. Copy the files off the unit before it is rebooted. df -h early matters: a full root filesystem is a documented cause of an embedded controller that goes unreachable, and from the outside it looks identical to a freeze.

What each symptom tells you:

• Touch dead, backend connection dead, unit offline at the same moment: a whole-device event. The kernel evidence decides it.
• SSH still connects: the scheduler runs, so this is your window. Take dmesg and the journal now.
• Ethernet link LED flashing, no ping, invisible to a network scan: the link is up and the IP stack is not answering. Note the LED pattern and the time.
• Mouse works on the kiosk page, touch dead: an input-driver fault on that unit. Separate ticket.
• Application log empty, no journal entries for the service: the evidence is kernel-side, in the ring buffer and on the console.

Why connect a serial console before the freeze happens?

Because the kernel prints an oops and its stack trace to the serial port, and once the hang takes SSH and the network down, that port is the only recorder still attached when the message arrives. You cannot attach it afterwards; a hard hang has no shell to configure. Set it up once on a bench unit:

• Find the debug UART. The module vendor documents the pin and the rate; 115200 baud is the usual setting on embedded boards.
• Add the console argument to the kernel command line in the bootloader, in the form console=<device>,115200, so kernel output goes to the port.
• Run a terminal on the bench PC that writes every line it receives to a timestamped file.

Leave the bench unit powered with the logger open. The office installation here waited 2 weeks for one freeze, so the logger has to sit through weeks of silence; an idle console writes almost nothing, and the file stays small until the day it matters.

Which kernel messages do I search for in the capture?

Search the saved files for oops, bug, panic, watchdog, blocked and mmc, because those words mark the failure families an embedded kernel produces: crashes, resets, watchdogs, hung tasks and storage faults. One grep covers everything:

grep -iE 'oops|bug|panic|watchdog|blocked|thermal|mmc' dmesg-freeze-*.txt console-*.log

The words sort the families apart. Oops and BUG lines are kernel crashes, and the stack trace printed under them names the driver, which is the biggest single clue you will get. panic is the kernel giving up. watchdog means the hardware or software watchdog fired. blocked marks a hung task: the kernel prints it when a task has been stuck past the hung-task timeout, 120 seconds by default, and the message names the wait it is stuck in. mmc lines point at storage, thermal at temperature.

Then line the capture up with the clock. The backend in this fleet flagged one unit offline at 03:59, so the first window to read was 03:55 to 04:00 in the console log and the journal.

How large should the persistent journal be?

Cap it with SystemMaxUse=200M in journald.conf: the built-in caps come from the journald documentation, 10 percent of the filesystem with a fixed top-end ceiling, and on a small root partition an explicit 200 MB keeps logging from eating the image. On an 8 GB root partition the 10 percent cap alone works out to 800 MB, which is a lot of log on a small image.

Size it to your boots. Read du -sh /var/log/journal after 24 hours, multiply by your longest boot in days, and double it. The unit captured frozen here had been up 1 week 6 days, about 13 days, so plan for boots measured in weeks. When the journal reaches its cap it discards the oldest boots first, so a tight cap costs you history and a loose cap costs you disk.

What does a worked example with a 35-device fleet look like?

Thirty-five controllers and a freeze rate of one per unit every two weeks give the fleet a frozen specimen roughly every ten hours, so the scarce resource is a recorder that was already running. The inputs, all from the field: 35 controllers delivered over 2 years, a freeze about once every 2 weeks per unit, and an uptime of 1 week 6 days on the unit that was captured frozen.

• Freeze rate per unit: 1 in 14 days, about 0.07 per day.
• Expected freezes across the fleet: 35 x 0.07 = 2.5 per day.
• One frozen specimen every 24 / 2.5 = 9.6 hours.

The fleet hands you a frozen unit roughly twice a day. What it cannot hand you is a recorder that was already running, and that is the whole case for the three changes above. The fleet also updated every unit to the latest long-term-support build and the freezes came back, which is its own data point: a fault that survives a major OS update lives below the application, in the kernel or a driver.

The capture arithmetic: 40.3 KB of dmesg is about 41,000 bytes, and at roughly 100 bytes a line that is about 410 lines. One grep covers them. The backend timestamp 03:59 sets the reading window, 03:55 to 04:00, first in the console log, then in boot -1.

Recommendation: Ship Storage=persistent with SystemMaxUse=200M in the image on every unit, keep one bench unit on a logging serial console at 115200 baud, and capture dmesg plus journalctl -b on every unit found frozen before anyone reboots it. Commit to this when the freeze recurs on a multi-week cycle and has survived an OS update; if the fault appears only on units with a screen while ping and SSH keep answering, spend the first day on the display chain instead.

Questions