One of my Raspberry Pis is freezing every night, because the amount of I/O and CPU to run incremental backups is too high, and is causing other processes to misbehave.
Trying to decide whether I want to run backups from a different system (all data is in NFS), or run a different backup software (currently using Deja Dup).
For now, since rebooting every morning is getting tiresome, I just created a script that will shutdown all the services before the backup starts, and bring everything up again after it completes. Should survive a couple of nights without intervention. #HomeLab.
Stopping all the services while running the backup not only stabilized the system, but it also cut the backup to 1/4 of the time. There's definitely some improvement to be made here. #HomeLab
Uh-oh. Spoke too soon. The Raspberry Pi just froze 8 hours after the backup has completed (with all services down). Poking around the logs, it seems something related to InfluxDB trying to compact old files, which overloads the CPU and saturates the I/O on the SD card (temp files?).
Suggestions? Upgrade to InfluxDB 1.11 or 2.7 (1.8 now)? Maybe convert everything to Victoria Metrics?
I would prefer to retain the data, if possible (historical knowledge).
Wow. My "erratic" Raspberry Pi never really recovered (see thread). I've been trying different things to no avail, only changing how long it takes to freeze.
This evening I decided to reboot it while I was physically next to it, and the house was silent, which allowed me to hear a weird buzzing noise coming out of it. The buzzing coincided with access to the memory card (the when the green light in the back blinks).
I really don't know what's going on. It's the first time I hear a buzzing sound coming out of a Raspberry Pi. Even weirder is the fact that it only happens when accessing the memory card.
Gladly, the way I configured things, the memory card is hot swappable, so I simply put it in another Pi that I had laying around doing nothing (from an abandoned project). It's been working fine, without the buzz.
Hopefully that was the underlying reason since the start, and now I can get rid of the problem for good. I just don't know what to do with this "half-working Pi" (it works well under light load, just freezes up under high load).
Fingers crossed.
@badnetmask My two most common failure modes on Pis is a failing SD card or a failing external power adapter.