Why we need to do that?
Reason 1: save your SD card / EMMC chip from dying earlier
And you need to know, the more you write to SD card, the less it lasts. So let's start. The idea of this exercise is to turn off as much write-heavy activities as we can while leaving system up and running.
Reason 2: I don't like a lot of automation, which Linux offers out of the box
In this sense, Linux is becoming a sort of mini-me of Windows, I hate so much. I can hardly stay calm when I see some windows service is eating up a lot of CPU / memory / doing some IO or sending something over the netwrok and I can't even get to know what that service is doing.
- if it's doing something crucial for OS to live, and killing it will break Windows?
- is it something auxiliary Windows can live without?
- does it mine some cryptocoins for Microsoft' benefit, but on my own hardware?
... because Windows is closed source, and you cannot get inside of it to see which of the above is true.
I also deeply hate Windows, when it decides it's time to apply some patches, perform some kind of housekeeping for whatever built-in stuff or do a system reboot to make a system upgrade. Without asking me.
As I said, unfortunately in Linux (especially in Debian linux) I see it's drifting towards Windows behavior, and nobody cares about it. See the proof#1 and proof#2.
But fortunately, this is still Linux, and it's open soruce with tonns of documentation and questions being already asked and answered. So we can finetune it whatever we want, and do that knowingly, without any risk to get something broken in the completely distant part of the system. This is why I love Linux so much.
Disclaimer
- If you have your SBC connected directly to Internet with all ports exposed and you're running tons of software from 3rd parties / non official repos / snap which are exposing itslef by opening these ports to outer world, inviting hackers to come in
- if you're using web browser of old version
... this might be an issue, yeah. But, if you'll be doing apt update & apt upgrade time-to-timeyourself, it's nothing different to have this automatic update services running.
Remove cron jobs
/etc/cron.d:
e2scrub_all orangepi-truncate-logs orangepi-updates sysstat
/etc/cron.daily:
apt-compat cracklib-runtime locate man-db samba
aptitude dpkg logrotate orangepi-ram-logging sysstat
/etc/cron.hourly:
fake-hwclock
/etc/cron.monthly:
/etc/cron.weekly:
man-db tor
TODO: I want to see it myself, how can I get that tag/flag value as a result of e2scrub_all working. And probably get notified about that, rather than fsck will be silently fixing some issue
/etc/cron.d/sysstat data collector for sysstat. Sysstat utilities are a collection of performance monitoring tools for Linux. These include sar, sadf, mpstat, iostat, tapestat, pidstat, cifsiostat and sa tools.
The cronjob is just a misery. It runs pretty much all the time /usr/lib/sysstat/debian-sa1 script, but that script exits if systemd is there - because sysstat was moved to systemd already. I've deleted that /etc/cron.d/sysstat
TODO: it's worth investigating where the raw statistics being put by systemd version of this grabber, to move that place to zram
/etc/cron.daily/apt-compat doesn't run if systemd is in place. Ho harm but I deleted that
/etc/cron.daily/aptitude a script that saves package states to a log file. Another logging/backup courtesy I never asked for. Deleting that
/etc/cron.daily/cracklib-runtime - part of cracklib-runtime for lame people who're using dictionary passwords. But we're not like them, right? Then purging the whole cracklib-runtime and libcrack2 packages together.
/etc/cron.daily/dpkg - part of dpkg. Backups the metadata from dpkg "database" about installed packages. Like if it could do something about it, if it finds something is broken. Hehe. Added "exit 0" to the top to disable it
/etc/cron.daily/locate - daily update to locate database. drop that. If you cannot find something with locate, and you're sure you have updated its index with updatedb recently and didn't change a bit since then - then the file is not there. No need to hammer your filesystem on a daily basis.
/etc/cron.daily/logrotate - is not doing anything when systemd is there. Can stay
/etc/cron.daily/man-db - also not doing anything when systemd is there (I'm assuming there's an appropriate timer, we'll look at them bit later). Can stay
Looks like a lot of Debian packages are made with having this idea in mind that users might want to switch from systemd to initd, so all these crontabs will start to matter again.
/etc/cron.daily/orangepi-ram-logging - not doing anything. logrotate.timer is doing all the job instead. Can stay
/etc/cron.daily/samba - if you remember, we needed samba for our EmulationStation, so we could copy roms from our Windows machine with using its File Explorer. This job makes a backup of /etc/samba/smbpasswd file to /var/backup if it has changed. Really, the guy who wrote it is just a genius of proper backup strategy. Joking, he's not. Remove the job.
/etc/cron.daily/sysstat - doesn't do anything. All job was moved to systemd timers. Can stay
/etc/cron.weekly/man-db - not being executed with systemd. Can stay
Remove/disable unneeded systemd timers
Wed 2022-08-31 22:10:00 MSK 5min left Wed 2022-08-31 22:00:59 MSK 3min 6s ago sysstat-collect.timer sysstat-collect.service
Thu 2022-09-01 00:00:00 MSK 1h 55min left Wed 2022-08-31 00:00:02 MSK 22h ago logrotate.timer logrotate.service
Thu 2022-09-01 00:00:00 MSK 1h 55min left Wed 2022-08-31 00:00:02 MSK 22h ago man-db.timer man-db.service
Thu 2022-09-01 00:07:00 MSK 2h 2min left n/a n/a sysstat-summary.timer sysstat-summary.service
Thu 2022-09-01 00:44:45 MSK 2h 40min left Wed 2022-08-31 00:44:45 MSK 21h ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Thu 2022-09-01 06:44:30 MSK 8h left Wed 2022-08-31 06:22:59 MSK 15h ago apt-daily-upgrade.timer apt-daily-upgrade.service
Thu 2022-09-01 17:18:53 MSK 19h left Wed 2022-08-31 20:13:53 MSK 1h 50min ago apt-daily.timer apt-daily.service
Sun 2022-09-04 03:10:13 MSK 3 days left Sun 2022-08-28 03:10:50 MSK 3 days ago e2scrub_all.timer e2scrub_all.service
Mon 2022-09-05 01:26:35 MSK 4 days left Mon 2022-08-29 00:23:07 MSK 2 days ago fstrim.timer fstrim.service
xxxx
xxxx
xxxx
xxxx
xxxx
xxxx
xxxx
xxxx
xxxx
xxxx
xxxx
Monitor what remaining processes are writing heavily to SD card and deal with it, one-by-one
- "raw" tools
- we can attempt to find any higher-level tools with using apt-rdepends -r [rawpackage])
- some random tools we found on internet (like fatrace and csysdig).
So let's start rolling on with simple things
fatrace
- to look after only specific events (-f W to monitor writes to files)
- to limit the scope to consider only one mounted device, from the current directory (-c), so I first walked into /
- to add timestamps to its log (-t)
I left it running for a while, and then I examined the resulted log file.
cd / ; fatrace -c -t -f W | tee /tmp/fatrace.log | tee -a /tmp/fatrace.log
The only unfortunate thing about fatrace is that it does not provide you a bit of info about amounts of data being written and you cannot overcome that, because the system interfaces it's using are not giving these figures either.
In my case I see it already what processess were constantly writing their shit very important data to my precious SD card. Mainly it was Firefox and down below I'll to teach it how not to update its bolloks sqlite database files inside of ~/.mozilla/firefox/
The problem however is not only about Firefox alone. All the rest browsers I tested were doing the very same thing. But don't you worry. We'll deal with them as well.
Thanks to fatrace I also discoverd some some vnstat daemon was collecing networking statistics and putting that to its own file to /var/lib/
What was that idiot, who installed that? Was it me? Given no other package was depending on that vnstat, apart from its own mini-me vnstati. Go to hell, both of you:
apt purge vnstat
iotop
On a contrast to previous tool, now you can see amounts of data written, but you cannot limit it to see only on a specific partition (like the / - mounted to SD card). So applicability of this tool is limited to see only amounts of data written, but not knowing where exactly that data was written to. Here are few examples how can you run it:
iotop -bktoqqq
In this mode I found iotop has a bug - it was not capturing write events from short-lived processes, like the process of screenshot creation. So it might miss some important share of IO load from such processess.
iotop -obPat
Same story here. Firefox is winning the race by pushing few Mb of its shitty cache to SD card within a matter of few minutes. Firefox, I hate you! Why you're doing that? I have plenty, you hear it, plenty of RAM free. You're ruining my precious SD card.
iostat
Every tool is using its own unique way of tracking IOs. So from iostat you can expect IO breakdown per disk. If you run it like this:
iostat -dzphN 10
... it will first show you the accumulated report since the systemboot (on a screenshot it's appearing on the very top) and then will be showing deltas every 10 seconds:
log2ram + zram
/dev/zram1 on /var/log type ext4 (rw,relatime,discard)
root@orangepi4-lts:~# df
Filesystem 1K-blocks Used Available Use% Mounted on
udev 1904036 8 1904028 1% /dev
tmpfs 395600 2932 392668 1% /run
/dev/mmcblk2p1 122600084 86461672 34849544 72% /
tmpfs 1978000 0 1978000 0% /dev/shm
tmpfs 5120 8 5112 1% /run/lock
tmpfs 1978000 16 1977984 1% /tmp
/dev/zram1 49560 19796 26180 44% /var/log
tmpfs 395600 44 395556 1% /run/user/1000
tmpfs /var/tmp tmpfs size=10M,nosuid 0 0
tmpfs /var/cache/samba tmpfs size=5M,nosuid 0 0
If you're using some other Linux distro, read how to configure log2ram here - https://ikarus.sg/extend-sd-card-lifespan-with-log2ram/
disable swap (makes sense for 4Gb+ RAM and higher models) or make sure swap is using zram
- I hardly seen my system was running out of memory. It was just once, due to the bug in Gwenview. But guys, I'm having luxurious 4 Gb of RAM
- Using compression mechanisms on zram will definitely hurt CPU, when it comes to moving something to SWAP or reading it back
NAME TYPE SIZE USED PRIO
/dev/zram0 partition 1.9G 416M 5
some further steps
https://raspberrypi.stackexchange.com/questions/169/how-can-i-extend-the-life-of-my-sd-card
Disable system.d services we don't need
│ └─119304 /usr/sbin/rpc.mountd --manage-gids
│ └─119752 /usr/sbin/blkmapd
│ └─119303 /usr/sbin/rpc.idmapd
│ └─16489 /usr/libexec/packagekitd
│ └─1123 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown
│ └─2296 /usr/libexec/upowerd
│ └─2072 /usr/libexec/accounts-daemon
│ └─745 /usr/sbin/haveged --Foreground --verbose=1
nfs-*.service
who needs nfs nowdays? drop that!
sudo nala remove nfs-common
packagekit.service
pi@orangepi4-lts:~ $ dpkg -S /usr/libexec/packagekitd
packagekit: /usr/libexec/packagekitd
pi@orangepi4-lts:~ $ apt info packagekit
Package: packagekit
Version: 1.2.2-2
Priority: optional
Section: admin
Maintainer: Matthias Klumpp <mak@debian.org>
Installed-Size: 2,857 kB
Depends: libglib2.0-bin, policykit-1, init-system-helpers (>= 1.52), libappstream4 (>= 0.10.0), libapt-pkg6.0 (>= 1.9.2), libc6 (>= 2.28), libgcc-s1 (>= 3.0), libglib2.0-0 (>= 2.54), libgstreamer1.0-0 (>= 1.0.0), libpackagekit-glib2-18 (>= 1.2.1), libpolkit-gobject-1-0 (>= 0.99), libsqlite3-0 (>= 3.5.9), libstdc++6 (>= 5.2), libsystemd0 (>= 214)
Recommends: packagekit-tools, systemd
Suggests: appstream
Breaks: libpackagekit-glib2-14 (<= 0.7.6-4), libpackagekit-qt2-2 (<= 0.7.6-4), packagekit-backend-apt (<< 1.0), packagekit-backend-aptcc (<< 1.0), packagekit-backend-smart (<< 1.0), packagekit-offline-update (<< 1.0), packagekit-plugin-click (<= 0.3.1), plymouth (<< 0.9.5)
Homepage: https://www.freedesktop.org/software/PackageKit/
Tag: admin::package-management, implemented-in::c, implemented-in::python,
role::program
Download-Size: 575 kB
APT-Manual-Installed: no
APT-Sources: http://deb.debian.org/debian bullseye/main arm64 Packages
Description: Provides a package management service
This is the abstraction layer which makes it possible for applications like KDE Discover to work on any kind of distros, no matter what software packaging tools they're using - dpkg/apt, rpm/yum/dnf, pacman - whatever.
If you want to keep using KDE Discover you'll need to keep that. I disabled it.
unattended-upgrades.service
pi@orangepi4-lts:~ $ dpkg -S /usr/share/unattended-upgrades/unattended-upgrade-shutdown
unattended-upgrades: /usr/share/unattended-upgrades/unattended-upgrade-shutdown
pi@orangepi4-lts:~ $ apt info unattended-upgrades
Package: unattended-upgrades
Version: 2.8
Priority: optional
Section: admin
Maintainer: Michael Vogt <mvo@debian.org>
Installed-Size: 334 kB
Depends: debconf (>= 0.5) | debconf-2.0, debconf, python3, python3-apt (>= 1.9.6~), python3-dbus, python3-distro-info, ucf, lsb-release, lsb-base, xz-utils
Recommends: systemd-sysv | cron | cron-daemon | anacron
Suggests: bsd-mailx, default-mta | mail-transport-agent, needrestart, powermgmt-base, python3-gi
Tag: admin::package-management, implemented-in::python, role::program,
suite::debian, works-with::software:package
Download-Size: 88.6 kB
APT-Manual-Installed: yes
APT-Sources: http://deb.debian.org/debian bullseye/main arm64 Packages
Description: automatic installation of security upgrades
Self explanatory. Remove that. I'll be able to install all my security updates myself!
apt purge unattended-upgrades
upower.service
pi@orangepi4-lts:~ $ dpkg -S /usr/libexec/upowerd
upower: /usr/libexec/upowerd
pi@orangepi4-lts:~ $ apt show upower
Package: upower
Version: 0.99.11-2
Priority: optional
Section: admin
Maintainer: Utopia Maintenance Team <pkg-utopia-maintainers@lists.alioth.debian.org>
Installed-Size: 420 kB
Depends: dbus, udev, libc6 (>= 2.17), libglib2.0-0 (>= 2.41.1), libgudev-1.0-0 (>= 147), libimobiledevice6 (>= 0.9.7), libplist3 (>= 1.11), libupower-glib3 (>= 0.99.8), libusb-1.0-0 (>= 2:1.0.8)
Recommends: policykit-1
Homepage: https://upower.freedesktop.org/
Tag: admin::power-management, hardware::power, hardware::power:acpi,
hardware::power:ups, implemented-in::c, interface::daemon,
role::program
Download-Size: 113 kB
APT-Manual-Installed: no
APT-Sources: http://deb.debian.org/debian bullseye/main arm64 Packages
Description: abstraction for power management
haveged.service
pi@orangepi4-lts:~ $ dpkg -S /usr/sbin/haveged
haveged: /usr/sbin/haveged
pi@orangepi4-lts:~ $ apt info haveged
Package: haveged
Version: 1.9.14-1
Priority: optional
Section: misc
Maintainer: Jérémy Bobbio <lunar@debian.org>
Installed-Size: 92.2 kB
Pre-Depends: init-system-helpers (>= 1.54~)
Depends: lsb-base (>= 3.2-14), libc6 (>= 2.17), libhavege2 (>= 1.9.13)
Suggests: apparmor
Homepage: https://issihosts.com/haveged/
Tag: implemented-in::c, interface::daemon, role::program, scope::utility,
security::cryptography
Download-Size: 39.1 kB
APT-Manual-Installed: yes
APT-Sources: http://deb.debian.org/debian bullseye/main arm64 Packages
Description: Linux entropy source using the HAVEGE algorithm
Random number generation daemon. I'm not joking. And it's important part of distribution, as a lot of crypto things are depending on having a truely random number being generated. Don't touch it. I eats just 3 megs of ram but it provides a truly randomization for /dev/random
accounts-daemon.service
Coming soonconfigure X to not generate huge .xsession-errors file (or move that file to /tmp
...
...
if (umask 077 && touch "$ERRFILE") 2> /dev/null && [ -w "$ERRFILE" ] &&
[ ! -L "$ERRFILE" ]; then
chmod 600 "$ERRFILE"
elif ERRFILE=$(tempfile 2> /dev/null); then
if ! ln -sf "$ERRFILE" "${TMPDIR:=/tmp}/xsession-$USER"; then
message "warning: unable to symlink \"$TMPDIR/xsession-$USER\" to" \
"\"$ERRFILE\"; look for session log/errors in" \
"\"$TMPDIR/xsession-$USER\"."
fi
else
errormsg "unable to create X session log/error file; aborting."
fi
exec >>"$ERRFILE" 2>&1
ln -s /tmp/$USER.xsession-errors ~/.xsession-errors
Disable journaling for ext4 filesystems
We're not running production server of a patients-life-critical application in hospital. If we loose a bit of info or some app will corrupt it's cache, if our SBC will be unexpectedly poweroff, we can survive that. We don't have such apps, who won't survive if. So let's go:
tune2fs -l /dev/mmcblk2p1
tune2fs -O ^has_journal /dev/mmcblk2p1
e2fsck -f /dev/mmcblk2p1
rebootConfigure your browsers to not write its cache that aggressively to your homedir
Firefox
browser.cache.disk.enable = false
browser.cache.disk_cache_ssl = false
+ I'll need to search for more, as it still writes to number of its internal sqlite files like:
pi@orangepi4-lts:~ $ grep -oE "\/home[^ ]*" /tmp/fatrace.log | sort | uniq -c
10 /home/pi/.mozilla/firefox/(x).default-esr/AlternateServices.txt
1 /home/pi/.mozilla/firefox/(x).default-esr/broadcast-listeners.json
1 /home/pi/.mozilla/firefox/(x).default-esr/broadcast-listeners.json.tmp
325 /home/pi/.mozilla/firefox/(x).default-esr/cookies.sqlite
2366 /home/pi/.mozilla/firefox/(x).default-esr/cookies.sqlite-wal
4 /home/pi/.mozilla/firefox/(x).default-esr/datareporting/aborted-session-ping
4 /home/pi/.mozilla/firefox/(x).default-esr/datareporting/aborted-session-ping.tmp
1 /home/pi/.mozilla/firefox/(x).default-esr/datareporting/archived/2022-09/xxxmain.jsonlz4
1 /home/pi/.mozilla/firefox/(x).default-esr/datareporting/archived/2022-09/xxxmain.jsonlz4.tmp
1 /home/pi/.mozilla/firefox/(x).default-esr/datareporting/session-state.json
1 /home/pi/.mozilla/firefox/(x).default-esr/datareporting/session-state.json.tmp
5 /home/pi/.mozilla/firefox/(x).default-esr/favicons.sqlite
123 /home/pi/.mozilla/firefox/(x).default-esr/favicons.sqlite-wal
28 /home/pi/.mozilla/firefox/(x).default-esr/formhistory.sqlite
122 /home/pi/.mozilla/firefox/(x).default-esr/formhistory.sqlite-journal
50 /home/pi/.mozilla/firefox/(x).default-esr/permissions.sqlite
135 /home/pi/.mozilla/firefox/(x).default-esr/permissions.sqlite-journal
54 /home/pi/.mozilla/firefox/(x).default-esr/places.sqlite
420 /home/pi/.mozilla/firefox/(x).default-esr/places.sqlite-wal
15 /home/pi/.mozilla/firefox/(x).default-esr/prefs-1.js
3 /home/pi/.mozilla/firefox/(x).default-esr/prefs.js
3 /home/pi/.mozilla/firefox/(x).default-esr/protections.sqlite
9 /home/pi/.mozilla/firefox/(x).default-esr/protections.sqlite-journal
1 /home/pi/.mozilla/firefox/(x).default-esr/sessionstore-backups/recovery.jsonlz4
121 /home/pi/.mozilla/firefox/(x).default-esr/sessionstore-backups/recovery.jsonlz4.tmp
8 /home/pi/.mozilla/firefox/(x).default-esr/SiteSecurityServiceState.txt
22 /home/pi/.mozilla/firefox/(x).default-esr/storage/default/moz-extensionxxx/idb/xxx-eengsairo.sqlite
18 /home/pi/.mozilla/firefox/(x).default-esr/storage/default/moz-extensionxxx/idb/xxx-eengsairo.sqlite-wal
4 /home/pi/.mozilla/firefox/(x).default-esr/storage/permanent/chrome/idb/xxxxAmcateirvtiSty.sqlite
4 /home/pi/.mozilla/firefox/(x).default-esr/storage/permanent/chrome/idb/xxxxAmcateirvtiSty.sqlite-wal
6 /home/pi/.mozilla/firefox/(x).default-esr/storage/permanent/chrome/idb/xxxxrsegmnoittet-es.sqlite
5 /home/pi/.mozilla/firefox/(x).default-esr/storage/permanent/chrome/idb/xxxxrsegmnoittet-es.sqlite-wal
4 /home/pi/.mozilla/firefox/(x).default-esr/webappsstore.sqlite
42 /home/pi/.mozilla/firefox/(x).default-esr/webappsstore.sqlite-wal
8 /home/pi/.mozilla/firefox/(x).default-esr/xulstore.json
6 /home/pi/.mozilla/firefox/(x).default-esr/xulstore.json.tmp
Might need to raise a bug about this
https://bugzilla.mozilla.org/buglist.cgi?query_format=specific&order=relevance+desc&bug_status=__all__&product=&content=places.sqlite-wal&comments=0
Qutebrowser
Chromium
Mesuring our success
Linux 5.18.5-rk3399 (orangepi4-lts) 09/02/2022 _aarch64_ (6 CPU)
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device
0.01 0.2k 0.0k 0.0k 5.3M 0.0k 0.0k mmcblk0
0.00 0.1k 0.0k 0.0k 3.2M 0.0k 0.0k mmcblk0p1
0.00 0.0k 0.0k 0.0k 348.0k 0.0k 0.0k mmcblk0boot0
0.00 0.0k 0.0k 0.0k 348.0k 0.0k 0.0k mmcblk0boot1
0.50 14.7k 0.9k 0.0k 499.5M 31.4M 0.0k mmcblk2
0.50 14.6k 0.9k 0.0k 497.5M 31.4M 0.0k mmcblk2p1
0.02 0.1k 0.0k 0.0k 2.3M 4.0k 0.0k zram0
0.14 0.0k 1.5k 0.0k 476.0k 51.3M 0.0k zram1
13:21:31 up 9:41, 2 users, load average: 1.16, 1.24, 1.22