Ultimate cleanup of Debian/Ubuntu/POP OS/Elementary/whatever-is-using-APT and RPM based distros

In the previous exercise we have removed all the extra/unwanted/unneeded services from our distro, this time we are going to reclaim back some disk space.

First of all, why you might want to do this kind of cleanup?

We all know and laugh on Windows 10/11 disk requirements, right? But out-of-the-box Linux distros are becoming nothing better than it. And the reason for that is, distro teams are trying to squeeze as much as they can into the distribution, so most use cases will be covered. It might not sound like a bad idea, but what is the point for you, yes, specifically you, to have installed on your disk (and running as a daemon) such a thing as CUPS, if you don't even have a printer at home, right? Or having some graphical themes for your GRUB, when you don't care about bootloader beauty? Or having SNAP deamon installed and running, if you don't like vaping long-bearded hipsters and all the novelties and prefer your software to be installed from distro repo? Or why on earth you need to spare 300+ Mb on your hard drive to have "wireless-regdb" package (wireless regulatory database) for a system without WiFi?

As I mentioned, I do own an SBC which is running either off EMMC or off SD card. As you can imagine, the storage there is not endless. I wanted my OS to be as compact as it could be, without hurting much to its operation capabilities and not by paying a price of having a limited amount of applications. Quite the opposite - I want my free space to be used by the applications I use and by my own content (photos, music, videos). 

Secondly, the less free space you have on your solid state drive, no matter what generation it is, the less it will span. This might sound like a joke, unless you investigate that on your own, how SSD is operating and how writes are distributed across free blocks. Logic here is very simple, whenever you update something on your disk, the SSD controller will likely mark the storage cell occupied with your "old" data as clean, without actually cleaning it, and copy modified things around to a new storage cell. This approach is called "copy-on-write", and this is what all solid-state-drive controllers are doing, underneath the hood. So the less free space you have on your drive, the more pressure those free cells will get.

Before we begin

Your next best friend should be a tool like KDE Filelight or GNOME Baobab (recently was renamed to generic "Disk usage analyzer"). 

You open it up, and look very carefully what takes the most space on your drive. 

Delete apt cache

Unfortunately apt (a Debian-based distros package manager) has a bad habit of leaving the trash behind. It's just coded that way, so it first downloads the package to a local "cache"- your hard drive - and only then it installs it. And guess what? It doesn't give a shit to wipe out whatever remained after it. It's like if you never emptied your "Downloads" folder in Windows :) 

So we can do that manually:

apt-get autoclean
apt autoclean
apt clean

Compact the jorunal

Quite time ago, Linux distributions have switched from keeping a good old plain text log files under /var/log to something new and shiny. That something new was called journalctl. Basically that's a service which keeps up a binary log from whatever other service or software who wants to put something into system log. Binary here is a good thing in terms of space occupied, because it's using some kind of compression. But the bad thing about it, is that 90% of desktop Linux users don't even know how to look into those journalctl logs and they never do. 

If you're not a big fan of archeological digging into your old journalctl records from a month ago, I strongly suggest you to limit, how much logs can journalctl write to your hdd.

sudo journalctl --vacuum-size=100M

Delete extra locales

du -hs /usr/share/locale/*
find /usr/share/locale/ -type f -exec dpkg -S {} \; | sort -u

TODO https://serverfault.com/questions/394610/remove-a-locale-in-ubuntu/1037183#1037183

Delete extra kernels you don't need

First figure it out, what kernel you're running now:

uname -r

Check out what other kernels you have installed - the below directory usually gives a good indication about what amount of hdd is being used by kernel modules, so you can give it a second thought:

du -hs /usr/lib/modules/*

Then go ahead to remove all the kernels (and kernel-specific packages) you no longer need. Instead of 5.15.0-50 in the below example, use the kernel version from above, one by one:

apt list --installed "*5.15.0-50*"

# compose a list manually

apt purge <list of packages>

Delete snaps / flatpacks you don't use or alltogether

flatpak list

apt autoremove flatpak
rm -rf /var/lib/flatpak

snap list

apt autoremove snapd

rm -rf /usr/lib/snap

Remove the swapfile / swap partition

xxxx

Find and delete the fattest software

Unfortunately, I found no easy tool to use, how can you measure what is the storage impact of the packages you installed manually, considering all the dependencies it brought, when the dependencies are only needed to run that your package. All the GUI tools that are coming with various Desktop Environments are doing the same mistake - whenever they're calculating the size occupied by a package, they don't consider its dependencies.

So I had to make my own simple scrip: 

Debian-based distros

cat << EOF > /usr/local/bin/apt-space-used-by
#!/bin/sh
out=\$(apt-get --assume-no autoremove \$1 2>&1)
ec=\$?
if [ \$ec -eq 1 ] ; then
 size=\$(echo -n "\$out" | grep "After this operation" | cut -d' ' -f4-5)
 size=\$(echo \$size | sed -e "s/[^0-9kMG,.]//g" | tr 'k' 'K')
 echo -n "\$size\\t"
else
 echo -n "0 (cannot delete)"
fi
echo -n "\$1\\t"
dpkg-query -W -f='\${binary:Summary}\\n' \$1
EOF
chmod a+x /usr/local/bin/apt-space-used-by

# before we go any further we need to cleanout all the orphan pacakges, as they will be bothering our little script
apt autoremove

# now if you want to see what packages you installed manually will free what space
apt-mark showmanual | xargs -I % sh -c "apt-space-used-by %" | sort -h

# ... or whatever other packages which came with your distro
echo "" > /tmp/final_report.txt
dpkg-query -W -f='${binary:Package}\n' | xargs -I % sh -c "apt-space-used-by %" | tee -a /tmp/final_report.txt
cat /tmp/final_report.txt | sort -h

RHEL-based distros

cat << EOF > /usr/local/bin/yum-space-used-by
#!/bin/sh
out=$(yum --assumeno erase $1 2>&1)
echo -n "$out" | grep -qE "^Freed space:"
ec=$?
if [ $ec -eq 0 ] ; then
 size=$(echo -n "$out" | grep -E "^Freed space:" | cut -d' ' -f3-4)
 size=$(echo $size | sed -e "s/[^0-9kMG,.]//g" | tr 'k' 'K')
 echo -ne "$size\t"
else
 echo -ne "0 (cannot delete)"
fi
echo -ne "$1\t"
rpm -q --queryformat="%{SUMMARY}" $1
echo ""
EOF
chmod a+x /usr/local/bin/yum-space-used-by


# before we go any further we need to cleanout all the orphan pacakges, as they will be bothering our little script
yum autoremove

# now if you want to see what packages you installed manually will free what space
yum history userinstalled | grep -v "Packages installed by user" | xargs -I % sh -c "yum-space-used-by %" | sort -h

# ... or whatever other packages which came with your distro
echo "" > /tmp/final_report.txt
rpm -qa | xargs -I % sh -c "yum-space-used-by %" | tee -a /tmp/final_report.txt
cat /tmp/final_report.txt | sort -h

A word of caution regarding the last command. Imagine you have git installed. The "git" package brings with it a set of mandatory dependencies, it couldn't live without, like "git-man". So if you delete "git-man", it will also delete "git". This is why you will see some that both "git" and "git-man" packages will free up the same amount of disk space.

Once you figured out what you're ready to remove run:

apt autoremove <pacakgename>

Upsize your partitions

It might be the case, that you do have some unallocated space on your drive.  That's quite easy to fix. Imagine you have a disk (/dev/sda) with a single partition (/dev/sda1) and some free space after that partition. You first run lsblk to confirm what kind of layout you have, then you run parted and resize that 1st partition (/dev/sda1) to occupy 100% of remaining free space. And the cherry on a cake - you upsize the filesystem. That's it. Everything can be done online, without the need for reboot.

lsblk
parted /dev/sda
print all
resizepart 1 100%
resize2fs /dev/sda1

TODO

# remove dev packages you installed manually

apt-mark showmanual | grep -E "\-dev$" | awk '{system("sudo apt-get --dry-run purge "$1)}'

# remove 

It's safe to remove the content of your trashbin:
~/.local/share/Trash

See also what kind of programs you might have already deleted, but they left behind their traces:
~/.local/share

Like in my case I had some trails left by Konqueror (browser) I was experimenting with, and then used "apt remove" instead of "apt purge"
It's just me being not very carefull, but there's a good thing about it, we can pick up all such traces in one go:

dpkg --get-selections | awk '$2=="deinstall" {system("sudo apt-get --dry-run purge "$1)}'
dpkg --get-selections | awk '$2=="deinstall" {system("sudo apt-get -y purge "$1)}'

What's the heck wrong with this software?

Balena etcher

One of the most famous and popular tools to burn ISOs to USB flash drives.
If you're running it on Windows, have a look, how much is wasted in these dirs:

• C:\Users\<youruser>\AppData\Local\Programs\balena-etcher
• C:\Users\<youruser>\AppData\Local\balena-etcher-updater\

and be surprised, how it came that the tool to write ISOs weights several times more, than a full-fledged Linux OS

Building up Google Photos replacement using your hardware

If you think long enough, you'll figure that there are number of components you'll need. 

But before we go into all these details further, I need to share my view of what I think should be the best way of organizing our media library. The most important thing is, it should be using EXIF tags as a primary source of metadata for organization. 

Why EXIF tags? 

Simply because they could be or already are written inside of your media files - both photos and videos. So you won't be depending on some specific media library application, which is storing all your organization entities (what files belong to what events / dates / folders) somewhere deep inside of its own proprietary database. We want a flexibility as we don't want to be trapped into vendor-locked-in situation, so we need to ensure we will be able to migrate from one app to another. In future, when the time comes. 

Speaking of EXIF tags here, I'm not a big fan of getting deep inside of things like ISO, exposure, saturation, all that camera-technical shit. I'm not that kind of crazy photographer person, I don't care about all that. But what I do need is to be able to mark a bunch of photos with some human-readable descriptive tag like "2022.09 - Going far east". So I can later see, what tags do I have in my library and look only on photos having that tag.

We can, for sure, organize our own library using good old folders approach. Like to store together  photos/videos belonging to the same event. But that approach alone lacks flexibility, if we want the same photo/video to appear in more than just a single "folder". Or if we want to find all photos/videos made in a particular place. Or if we want to find all photos, we tagged as "retro cars" or "skiing in mountains". So folders might be nice for physical storage organization, but not as a way to access your media library.

Mobile app to view / modify media

First of all we need a photo gallery application on our mobile phone, as the most shots we're doing using it. We need to navigate through our photos/videos and modify them as needed. 

Requirements:

- modify dates of photos / videos
- modify GPS locations for photos / videos
- assign/modify custom EXIF tags for photos / videos
- recognize faces and objects in  photos and assign tags for them automatically
- explore your media library on your phone, utilizing all the above - dates, GPS location, custom text tags
- built-in map, so we could use it for navigation
- bulk editing features (to assign GPS, date and text tags to a bunch of items)
- built-in video player
- showing thumbnails for video

The best app I found is an open-source Aves. It's almost perfect, just missing support for modifying dates / tags for videos, but developer is there, he's super responsive and claiming this feature is already planned. I raised couple of small bugs and features requests and they were all processed. I do admire that guy. And he is among the minority (unfortunately) who really understand it, how to make proper user-friendly UI.

The next best thing might have been a proprietary app called Piktures, but it has an idiotic bug they don't recognize. And it's a proprietary shit. And it's missing half of features Aves have.

There's also a proprietary Android app called "F-Stop" but it's buggy as hell and UI is made for goblins. And the free version cannot do much.

So really, there's not many alternatives to Aves, so we stick to it. Thibault Deckers, mate, you're the best!

A way to sync our media across  devices

Google Photo mobile app does that for you, it's uploading photos from the device' internal storage to their own servers. So we need a way to do the same ourselves, as we're building a Google Photo replacement. 

Ideally you want to have all your devices to be synced with each other, like if you shoot a new photo on your phone, it should appear on your PC and on your tablet. If you modified that photo later on your PC, like you set proper GPS coordinates your phone "forgot" to set, this photo should be synced back to your phone and tablet. And all of that should work over the internet securely, as you might be away from home WiFi. This is called two-way sync.

In my previous post I explained how this can be achieved by using Tor and apps like FolderSync. Tor we need here just to bypass NAT, if our home PC is running behind ISP cgNAT and our own router's NAT. If you have a white IP address at home (or IPv6), you might not need it.

But there's also an open source app called SyncThing. You can also build a two-way synchronization with it and developers are claiming it doesn't require anything specific to bypass NAT, if you want to sync over internet. If you're windows user, give it a shot with SyncTrayzor, which is nice GUI wrapper for SyncThing. The setup is quick and easy, so you can share any folders from your Android to be permanently in sync with folders on your PC.

Desktop app to view / modify your media library

This is something, that Google Photo doesn't have. Google is only offering you a Web application. But we want to be able to browse our photo library (which is kept synced to our PC and all other devices as explained in the previous section) on a PC the very same convenient way how we do that on our phone with using Aves. 

Requirements for this app are the same as for the phone app I listed above.

Given we're on Linux we can benefit from a variety of open source apps. I tested lot of them and the only I can suggest is DigiKam, so we take it.

Web application to access your photos from remote PC and share then

This is the last piece in puzzle. And it's also optional and might be needed:

- if you're planning to share your photos by giving readonly access to specific albums to your friends & family (by just giving them some URL)

- if you're planning to view your whole media library while being away from your home network

XXXX

Ultimate guide to Linux Photo / Media managers

gThumb 😉

https://gitlab.gnome.org/GNOME/gthumb

(+) UI is good. Minimalistic and sleek.
(+) It recognized my tags on photos I set in Aves
(+) I was able to create "virtual folders", as they call it, based on those tags - with all my photos in it. So it does support a good navigation based on tags
(+) It does support bulk tags assignment
(+) it does support proper sorting, based on Exif information
(+) it does have a mini-map which shows current photo location

(*) It has built-in videoplayer, but it doesn't quite work for me, so I need to investigate
(*) Video thumbnails are not shown, need to investigate it

(-) Tags on video files are not considered -  #249
(-) Doesn't have any support for faces recognition
(-) You cannot change GPS location for photos
(-) You cannot use map to navigate through your photos

I tested all that using  backported version 3.12.2:
apt install gthumb=3:3.12.2-2~bpo11+1 gthumb-data=3:3.12.2-2~bpo11+1

Initially this package was missing "map view" plugin. It looked like it was just forgotten to be enabled by not providing a library during package creation. I asked for a clarification from a guy, who supports this package and Debian and he  fixed that. So we have now version 3:3.12.2-3~bpo11+1

deepin-album

https://github.com/linuxdeepin/deepin-album 

A part of Chinese GNU/Linux distribution called Deepin.
It doesn't appear in Debian 11 Stable, only in Testing.
And, as always, the version there is bit behind to the current stable version from developer. 

When I attempted to see what will be an effect of taking that package from testing, I was amazed by the number of packages that were about to be upgraded in my system. So I'd rather search for some alternative ways to get it installed. Neither flathub,

shotwell 😉

https://wiki.gnome.org/Apps/Shotwell

Looks very promising. It indexes folders you'll tell him to and keeps that index in sqlite file of ~/.local/share/shotwell/data/photo.db
Plus it also generates thumbnails to ~/.cache/showtell

Upon the startup it asked to process all photos from ~/Pictures, and it took it few minutes to process my library. It immediately picked up all the tags and you can browse images by dates / events and tags. 

(+) very nice UI, like iPhoto
(+) read tags from photos, can navigate using tags
(+) automatically create "events" - virtual folders you can use for navigation
(+) reads photos geodata
(+) reads face tags
(+) can create face tags manually (0.30.16+)

(-) there's no map - no way to navigate photos using map
(-) you cannot modify tags and GPS on videos
(-) thumbnails for videos are not shown
(-) it doesn't have built-in video player
(-) doesn't show dates/GPS/tags for video files at all
(-) doesn't have faces recognition
(-) I also noticed it extensively writes back to files with moving "last updated" file attribute. Even if it's not being asked. Like upon the most recent start up, it went and updated a lot of my files, just because it found that these files got some additional tags I set in the other program. To me it's a bug.

It looks like it got some additional features in the latest version but compiling it manually is a pain. Flatpak has version 0.30.16 but it was far from ideal and it was missing to read GPS from some of my photos properly, while the version from Debian repo 0.30.11 did that just fine.
Update: Flatpak got 0.30.17 version, and it's slightly better.

KPhotoAlbum 😖

https://www.kphotoalbum.org/user-support/3minutetour/

(*) The UI is very "nontraditional" to say the least. You forced to navigate your library in a step-by-step manner, by applying a filter on a top of filter, with using some fullscreen "Pick a filter" window. 

(*) The initial indexing process is taking ages and running on all CPU cores. I think guys could have looked up how the same is done in ShotWell. Next runs are faster, but still it takes too long

(*) it shits a lot by leaving its internal directories all across the places. It creates whatever .comments directories everywhere in the folders with pictures. It creates a lot of .hidden directories and files in a root folder you set it to scan. I don't like this kind of abusive behavior inside of my media library

(-) kphotoalbum failed to recognize my custom description EXIF tags
(-) it has no ability to navigate by those EXIF tags
(-) it doesn't write updates you've made to tags down to media files

(?) haven't tested if it can update geodata in media files, but most probably it cannot. Looks like whatever changes you're doing are done outside

(+) it can show photos on map
(+) it has built-in video player
(+) it looks like it has some Face tagging features, but they're not very automatic, so you have to process all your images one by one

Geequi 😏

https://www.geeqie.org/

https://github.com/BestImageViewer/geeqie#features

Quite an advanced application. It's closer Digikam than any other above apps. It has a decent number of configurable options, but UI is full of craziness. Not that KPhotoAlbum craziness, but it's definitely not an Aves or iPhoto.

An interesting note - it looks like it doesn't use any kind of database to index your media. Such approach has both pros and cons. Main con is - when you get into some folder with a lot of items, it takes a while for Geequi to load everything. And you won't be able to find any photos across all your folders.

== version 1:1.6-9+deb11u1 (from debian repo) ==

(+) generates thumbnails for video
(-) doesn't have built-in video player
(+) recognized tags on photos
(-) but doesn't show all of them in a single place, so no tag-based navigation
(+) recognized GPS from photo and shows them on map
(-) you cannot pick a photo / number of photos on map to see them closer. Map navigation is very limited
(-) doesn't recognize dates, tags and GPS from video - raised #1034
(+) you can set/update GPS coordinates for photos
(-) crashes a lot
(-) searching is over-complicated

After playing with it for half an hour I found below bugs:
1) Preferences -> keywords -> search -> fill the list with duplicates
2) Cannot affect the starting folder. App always starts from home folder
3) Pressing "F" to go into fullscreen shows just black screen
4) I do have a sorting applied by "Exif date original". When I enter some folder with large amount of items, for the first time during the session, the very first file gets selected, then all metadata is getting loaded, sorting being applied and the file I was standing on appears somewhere in the random place, not the first one. This is somewhat expected from the logic point of view, but very inconvenient. How it should be done instead is:
-  the file picking pane should be disabled with some refresh animation being shown while all the images are getting loaded
- not to set the focus on the very first item which loads before all metadata is being processed from that folder

I decided to give it a try with more recent version. Flathub has 1.7.3 so I installed it.

== 1.7.3. from flathub ==

(+) crashes less than 1.6.9
(-) missing mini-map for GPS data (1.6.9 had it) - #996 has workaround
(-) missing thumbnails for video, cannot play video
(-) has the same bugs as listed explained above, except of black full screen

== 2.0.1 (latest version on github) ==

I haven't tested the latest version (2.0.1), as they don't publish appImage for aarch64 - raised #1035 for that. Will probably return back to it one day and compile it from sources. But so far I'm moving on. 

Geotagging ☝

https://github.com/jmlich/geotagging

Added it here as a kind of honorable mention. It's not really a photo / video manager, but it's a nice tool to enrich your photos with GPS data, if you had any kind of GPS tracker with you, which was writing a .gpx track file.

Gnome-photos 😡

https://wiki.gnome.org/Apps/Photos
https://gitlab.gnome.org/GNOME/gnome-photos

Current version is 43 and flathub has it, wow! I pay respect to gnome-photos developers to integrate their project properly to flathub. There's some issue with flatpak version of application though, it doesn't work properly: whenever I click to open a folder with pictures nothing happens, like some call to file picking dialog fails. Yes, I know, the developer left quite extensive instructions on how to troubleshoot it, but I see that from the screenshots the app doesn't do any of what we listed as our requirements.

gwenview 😑

https://apps.kde.org/gwenview/

(-) Even though it supports tags, it failed to recognize my tags on photos. The tags I set in Aves are appearing as EXIF / XMP / Subject, but it looks like gvenview is using it's own place to store it's own tags

(-) you cannot use tags for navigation - like to search by them

(-) all current versions are prune to high memory consumption and lead to system crashes. I already reported couple of bugs and they were even fixed, but you need to take the latest DEV version to get those fixes

(-) It doesn't have mini-map, out-of-the-box 

Memories / showmypictures 😡

https://anufrij.org/memories/
https://github.com/artemanufrij/showmypictures

Looks nice, was initially made for Elementary OS, but the project looks completely abandoned. Last update was made in 2018. 

Debian repository doesn't have it, flathub doesn't have it, appimagehub doesn't have it, snapcraft doesn't have it. Even modern Elementary OS releases don't have it. So skipping it. 

Photonic 😒

https://github.com/oferkv/phototonic/

Last update in 2018 :( project is dead

PhotoQT 💔

https://photoqt.org/
https://gitlab.com/lspies/photoqt

Actively developed, claims to support EXIF tags, video and face tagging. Debian repository doesn't have it, but the good thing flathub has it and it's the same latest stable version as being developed on gitlab - so I do pay respect to developers for doing packaging properly. I've been testing version 2.9.1

(*) the app doesn't use any database to store metadata. This is what it has in similar to Geeqie. So be prepared to limitations

(+) very-very nice looking UI, really one of the best
(+) it can do manual face tagging

(-) sorting of photos only by dates from files and not Exif date - #166
(-) it doesn't show all Exif tags, like Exif subject I set in Aves
(-) you cannot modify Exif dates, Exif tags or GPS coordinates
(-) it cannot navigate using Exif tags / nor find images by tags

(-) it doesn't have built-in map. If you click on GPS coordinates in Exif pane it opens up OpenStreetView in a web browser


Bugs:

- when I assign a face tags on a photo, PhotoQT overrides existing EXIF tags I had in a file (EXIF Subject). This is scary shit - #164
- (probably my own local bug) it doesn't play video correctly, only sound was appearing
- small UI bug - #165

QView 💀

https://interversehq.com/qview/

 

Cannot compete at all, as it's just a single file viewer with no surfing capabilities

Photoprism

https://photoprism.app/

xxx

KDE Pix

https://apps.kde.org/pix/

xxx

KGeoTag

https://apps.kde.org/kgeotag/ 

https://kgeotag.kde.org/

xx

Mint PIX 💀

https://github.com/linuxmint/pix

Fork of gThumb for Linux Mint. Some minor things are getting polished, but nothing big

Darktable 💔

https://www.darktable.org/

Debian 11 stable has some outdated version like 3.4.1, so we're installing it from flathub. Good to see it has the most recent version 4.0.1

I played like 20 minutes with it and first impression is - WOW! Compared to all the above applications, it's like super champ. But it has few major flaws, which are making it impossible for us to use it. Not that they're flaws, they're more like principles developers have put into the basic design of it. I'll list them here:

(-) it doesn't support video in any form. Because it's photo editor.
(-) it deliberately not writing any metadata back as EXIF to files. It claims it's non-destructive app, no matter what. So all the changes are written to sidecar XMP files, which is a bullshit. 

Digikam 💓💓💓

https://www.digikam.org/
Source code: https://invent.kde.org/graphics/digikam
Forum: https://forum.kde.org/viewforum.php?f=255
Bugzilla: https://bugs.kde.org/buglist.cgi?list_id=2164853&product=digikam&resolution=---

This is a well known monster I know and love. It can do all the stuff we need.
We might have started with it, but hey, you would never looked at the above software first of all, right?

However, there are few important bits here: 

1) You'll need to make sure to take the version at least >= 7.3, this is where Digikam started to use exiftool. So you'll be able to read/write EXIF tags not only from photos but also from videos. 

2) Don't take the flatpak version until #44 is solved. As you won't be able to use exiftool. See also this

3) Considering #416516, #436876 and #366348 we might need to wait till version 8 (or build it ourselves), as exiv2 is still being used

4) I did my own tests with version 7.9 compiled from source code and whatever tags I was putting to my mp4 video files weren't written back to files by Digikam. Tested the same with last stable version 7.8. Raised #459928
I tried to build version 8.0 (from master) but it was failing.

Overall comparison

app name	tag support	gps support	faces recognition	comments
gThumb	Only for photos	Only for photos. Cannot be used for navigation. Cannot update	No	Good, but no metadata support for video
deepin-album	?	?	?	Wasn't tested as it's hard to get
ShotWell	Only for photos	Partially for photos	Will be in 0.31/32	Very promising but currently lacks video support and geotags based navigation
KPhotoAlbum	No	Yes	Very manual	without proper tag support it's useless for my purposes
Geequi	Partially, for photos. You can't navigate by them	Partially, for photos. You can't navigate by them	Very manual	Without proper video support and navigation it's useless
Geotagging	?	?	?	Wasn't tested
Gnome-photos	?	?	?	Wasn't tested, as flatpak version doesn't work properly
gWenView	Not really	No	No	Useless for us
Memories / showmypictures	?	?	?	Wasn't tested as it seems to be abandoned in 2018. Screenshots are awesome, like iPhoto :)
Photonic	?	?	?	Wasn't tested as it seems to be abandoned in 2018
PhotoQT	Not really	No	Manual tagging with overriding your existing EXIF tags because of bug	Looks nice, but has some critical bugs
QView	tag_support	gps_support	faces_recognition	comments
Photoprism	tag_support	gps_support	faces_recognition	comments
KDE Pix	tag_support	gps_support	faces_recognition	comments
KGeoTag	tag_support	gps_support	faces_recognition	comments
Mint PIX	tag_support	gps_support	faces_recognition	comments
Darktable	Only for photos	Only for photos	?	High quality app but cannot write metadata back to photos and lacks support for any video files at all
DigiKam	Yes	Yes	Yes	The most feature-rich program

Exploring open-source android applications

Today I have just few words for you. They are ...

Aves

URL: https://github.com/deckerst/aves

This is basically the best media gallery you can find for Android. Not only because it's open source, but it's actually one of the most feature-rich application which is being actively developed. What it can do now:

- read GPS data from your photos / videos and display them on map, like Google Photo does. But unlike Google Photo, Aves doesn't leak your data anywhere outside of your phone.  You can then use a map to find what photos you've made in specific places.

- update GPS data (so far only for photos). Typical use case: if you have a bunch of photos shared by your friends via messengers (like WhatsApp), they usually missing GPS data on them. But with Aves you can update them to whatever you need

- assign any custom free-text tags on your photos and search photos by them. Typical example: instead of putting your photos into different directories, you can assign one or multiple tags on them, like "2022 - Trip to Europe" or "Outdoor" and then you'll be able to see all your photos tagged with this tag. It's much more convenient and flexible way of organizing your media library, provided growing support for these tags from desktop applications, like DigiKam.

- fix date & time on your photos

More features are coming:

- objects and face recognition (offline! no google involved)

- assign GPS and tags for videos

Speaking of features, the only closest application I could find was an app named "Piktures" but it's proprietary, paid and it has a idiotic bug which basically prevents you from using free version. I reported it to developers, but they were too lazy to fix it.

LibreTube

URL: https://github.com/libre-tube/LibreTube

Google is known for controversial style of artificially limiting features in their free apps, to unblock them only if you pay some subscription fee. Like YouTube mobile app. Without paying you won't be able to download videos to see them later, when you have no internet connection. Or to listen to videos in a background, with your phone screen being turned off.

But that's not the case, if you're using LibreTube. Not only it offers you these features for free, out of the box, but it also allows you to watch youtube videos without watching annoying ads or being registered at all. You still can have your subscription lists and watch history, but they'll be stored locally, on your phone.

I think it's just awesome!

Speech Recognition and Text-to-Speech using ML. Part 2: going offline

In the first part we have set up a very simple but efficient voice assistant which can listen to your commands and react on them. It starts up as soon as internet and audio appears in system. And yes, this is the limitation of our very first implementation - it needs internet as it uses Google TTS and STT engines.

But this time, we'll try to replace them with something which can be executed 100% offline. And I have couple of candidates, apart from the ones we talked before

Mozilla Deep Speech

The most recent version of DeepSpeech right now is v.0.9.3 and it requires Python 3.7. whereas we're running 3.9 in our Debian. I found it out, when I attempted to install the deepspeech-0.9.3-cp37-cp37m-linux_aarch64.whl and it was giving me this confusing message:

pi@orangepi4-lts:~/projects $ pip3 install https://github.com/mozilla/DeepSpeech/releases/download/v0.9.3/deepspeech-0.9.3-cp37-cp37m-linux_aarch64.whl

ERROR: deepspeech-0.9.3-cp37-cp37m-linux_aarch64.whl is not a supported wheel on this platform.

As I figured it out, I should have checked --verbose flag and then it became more apparent that it attempted to install things under 3.7 directories, whereas current version of python was 3.9. Python is such a mess.  

As Debian repo doesn't have 3.7 I decided to build it myself. It's done pretty straightforward: download -> configure -> make -> make install. I just crosschecked the very first hit I found on google and it went just fine.

To be continued

https://deepspeech.readthedocs.io/en/v0.9.3/

coqui-ai TTS and SST

https://github.com/coqui-ai

https://tts.readthedocs.io/en/latest/

Coqui-Ai TTS - Text to speech

I was playing with TTS (text-to-speech) using my Intel Atom Z8350 tablet - Asus Transformer Book T102HA, as I'm on vacation right now, away from my little SBC device at home. It was almost a smooth ride. Saying "almost", because in Windows 10 it took me a while to understand two main dependencies:

1) you need to have python 3.9, not 3.10

2) you need to install windows 10 sdk and build tools, using "Visual Studio Installer". And that thing brings like 5 additional Gb of various DLLs, header files and tools to your Windows 10. Kind a crazy.

Eventually you can do:

pip3 install tts

And check this out:

tts --text "This is another test using tacotron2"          --out_path speech_tacotron2.wav           --model_name "tts_models/en/ek1/tacotron2"

tts --text "This is another test using tacotron2-DDC"      --out_path speech_tacotron2-DDC.wav       --model_name "tts_models/en/ljspeech/tacotron2-DDC"

tts --text "This is another test using tacotron2-DDC_ph"   --out_path speech_tacotron2-DDC_ph.wav    --model_name "tts_models/en/ljspeech/tacotron2-DDC_ph"

tts --text "This is another test using glow-tts"           --out_path speech_glow.wav                --model_name "tts_models/en/ljspeech/glow-tts"

tts --text "This is another test using speedy-speech"      --out_path speech_speedy.wav              --model_name "tts_models/en/ljspeech/speedy-speech"

tts --text "This is another test using tacotron2-DCA"      --out_path speech_tacotron2.wav           --model_name "tts_models/en/ljspeech/tacotron2-DCA"

tts --text "This is another test using vits"               --out_path speech_vits.wav                --model_name "tts_models/en/ljspeech/vits"

tts --text "This is another test using fast_pitch"         --out_path speech_fast_.wav               --model_name "tts_models/en/ljspeech/fast_pitch"

tts --text "This is another test using vits"               --out_path speech_vits.wav                --model_name "tts_models/en/vctk/vits"

tts --text "This is another test using fast_pitch"         --out_path speech_fast_.wav               --model_name "tts_models/en/vctk/fast_pitch"

tts --text "This is another test using tacotron-DDC"       --out_path speech_tacotron.wav            --model_name "tts_models/en/sam/tacotron-DDC"

tts --text "This is another test using capacitron-t2-c50"  --out_path speech_capacitron-t2-c50.wav   --model_name "tts_models/en/blizzard2013/capacitron-t2-c50"

tts --text "This is another test using capacitron-t2-c150" --out_path speech_capacitron-t2-c150.wav  --model_name "tts_models/en/blizzard2013/capacitron-t2-c150"

This will download a number of different voice and vocoder models (if something fails for you, check the output of tts --list_models command first, and adjust the above) and synthesize the phrase given as --text parameter using them.

You can then go and listen the result, using your favorite media player, or download SoX to use aplay.

Given I was getting very unstable performance figures, like what time does it take to generate the same sentence using different combinations of models/vocoders, I drafted a very simple script to try them all out multiple times, so I'll have a more data to compare - https://github.com/kha84/tts-comparison

I figured it out, that for my hardware the balance between best performing and most accurate  spelling was to use tts_models/en/ljspeech/glow-tts

Tensor Flow TTS

https://github.com/TensorSpeech/TensorFlowTTS

AprirASR TTS

https://github.com/abb128/april-asr

Used in LiveCaptions, a wonderful application for Gnome to generate English captions from the played audio

OpenAI Whisper TTS

https://github.com/openai/whisper

Whisper.CPP TTS

A re-implementation of OpenAI's Whisper but on C++, which is leaner to resources

https://github.com/ggerganov/whisper.cpp

Mimic3 TTS

https://github.com/MycroftAI/mimic3

MaryTTS

http://mary.dfki.de/

Speech Recognition and Text-to-Speech using ML. Part 1: Build your own Alexa / Siri

As you know, one of the project goals is to have your own "Alexa" or "Siri" replacement in our little SBC. A lot of work was already done by a number of talented folks, so we won't shy to just take results of their work and reuse it in our favor.

But at the beginning, let's start with some very basic stuff here. Like making sure we can record and playback audio, then we'll play with couple of entirely offline text-to-speech software. 

Check some basic stuff

arecord / aplay

These two guys worked for me out of the box. The recording is happening on the internal microphone (the quality is so-so), playback via the currently selected device in PulseAudio - in my case through TV connected with HDMI

arecord -d 5 -r 48000 test.wav
aplay test.wav

 

Text-to-speech

espeak

Espeak was around for a long while. It sounds very robotics, no matter what kind of voice you choose.

apt install espeak

espeak "Hello there!

espeak --voices

espeak -v en-gb "Text will be converted to speech."

Festival

Festival is also a long standing player. And it also sounds very robotic, by default, but we can install some additional "language packages" which make it bit better. And there're also commercial packages existing on a market, but they're not as good as their pricetag 

apt install festival festvox-ru festvox-en1 festvox-us1 festvox-us2 festvox-us3 festlex-cmu festlex-poslex

Below commands must be executed under unprivileged user (non-root): 


echo "This is the best as it can get" | festival --tts

echo '(SayText "This is the best as it can get")' | festival --pipe

I can see it clearly, when the software was designed by morons. Speaking of Festival you cannot easily switch between different voice packs, you have to update configs or use interactive mode:

festival
(voice.list)
(voice_us1_mbrola)
(SayText "This is now a different voice")
(voice_us2_mbrola)
(SayText "This is now a different voice")
(voice_us3_mbrola)
(SayText "This is now a different voice")
(voice_en1_mbrola)
(SayText "This is now a different voice")

If you want some voice to become a default one, make a change to /usr/share/festival/siteinit.scm search for a line voice_default

Look this video where a nice guy explaining how to download more voices - https://www.youtube.com/watch?v=Ju_X11JyRSE

I checked the website he mentioned, all voice packs are old as dinosaurs' poo, so to me it doesn't really make sense to spend on it any additional time

pico2wave 

I got tired of those robotic voices. So I just looked for couple of youtube videos how it sounds and wasn't really impressed. So skipping it.

Google text-to-speech engine

If you're on Debian - don't install it from repo. The version from repo is outdated and doesn't work anymore. Install it via pip, it will take much freshier version:

pip install gTTS
 

Let's play with it now:

~/.local/bin/gtts-cli 'This is the test from Google text to speech services' --output /tmp/test.mp3
ffplay -nodisp -autoexit /tmp/test.mp3

Well, the quality is just increadable. I'll probably go online :) but need to understand, this Google can close this backdoor at anytime, so if we're building something we don't want to support all the time, we'll need to have a fallback option, line one of the above. So far I just made a simple and stupid wrapper, we'll extend it later on, if needed, to analyze exit code from gtts-cli and switch to something different, like espeak:

cat /usr/local/bin/sayit
#!/bin/sh
if [ -z "$1" ]
then
  echo "Error: no input parameter is given"
  exit 0
fi
tfile="$(mktemp /tmp/tts.XXXXXXXXX.mp3)" || exit 1
~pi/.local/bin/gtts-cli "$1" --output $tfile
ffplay -nodisp -autoexit $tfile
rm $tfile

Speech recognition

Let's start with something simple like this project - https://github.com/ProgrammingHero1/romantic-alexa
 

I've made a small fork of it - https://github.com/kha84/pgttsjaisrar - with fixing some obvious issues and swapped the ugly text-to-speech python lib with our lovely above shell file, made it to run as systemd service and, hell, it works just well out of box. 

Stay tuned for the part 2, where we going to make it to work offline - https://orange-pi-4-lts.blogspot.com/2022/09/speech-recognition-and-text-to-speech_3.html

"Debloating" your Debian Linux even further

Why we need to do that?

Well, there're two main reasons for that. One is kinda important for everyone, who're running their SBCs out of SD cards, other is my own deep personal preference. 

 

Reason 1: save your SD card / EMMC chip from dying earlier

We're running Debian Linux, which was put together while keeping in mind desktop machines with HDD or SSD drives. What we have in our SBC are eMMC (at best) or SD card. 

 

The main difference between desktop-grade SSD disk and SD card is that SSD disk' controller is much more advanced. It protects the storage cells by distributing write operations evenly. SD card controllers also do that kind of thing, but not that good, because of obvious reasons (cost saving + smaller form factor).

And you need to know, the more you write to SD card, the less it lasts. So let's start. The idea of this exercise is to turn off as much write-heavy activities as we can while leaving system up and running. 

 

Reason 2: I don't like a lot of automation, which Linux offers out of the box

In this sense, Linux is becoming a sort of mini-me of Windows,  I hate so much. I can hardly stay calm when I see some windows service is eating up a lot of CPU / memory / doing some IO or sending something over the netwrok  and I can't even get to know what that service is doing.

- if it's doing something crucial for OS to live, and killing it will break Windows?
- is it something auxiliary  Windows can live without?
- does it mine some cryptocoins for Microsoft' benefit, but on my own hardware?

... because Windows is closed source, and you cannot get inside of it to see which of the above is true.

I also deeply hate Windows, when it decides it's time to apply some patches,  perform some kind of housekeeping for whatever built-in stuff or do a system reboot to make a system upgrade. Without asking me.

As I said, unfortunately in Linux (especially in Debian linux) I see it's drifting towards Windows behavior, and nobody cares about it. See the proof#1 and proof#2.

But fortunately, this is still Linux, and it's open soruce with tonns of documentation and questions being already asked and answered. So we can finetune it whatever we want, and do that knowingly, without any risk to get something broken in the completely distant part of the system. This is why I love Linux so much.

Disclaimer

 

A word of warning here. I'm going to turn off a lot of stuff here. You might hear it from others (probably they will even be screaming at you) that all the things I suggested to turn off here is crucial for your own existence. 

 

Make a pause there and take a deep breath. Think of the worst possible scenario. Like some critical security bug was found in Debian and given you turned off automatic updates,  your system still has that bug and potentially vulnerable, if ..

 

• If you have your SBC connected directly to Internet with all ports exposed and you're running tons of software from 3rd parties / non official repos / snap which are exposing itslef by opening these ports to outer world, inviting hackers to come in
  
  
• if you're using web browser of old version 

... this might be an issue, yeah. But, if you'll be doing apt update & apt upgrade time-to-timeyourself, it's nothing different to have this automatic update services running.

Remove cron jobs

 

Let's see what we have:

 

root@orangepi4-lts:~# ls /etc/cron.*
/etc/cron.d:
e2scrub_all  orangepi-truncate-logs  orangepi-updates  sysstat

/etc/cron.daily:
apt-compat  cracklib-runtime  locate     man-db                samba
aptitude    dpkg              logrotate  orangepi-ram-logging  sysstat

/etc/cron.hourly:
fake-hwclock

/etc/cron.monthly:

/etc/cron.weekly:
man-db  tor

 

A lot of things from there were moved to systemd timers, we'll deal with them just a bit later.

 

/etc/cron.d/e2scrub_all performs a check for ext2-4 file systems and marks corrupted filesystem with a tag, so fsck will fix that on next mount. Practically on next reboot. Leave that so far, but its existence is very questionable, given the actual fix could happen not earlier than upon next reboot.

TODO: I want to see it myself, how can I get that tag/flag value as a result of e2scrub_all working. And probably get notified about that, rather than fsck will be silently fixing some issue

/etc/cron.d/orangepi-truncate-logs runs a shell script (/usr/lib/orangepi/orangepi-truncate-logs) every 15 minutes to truncate lot of logs. Looks good. But it somehow clashes with log2ram (described below). I'm gonna leave it so far.

/etc/cron.d/orangepi-updates calls shell script /usr/lib/orangepi/orangepi-apt-updates on a daily basis and after reboots to install updates silently. Part of orangepi-bsp-cli-orangepi4-lts package. Remove that file Some smartalek from OrangePi decided he knows it better than me.

/etc/cron.d/sysstat data collector for sysstat. Sysstat utilities are a collection of performance monitoring tools for Linux. These include sar, sadf, mpstat, iostat, tapestat, pidstat, cifsiostat and sa tools.
The cronjob is just a misery. It runs pretty much all the time /usr/lib/sysstat/debian-sa1 script, but that script exits if systemd is there - because sysstat was moved to systemd already. I've deleted that /etc/cron.d/sysstat
TODO: it's worth investigating where the raw statistics being put by systemd version of this grabber, to move that place to zram

/etc/cron.daily/apt-compat doesn't run if systemd is in place. Ho harm but I deleted that

/etc/cron.daily/aptitude a script that saves package states to a log file. Another logging/backup courtesy I never asked for. Deleting that

/etc/cron.daily/cracklib-runtime - part of cracklib-runtime for lame people who're using dictionary passwords. But we're not like them, right? Then purging the whole cracklib-runtime and libcrack2 packages together.

/etc/cron.daily/dpkg - part of dpkg. Backups the metadata from dpkg "database" about installed packages. Like if it could do something about it, if it finds something is broken. Hehe. Added "exit 0" to the top to disable it

/etc/cron.daily/locate - daily update to locate database. drop that. If you cannot find something with locate, and you're sure you have updated its index with updatedb recently and didn't change a bit since then - then the file is not there. No need to hammer your filesystem on a daily basis.

/etc/cron.daily/logrotate - is not doing anything when systemd is there. Can stay

/etc/cron.daily/man-db - also not doing anything when systemd is there (I'm assuming there's an appropriate timer, we'll look at them bit later).  Can stay

Looks like a lot of Debian packages are made with having this idea in mind that users might want to switch from systemd to initd, so all these crontabs will start to matter again.

/etc/cron.daily/orangepi-ram-logging - not doing anything. logrotate.timer is doing all the job instead. Can stay

/etc/cron.daily/samba - if you remember, we needed samba for our EmulationStation, so we could copy roms from our Windows machine with using its File Explorer. This job makes a backup of /etc/samba/smbpasswd file to /var/backup if it has changed. Really, the guy who wrote it is just a genius of proper backup strategy. Joking, he's not. Remove the job.

/etc/cron.daily/sysstat - doesn't do anything. All job was moved to systemd timers. Can stay

/etc/cron.weekly/man-db - not being executed with systemd. Can stay

 

Remove/disable unneeded systemd timers

As we seen above, a lot of things were migrated from cron to systemd timers. Cronned scripts are just silently existing if they see the systemd is there. So it's time to deal with systemd timers to see what we can safely disable:

 

systemctl list-timers --all

 

NEXT                        LEFT          LAST                        PASSED       UNIT                         ACTIVATES
Wed 2022-08-31 22:10:00 MSK 5min left     Wed 2022-08-31 22:00:59 MSK 3min 6s ago  sysstat-collect.timer        sysstat-collect.service
Thu 2022-09-01 00:00:00 MSK 1h 55min left Wed 2022-08-31 00:00:02 MSK 22h ago      logrotate.timer              logrotate.service
Thu 2022-09-01 00:00:00 MSK 1h 55min left Wed 2022-08-31 00:00:02 MSK 22h ago      man-db.timer                 man-db.service
Thu 2022-09-01 00:07:00 MSK 2h 2min left  n/a                         n/a          sysstat-summary.timer        sysstat-summary.service
Thu 2022-09-01 00:44:45 MSK 2h 40min left Wed 2022-08-31 00:44:45 MSK 21h ago      systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Thu 2022-09-01 06:44:30 MSK 8h left       Wed 2022-08-31 06:22:59 MSK 15h ago      apt-daily-upgrade.timer      apt-daily-upgrade.service
Thu 2022-09-01 17:18:53 MSK 19h left      Wed 2022-08-31 20:13:53 MSK 1h 50min ago apt-daily.timer              apt-daily.service
Sun 2022-09-04 03:10:13 MSK 3 days left   Sun 2022-08-28 03:10:50 MSK 3 days ago   e2scrub_all.timer            e2scrub_all.service
Mon 2022-09-05 01:26:35 MSK 4 days left   Mon 2022-08-29 00:23:07 MSK 2 days ago   fstrim.timer                 fstrim.service

 

Same bloatware. Let's get it cleaned:

xxxx

xxxx

xxxx

xxxx

xxxx

xxxx

xxxx

xxxx

xxxx

xxxx

xxxx

Monitor what remaining processes are writing heavily to SD card and deal with it, one-by-one

For that we're going to use some very nice software Linux can offer us for free. Here's the picture for attracting your attention:


What we can use here to accomplish our task:

• "raw" tools like pcstat, pidstat, iostat, iotop and blkstat
  
  
• we can attempt to find any higher-level tools with using apt-rdepends -r [rawpackage])
  
  
• some random tools we found on internet (like fatrace and csysdig). 

So let's start rolling on with simple things

fatrace

I really fell in love with that tool. In the call below I asked it:

• to look after only specific events (-f W to monitor writes to files)
• to limit the scope to consider only one mounted device, from the current directory (-c), so I first walked into /
  
• to add timestamps to its log (-t)

I left it running for a while, and then I examined the resulted log file.

cd / ; fatrace -c -t -f W | tee /tmp/fatrace.log | tee -a /tmp/fatrace.log

 

The only unfortunate thing about fatrace is that it does not provide you a bit of info about amounts of data being written and you cannot overcome that, because the system interfaces it's using are not giving these figures either.


In my case I see it already what processess were constantly writing their shit very important data to my precious SD card. Mainly it was Firefox and down below I'll to teach it how not to update its bolloks sqlite database files inside of ~/.mozilla/firefox/ 

The problem however is not only about Firefox alone. All the rest browsers I tested were doing the very same thing. But don't you worry. We'll deal with them as well.

Thanks to fatrace I also discoverd some some vnstat daemon was collecing networking statistics and putting that to its own file to /var/lib/
What was that idiot, who installed that? Was it me? Given no other package was depending on that vnstat, apart from its own mini-me vnstati. Go to hell, both of you:

apt purge vnstat

iotop

On a contrast to previous tool, now you can see amounts of data written, but you cannot limit it to see only on a specific partition (like the / - mounted to SD card). So applicability of this tool is limited to see only amounts of data written, but not knowing where exactly that data was written to. Here are few examples how can you run it:


iotop -bktoqqq

In this mode I found iotop has a bug - it was not capturing write events from short-lived processes, like the process of screenshot creation. So it might miss some important share of IO load from such processess.



iotop -obPat 

 

Same story here. Firefox is winning the race by pushing few Mb of its shitty cache to SD card within a matter of few minutes. Firefox, I hate you! Why you're doing that? I have plenty, you hear it, plenty of RAM free. You're ruining my precious SD card.


iostat 

Every tool is using its own unique way of tracking IOs. So from iostat you can expect IO breakdown per disk. If you run it like this:


iostat -dzphN 10

... it will first show you the accumulated report since the systemboot (on a screenshot it's appearing on the very top) and then will be showing deltas every 10 seconds: 


 


log2ram + zram

If you used distro from Rearm.IT, everything should be already configured. Just check that you do have /dev/zram devices and below filesystems are mounted from it:

 

root@orangepi4-lts:~# mount | grep zram
/dev/zram1 on /var/log type ext4 (rw,relatime,discard)
root@orangepi4-lts:~# df
Filesystem     1K-blocks     Used Available Use% Mounted on
udev             1904036        8   1904028   1% /dev
tmpfs             395600     2932    392668   1% /run
/dev/mmcblk2p1 122600084 86461672  34849544  72% /
tmpfs            1978000        0   1978000   0% /dev/shm
tmpfs               5120        8      5112   1% /run/lock
tmpfs            1978000       16   1977984   1% /tmp
/dev/zram1         49560    19796     26180  44% /var/log
tmpfs             395600       44    395556   1% /run/user/1000
 

In my case I can see that /var/tmp is not using zram so I need to fix it. Edit /etc/fstab and make sure you have these lines:

tmpfs /tmp tmpfs defaults,nosuid 0 0
tmpfs /var/tmp tmpfs size=10M,nosuid 0 0
tmpfs /var/cache/samba tmpfs size=5M,nosuid 0 0
 

If you modified any of those (or added missing ones, like I did), run mount -a to remount everything without the need of rebooting.

If you're using some other Linux distro, read how to configure log2ram here - https://ikarus.sg/extend-sd-card-lifespan-with-log2ram/

disable swap (makes sense for 4Gb+ RAM and higher models) or make sure swap is using zram

Even though I see in my Linux distro the swap is already mounted from zram0, to me it doesn't make much sense to have it like that. The only positive thing about it, is that zram uses a compression methods, so if your system will be swapping, the swap will be same in memory, but compressed. In the ohter hand:

• I hardly seen my system was running out of memory. It was just once, due to the bug in Gwenview. But guys, I'm having luxurious 4 Gb of RAM
  
  
• Using compression mechanisms on zram will definitely hurt CPU, when it comes to moving something to SWAP or reading it back
  

Here's how I have it:
 

root@orangepi4-lts:~# swapon
NAME       TYPE      SIZE USED PRIO
/dev/zram0 partition 1.9G 416M    5
 

I decided to keep this swap on zram so far, given it's still using memory and not the SD card. 

some further steps

https://raspberrypi.stackexchange.com/questions/169/how-can-i-extend-the-life-of-my-sd-card

Disable system.d services we don't need

Disclaimer: you really should know what you're doing and look into more details of what you're exactly disabling

In my case, I don't want any "automated" or "unattended" software upgrades / updates to happen, I don't want my computer to do something I can do myself. So I want all that disabled or even removed. 

 

Let's see what exactly we have running (I'm skipping some output from systemctl for the services I do want and I know what they're doing):

$ systemctl status

             ├─nfs-mountd.service
             │ └─119304 /usr/sbin/rpc.mountd --manage-gids

             ├─nfs-blkmap.service
             │ └─119752 /usr/sbin/blkmapd

             ├─nfs-idmapd.service
             │ └─119303 /usr/sbin/rpc.idmapd

             ├─packagekit.service
             │ └─16489 /usr/libexec/packagekitd

             ├─unattended-upgrades.service
             │ └─1123 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown

             ├─upower.service
             │ └─2296 /usr/libexec/upowerd

             ├─accounts-daemon.service
             │ └─2072 /usr/libexec/accounts-daemon

             ├─haveged.service
             │ └─745 /usr/sbin/haveged --Foreground --verbose=1

nfs-*.service

who needs nfs nowdays? drop that!

sudo nala remove nfs-common

packagekit.service

pi@orangepi4-lts:~ $ dpkg -S /usr/libexec/packagekitd
packagekit: /usr/libexec/packagekitd
pi@orangepi4-lts:~ $ apt info packagekit
Package: packagekit
Version: 1.2.2-2
Priority: optional
Section: admin
Maintainer: Matthias Klumpp <mak@debian.org>
Installed-Size: 2,857 kB
Depends: libglib2.0-bin, policykit-1, init-system-helpers (>= 1.52), libappstream4 (>= 0.10.0), libapt-pkg6.0 (>= 1.9.2), libc6 (>= 2.28), libgcc-s1 (>= 3.0), libglib2.0-0 (>= 2.54), libgstreamer1.0-0 (>= 1.0.0), libpackagekit-glib2-18 (>= 1.2.1), libpolkit-gobject-1-0 (>= 0.99), libsqlite3-0 (>= 3.5.9), libstdc++6 (>= 5.2), libsystemd0 (>= 214)
Recommends: packagekit-tools, systemd
Suggests: appstream
Breaks: libpackagekit-glib2-14 (<= 0.7.6-4), libpackagekit-qt2-2 (<= 0.7.6-4), packagekit-backend-apt (<< 1.0), packagekit-backend-aptcc (<< 1.0), packagekit-backend-smart (<< 1.0), packagekit-offline-update (<< 1.0), packagekit-plugin-click (<= 0.3.1), plymouth (<< 0.9.5)
Homepage: https://www.freedesktop.org/software/PackageKit/
Tag: admin::package-management, implemented-in::c, implemented-in::python,
 role::program
Download-Size: 575 kB
APT-Manual-Installed: no
APT-Sources: http://deb.debian.org/debian bullseye/main arm64 Packages
Description: Provides a package management service

This is the abstraction layer which makes it possible for applications like KDE Discover to work on any kind of distros, no matter what software packaging tools they're using - dpkg/apt, rpm/yum/dnf, pacman - whatever. 

If you want to keep using KDE Discover you'll need to keep that. I disabled it.

unattended-upgrades.service

pi@orangepi4-lts:~ $ dpkg -S /usr/share/unattended-upgrades/unattended-upgrade-shutdown
unattended-upgrades: /usr/share/unattended-upgrades/unattended-upgrade-shutdown
pi@orangepi4-lts:~ $ apt info unattended-upgrades
Package: unattended-upgrades
Version: 2.8
Priority: optional
Section: admin
Maintainer: Michael Vogt <mvo@debian.org>
Installed-Size: 334 kB
Depends: debconf (>= 0.5) | debconf-2.0, debconf, python3, python3-apt (>= 1.9.6~), python3-dbus, python3-distro-info, ucf, lsb-release, lsb-base, xz-utils
Recommends: systemd-sysv | cron | cron-daemon | anacron
Suggests: bsd-mailx, default-mta | mail-transport-agent, needrestart, powermgmt-base, python3-gi
Tag: admin::package-management, implemented-in::python, role::program,
 suite::debian, works-with::software:package
Download-Size: 88.6 kB
APT-Manual-Installed: yes
APT-Sources: http://deb.debian.org/debian bullseye/main arm64 Packages
Description: automatic installation of security upgrades 

Self explanatory. Remove that. I'll be able to install all my security updates myself!

apt purge unattended-upgrades

upower.service

pi@orangepi4-lts:~ $ dpkg -S /usr/libexec/upowerd
upower: /usr/libexec/upowerd
pi@orangepi4-lts:~ $ apt show upower
Package: upower
Version: 0.99.11-2
Priority: optional
Section: admin
Maintainer: Utopia Maintenance Team <pkg-utopia-maintainers@lists.alioth.debian.org>
Installed-Size: 420 kB
Depends: dbus, udev, libc6 (>= 2.17), libglib2.0-0 (>= 2.41.1), libgudev-1.0-0 (>= 147), libimobiledevice6 (>= 0.9.7), libplist3 (>= 1.11), libupower-glib3 (>= 0.99.8), libusb-1.0-0 (>= 2:1.0.8)
Recommends: policykit-1
Homepage: https://upower.freedesktop.org/
Tag: admin::power-management, hardware::power, hardware::power:acpi,
 hardware::power:ups, implemented-in::c, interface::daemon,
 role::program
Download-Size: 113 kB
APT-Manual-Installed: no
APT-Sources: http://deb.debian.org/debian bullseye/main arm64 Packages
Description: abstraction for power management

This service provides a various information about electrical power for your PC and linked devices, like remaining charge of a battery of your laptop or bluetooth mouse. I tried to remove it, but it also removes so many things with it (like sddm), so unfortunately you have to keep that beast. In my case, this service doesn't provide a correct information of remaining battery from connected bluethooth gamepads:

 

upower -d

Will raise a bug about this issue.

haveged.service

pi@orangepi4-lts:~ $ dpkg -S /usr/sbin/haveged
haveged: /usr/sbin/haveged
pi@orangepi4-lts:~ $ apt info haveged
Package: haveged
Version: 1.9.14-1
Priority: optional
Section: misc
Maintainer: Jérémy Bobbio <lunar@debian.org>
Installed-Size: 92.2 kB
Pre-Depends: init-system-helpers (>= 1.54~)
Depends: lsb-base (>= 3.2-14), libc6 (>= 2.17), libhavege2 (>= 1.9.13)
Suggests: apparmor
Homepage: https://issihosts.com/haveged/
Tag: implemented-in::c, interface::daemon, role::program, scope::utility,
 security::cryptography
Download-Size: 39.1 kB
APT-Manual-Installed: yes
APT-Sources: http://deb.debian.org/debian bullseye/main arm64 Packages
Description: Linux entropy source using the HAVEGE algorithm
 

Random number generation daemon. I'm not joking. And it's important part of distribution, as a lot of crypto things are depending on having a truely random number being generated. Don't touch it. I eats just 3 megs of ram but it provides a truly randomization for /dev/random

accounts-daemon.service

Coming soon

configure X to not generate huge .xsession-errors file (or move that file to /tmp

If you were following me with all the above steps, you might have noticed (with the help of fatrace) that a lot of stuff is written to ~/.xsession-errors file. This is how X.Org is configured by default in  /etc/X11/Xsession file:

...

ERRFILE=$HOME/.xsession-errors
...

# attempt to create an error file; abort if we cannot
if (umask 077 && touch "$ERRFILE") 2> /dev/null && [ -w "$ERRFILE" ] &&
  [ ! -L "$ERRFILE" ]; then
  chmod 600 "$ERRFILE"
elif ERRFILE=$(tempfile 2> /dev/null); then
  if ! ln -sf "$ERRFILE" "${TMPDIR:=/tmp}/xsession-$USER"; then
    message "warning: unable to symlink \"$TMPDIR/xsession-$USER\" to" \
             "\"$ERRFILE\"; look for session log/errors in" \
             "\"$TMPDIR/xsession-$USER\"."
  fi
else
  errormsg "unable to create X session log/error file; aborting."
fi

exec >>"$ERRFILE" 2>&1

 

Given our homedirs are on the SD card, we don't want these permanent writes being made to it with such logs. Let's fix that by changing that file to a symlink pointing somewhere to /tmp.  Tempdir is mounted as tmpfs (essentially - to memory) so we will avoid burden of constant writes to SD:

mv ~/.xsession-errors ~/.xsession-errors.bak
ln -s /tmp/$USER.xsession-errors ~/.xsession-errors

Another option will be to configure X to write only critical errors, but I'm fine with my current option now.

Disable journaling for ext4 filesystems

We're not running production server of a patients-life-critical application in hospital. If we loose a bit of info or some app will corrupt it's cache, if our SBC will be unexpectedly poweroff, we can survive that. We don't have such apps, who won't survive if. So let's go:



tune2fs -l /dev/mmcblk2p1
tune2fs -O ^has_journal /dev/mmcblk2p1
e2fsck -f /dev/mmcblk2p1
reboot

Configure your browsers to not write its cache that aggressively to your homedir

Same as above - as we figured it, browsers tend to write a lot of things to your ~/.cache or ~/.config or ~/.mozila or whatever else in your home dir. Some of the stuff we probably want to be written there, like cookies. But most of other stuff you'll see is written there is just lazy browser developers or plugin who didn't pay enough attention to such important details.

Firefox

In your address string put "about:config" and hit enter. Accept the warning and proceed. We need to modify these setitngs:

browser.cache.disk.enable = false

browser.cache.disk.smart_size.enabled = false
browser.cache.disk_cache_ssl = false

+ I'll need to search for more, as it still writes to number of its internal sqlite files like:

pi@orangepi4-lts:~ $ grep -oE "\/home[^ ]*" /tmp/fatrace.log | sort | uniq -c
     10 /home/pi/.mozilla/firefox/(x).default-esr/AlternateServices.txt
      1 /home/pi/.mozilla/firefox/(x).default-esr/broadcast-listeners.json
      1 /home/pi/.mozilla/firefox/(x).default-esr/broadcast-listeners.json.tmp
    325 /home/pi/.mozilla/firefox/(x).default-esr/cookies.sqlite
   2366 /home/pi/.mozilla/firefox/(x).default-esr/cookies.sqlite-wal
      4 /home/pi/.mozilla/firefox/(x).default-esr/datareporting/aborted-session-ping
      4 /home/pi/.mozilla/firefox/(x).default-esr/datareporting/aborted-session-ping.tmp
      1 /home/pi/.mozilla/firefox/(x).default-esr/datareporting/archived/2022-09/xxxmain.jsonlz4
      1 /home/pi/.mozilla/firefox/(x).default-esr/datareporting/archived/2022-09/xxxmain.jsonlz4.tmp
      1 /home/pi/.mozilla/firefox/(x).default-esr/datareporting/session-state.json
      1 /home/pi/.mozilla/firefox/(x).default-esr/datareporting/session-state.json.tmp
      5 /home/pi/.mozilla/firefox/(x).default-esr/favicons.sqlite
    123 /home/pi/.mozilla/firefox/(x).default-esr/favicons.sqlite-wal
     28 /home/pi/.mozilla/firefox/(x).default-esr/formhistory.sqlite
    122 /home/pi/.mozilla/firefox/(x).default-esr/formhistory.sqlite-journal
     50 /home/pi/.mozilla/firefox/(x).default-esr/permissions.sqlite
    135 /home/pi/.mozilla/firefox/(x).default-esr/permissions.sqlite-journal
     54 /home/pi/.mozilla/firefox/(x).default-esr/places.sqlite
    420 /home/pi/.mozilla/firefox/(x).default-esr/places.sqlite-wal
     15 /home/pi/.mozilla/firefox/(x).default-esr/prefs-1.js
      3 /home/pi/.mozilla/firefox/(x).default-esr/prefs.js
      3 /home/pi/.mozilla/firefox/(x).default-esr/protections.sqlite
      9 /home/pi/.mozilla/firefox/(x).default-esr/protections.sqlite-journal
      1 /home/pi/.mozilla/firefox/(x).default-esr/sessionstore-backups/recovery.jsonlz4
    121 /home/pi/.mozilla/firefox/(x).default-esr/sessionstore-backups/recovery.jsonlz4.tmp
      8 /home/pi/.mozilla/firefox/(x).default-esr/SiteSecurityServiceState.txt
     22 /home/pi/.mozilla/firefox/(x).default-esr/storage/default/moz-extensionxxx/idb/xxx-eengsairo.sqlite
     18 /home/pi/.mozilla/firefox/(x).default-esr/storage/default/moz-extensionxxx/idb/xxx-eengsairo.sqlite-wal
      4 /home/pi/.mozilla/firefox/(x).default-esr/storage/permanent/chrome/idb/xxxxAmcateirvtiSty.sqlite
      4 /home/pi/.mozilla/firefox/(x).default-esr/storage/permanent/chrome/idb/xxxxAmcateirvtiSty.sqlite-wal
      6 /home/pi/.mozilla/firefox/(x).default-esr/storage/permanent/chrome/idb/xxxxrsegmnoittet-es.sqlite
      5 /home/pi/.mozilla/firefox/(x).default-esr/storage/permanent/chrome/idb/xxxxrsegmnoittet-es.sqlite-wal
      4 /home/pi/.mozilla/firefox/(x).default-esr/webappsstore.sqlite
     42 /home/pi/.mozilla/firefox/(x).default-esr/webappsstore.sqlite-wal
      8 /home/pi/.mozilla/firefox/(x).default-esr/xulstore.json
      6 /home/pi/.mozilla/firefox/(x).default-esr/xulstore.json.tmp

Might need to raise a bug about this
https://bugzilla.mozilla.org/buglist.cgi?query_format=specific&order=relevance+desc&bug_status=__all__&product=&content=places.sqlite-wal&comments=0

Qutebrowser

tbc

Chromium

See point 4 here - https://raspberrypi.stackexchange.com/a/62549/148860

Mesuring our success

 

After all the above fixes being in place, we can run iostat again to see the accumulated IO stats from the very last boot. Behold, this is my system after 13 hours uptime:

Every 2.0s: iostat -dzphN ; uptime                                                                                                                      orangepi4-lts: Fri Sep  2 13:21:31 2022

Linux 5.18.5-rk3399 (orangepi4-lts)     09/02/2022      _aarch64_       (6 CPU)

      tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd Device
     0.01         0.2k         0.0k         0.0k       5.3M       0.0k       0.0k mmcblk0
     0.00         0.1k         0.0k         0.0k       3.2M       0.0k       0.0k mmcblk0p1
     0.00         0.0k         0.0k         0.0k     348.0k       0.0k       0.0k mmcblk0boot0
     0.00         0.0k         0.0k         0.0k     348.0k       0.0k       0.0k mmcblk0boot1
     0.50        14.7k         0.9k         0.0k     499.5M      31.4M       0.0k mmcblk2
     0.50        14.6k         0.9k         0.0k     497.5M      31.4M       0.0k mmcblk2p1
     0.02         0.1k         0.0k         0.0k       2.3M       4.0k       0.0k zram0
     0.14         0.0k         1.5k         0.0k     476.0k      51.3M       0.0k zram1


 13:21:31 up  9:41,  2 users,  load average: 1.16, 1.24, 1.22
 

Just 30 Mb of writes for 13 hours and I do have a lot of daemons runnig. Launch of any web browser and few minutes serfing still adds to this picture like +30 Mb of data being written to SD, so this is something we still have to handle. But now it's way better than it was before.