How to sync your photos from android over internet to your computer

I used to use Google Photos. And Apple photos. And Flikr. And God knows what other services. And I paid. Paid a monthly subscription for all of them. But then, I decided - what a hell!

So I started to bring all my photos back offline. It took me a while, but thanks to Google, Apple and Flickr - they  offer this kind of option to bulk download all your digital assets.

So now I have an external 2Tb HDD + I bought an additional 1 Tb SSD to my main PC and have all my archive of photos there.

It's still unsorted, or should I say partially sorted, with lots of duplicates, because during last years I was switching from iPhone to Android (and even back for some short time, while my Android phone was in the repair shop). And I still own both iPhone as a backup phone and lovely iPad Mini.

So what I wanted to have is a Google Photo replacement free download no adds no registration no monthly subscription how many SEO specialists you need to replace a lightbulb.

I drafted some major requirements of how I wanted it to work:

Over the air backups from phone(s) to computer

I want my photos to be backed up from my phone (both Android and iPhone) over the air securely to some location, so in case if my phone breaks I won't care about lost photos

Two-way sync

I want a two-way sync, i.e. if I delete some photos from my phone I want them to disappear from all my other devices - computer, tablets, whatever is configured. Or if I have some spare time to sort out photos on my computer, and I delete something, I want that to be deleted on phone as well.

This requirement is bit controversial, if it's not implemented properly. It means that I have to have a full copy of all my photos on my all devices - phone, iPad, computer. Otherwise if something will be missing in one place, this gap should be processed by sync process:

- either to delete the same files on other devices
- or to bring them back to this device 

Having full copy of all photos is not a problem for SBC, as we can attach bigger SD card or even multi-Tb HDD to it. But it is a problem for mobile devices, which are limited in their storage.

To overcome this negative side effect we will be syncing just last year worth of photos. I think every mobile device can handle that amount of photos. It means that if you'll be running out of space on your phone, you can easily delete photos and videos older than year ago, and this your deletion will not be propagated to other devices. 

Jumping ahead, this is where you can do this in FolderSync:

Implementation

This was quite easy. On my Orange PI 4 LTS (can't stop showing off ;-) I configured sshd. I also have tor there configured to expose 'tor hidden service' so I can SSH from my phone to my SBC from any remote point of the world

Then on my Android device I also have tor installed. The app is called "Orbot" and it's official Tor client. It's a bit  laggish but generally works fine.

Then I installed an app called "Folder Sync". If you know about any open-source android app like that - let me know.  Folder Sync has a free version with adds and it works quite well for our purposes. I have configured Folder Sync to connect to my tohostnamef8j28jfh9jfjh7sdhf2.onion:2222 and sync couple of Android folders (with camera pictures and screenshots) to appropriate folders on my SBC. It works just great, but slow.

When I am at home and my phone can use home WiFi  I also configured few other sync pairs with the same folders, thanks to FolderSync flexibility, but this time they will only trigger when my phone sees my WiFi SSID - so when I'm at home it reaches to my server without Tor. This thing is optional, and I did that just to check how flexible FolderSync is. You can skip this. 

TODO

If the connection speed over the internet is not enough (because of using Tor),  I probably have to back off and configure a tunnel using Cloudflare.

Setup and configure few web services to act as online gallery. Candidates are:

• Photoprism
• Piwigo
• Lychee
  
• Plex
• Librephotos
• Nextcloud

Some of them can even do the sync.

Quick guide to Tor

Installation

The apt takes care of everything, including creation of proper systemd units and starting the service up. The only additional thing we install on a top of tor is torify which makes it super easy to wrap all the network traffic from any binary to be tunneled into SOCKS5 proxy created by tor: 

apt install tor torify

Using torify wrapper

torify curl icanhazip.com
torify lynx rutracker.org
torify --shell

Configure Tor as https proxy (skip that)

In some rare cases you might want to use HTTP proxy instead of SOCKS5.
Tor can create you a local one out of the box for you. For that modify tor config and restart the service:

echo "HTTPTunnelPort 9080" >> /etc/tor/torrc
service tor restart

Be careful while using it, it's not "normal" HTTP proxy, as doc is suggesting it's HTTP CONNECT type of proxy. Don't know exactly what that is, need to google that, but my simple test worked just fine:

https_proxy=localhost:9080 curl https://icanhazip.com

Configure SOCKS5 proxy in KDE

For some weird reason Chromium is too limited in terms of how can you configure it from UI side, proxy-wise. Like in my case it can only "nest" proxy settings from the system - in my case from KDE systemsettings. 

1) open "System Settings" KDE app or run it /usr/bin/systemsettings5

2) navigate yourself to "proxy" configuration. Just type "proxy" in a top left search

3) select "Use manually specified proxy configuration option"

4) leave all the fields blank apart from "SOCKS5 proxy", where you put localhost port 9050

5) if you want that proxy (tor) to be used only for specific domains, specify them in the "Exceptions" list, comma separated and tick the checkbox "Use proxy settings only for addresses in the Exception list". If you want to include some domain with all its subdomains you'll need to precede it with dot: ".domain.com"
6) hit apply it should be picked up immediately by Chromium - just go there and test, no need for relaunch 

Create another application link for Chromium to run it under Tor SOCKS5 proxy

As an alternative option  I have just created another "application link" for Chromium but this time it starts the browser with CLI option to use SOCKS5 for everything:

root@orangepi4-lts:/usr/share/applications# diff chromium-tor.desktop chromium.desktop  
3c3
< Name=Chromium over Tor
---
> Name=Chromium Web Browser
131c131
< Exec=/usr/bin/chromium --proxy-server="socks5://localhost:9050" --user-data-dir=/tmp %U

---
> Exec=/usr/bin/chromium %U

And that's it. Now if I need to open some website which is blocked in my country I just open that "Chromium over tor" link. 

If you gonna make any further modifications to this .desktop file, make sure to delete the app from "favorites" and add it again. The trick is, when you add the app to "favorites" in KDE, it looks like it silently copies that .desktop file from /usr/share/applications to somewhat else location, so whatever changes you're making to original .desktop file from /usr are not getting reflected on your shortcut from Favorites tab.  

The  --user-data-dir trick was needed to actually launch a second Chromium instance, if you already have an instance of browser running without SOCKS proxy. Without it, Chromium just launch an additional process to existing group of processes, where that --proxy-server parameter wasn't set. 

https://superuser.com/questions/1281208/command-line-option-to-open-new-chrome-process-group 

Much better option - use Firefox + plugins

Firefox also a bit lazy one in terms of properly fetching KDE system settings and it requires its own configuration. But there're plenty of plugins you can use.
See here for more details - https://orange-pi-4-lts.blogspot.com/2022/08/few-notes-on-web-browsers-they-all-are.html

So I have Tor up and running. And I configured my browser to use it as SOCKS5 proxy. What's next?

Well if you want to bypass territorial restrictions, like if your government is blocking some websites - you can use Tor as a transport for your traffic to bypass all those your local firewalls and visit web sites you want. It works like that:

1) Your web browser will be connecting to SOCKS5 proxy, which is also running on your machine, created by Tor.

2) Tor's SOCKS5 proxy will be taking your traffic and route it through number (usually 3) of Tor network relay until it exits tor network and goes back to public internet via one of the Tor's network exit nodes. 

 

3) You can affect tor to tell it what exit node to use. Or rather what country that exit node should belong to. Because if you're living in country A which is blocking a website B located in C, it doesn't make sense to route your HTTP requests from browser via Tor in a way they will exit from tor network to Public Internet via exit node located in country A :)

If you want to access your home PC from eveywhere, even though it might be located behind your router's NAT, which might be located behind your internet service provider' NAT (Carrier Grade NAT - or CGNAT) you can spin up a tor hidden service. It's a pretty straighforward thing to do:

1) choice how you want to access your home pc, like by the means of what service - sshd? xrdp? vnc? set that up and make sure it's working in your LAN (nmap localhost, try to make a local connection from other device within the same network).

Make sure you set your service up in a secured manner, which means no simple passwords, no logins under root, traffic should be encrypted - all that stuff. Check /etc/shadow just to see you don't have any extra users with defined passwords. Check /etc/passwd to ensure you don't have any unknown local user records allowing someone to log in:

cat /etc/shadow
cat /etc/passwd | grep -v nologin

2) edit /etc/tor/torrc to add (or uncomment) these lines:

HiddenServiceDir /var/lib/tor/hidden_service/
HiddenServicePort 2222 127.0.0.1:22

What it means is that within the Tor network your host will be exposing a port number 2222 (just for the sake of obfuscating things a bit). Whatever connections will be made to it will go to your localhost' 22 port (which is my SSHD port). Adjust it to your taste if you want to. Like if you want to use different method of accessing you will

3) restart tor and see what onion hostname you've got

systemctl restart tor

cat /var/lib/tor/hidden_service/hostname

4) that's it! now you can go to some other host in any remote place in the world, configure Tor there and establish a connection to your h82ufjjd293kd29fj.onion:2222 port via SSH. Just make sure the client program you'll be using (ssh client, rdp client, vnc client) will be routing its traffic to that host local SOCKS5 proxy provided by tor

Using tor bridges

It might be the case, that your local tor service won't be able to connect to Tor network. This happens in countries which are attempting to ban Tor - so they force their ISPs (internet service providers) to block connections to tor entry nodes.

But the great thing is, community is helping us in form of running additional entry nodes, aka tor bridges. In order to configure them you'll need to do the following:

1) install obfs4proxy:

apt install obfs4proxy

2) go to https://bridges.torproject.org/bridges/?transport=obfs4 enter the CAPTCHA and get some bridges for you. They will be in a format like:

obfs4 123.45.67.89:1234 JISJGOSESDFOKF3 cert=F3g9j29jfxxx/UUIJN3d8Qf77NQw iat-mode=0

The more you get - the better.

 

3) add the below lines to your /etc/tor/torrc:

UseBridges 1
ClientTransportPlugin obfs4 exec /usr/bin/obfs4proxy
Bridge obfs4 123.45.67.89:1234 JISJGOSESDFOKF3 cert=F3g9j29jfxxx/UUIJN3d8Qf77NQw iat-mode=0 

 

4) restart tor and see how it goes now:

/etc/init.d/tor restart
journalctl -u tor@default -a -f

 

Homework

- modify tor config even more to accept connections from anyone in my home WiFi network, limit the relay traffic, get to know how can I monitor network activity coming through tor.service

- to read more about tor, torify, SOCKS5

- why does Tor officially publishes list of IPs for their exit nodes (https://check.torproject.org/torbulkexitlist)? It doesn't make any sense. Answer

- how does the tor service on my PC know where to connect? Does it have any list of IPs or something? Just curious if these IPs will be blocked in my country and what will be the workaround

- speaking of HTTP_PROXY, if I can use http_proxy to access https resources?