How to create a decent file system structure and not become insane.

There is a noticeable disparity between the places we expect to see things, and where we really see them.

You may have an excellent task manager, but it will not be of use if you do not open it. And conversely, if you see unexpected things where they should not be, you are more likely to react upon them.

Imagine your wife leaving you a message inside the code file or the document you are currently working on. You are much more likely do something!

Actionable insight I am configuring my system to put reminders, and notifications right at the home directory.

The title is a little be clickbait-ish; in fact, human attention volume can be larger, say, up to 14 items, but the scale is about that large. If you are having more items in a directory, your brain will select it’s own native number of items (7-14), and will ignore the rest.

It particular, it means that it is likely that each directory you make, in general should have no more than “your natural” number of items. Self-check: my home directory (“~/”) has 28 items, and I ignore most of them, except about 7. However, I do notice all unexpected files in a directory quite quickly.

This number is trainable, as most human skills are, but not extensively. You, perhaps, can raise it from 7 to 14, but not to 50.

However, I know two ways of “tricking” this number.

The first way is to give shorter names, or even hide the directories that “you know are there”. Since you remember that they are there, your brain ignores them when it sees them, but you can still see them at the visualisation interface (make sure you have one).

The second exception to this rule is case when items are somehow dependent on each other. If the items have some natural ordering (perhaps according to some date, or a human name), you can have more than your “fixed number”.

Why am I so keen on increasing this number? Can’t we just make groups and subgroups? The answer is “not really”.

Each time you go inside a directory, you are having a context switch, which means you are loosing a bit of context. In other words, the depth of your file system tree also matters. It matters less than breadth, but still. Keep your brain data structures tight.

A dashboard is a misleading thing. Remember the trick that I have given in the previous chapter that can be used to increase the number of items in a directory? (Adding “implicit” items that your brain ignores.)

Here we see the same effect, but in the opposite direction. If you have a dashboard, you are getting “a feeling” that you are up to date with the information, but in reality, your brain starts to ignore things it is getting used to. At least give your dashboard more contrast.

I still have one, and I do have a habit of checking upon it, but it is less useful that I hoped.

Notifications are vital, which also means that they are extremely expensive. Notifications can save you a lot of grey hair if they arrive timely and warn you about something important, but many notifications will blow your mind, they are very expensive to process.

Opinion point: This is why “free”, commercials-funded services are in reality much more expensive than those you pay for. Paid services just eat your money, you can make new. “Costless” ones are eating your life, and you are not getting a new one.

Heuristic: if you cannot keep your notifications at their place, the (bad) trick is to subscribe to too many. Yes, you are loosing important ones, and losing the ability to get early notices, but this is still better than having your life eaten by ads.

Another important point is to get notifications “when and where” you need them. It is not much help to get an important notification from your server while you are driving your car. You cannot react on it, and thus you are: (1) losing energy on processing this notification, (2) losing energy on rescheduling it, (3) maybe wasting time on mitigating it.

Is that obvious?

Essentially, there are two ways of getting new “TODO” items into your list:

1. Notifications
2. Exploration

TODO items is what the skeleton of your life consists of. It is important to notice that the organism does not only consist of the skeleton. The “taste” of life, the “moments of happiness” are impossible to get planned, but if you do not have a solid skeleton, those “happy moments” have nothing to get entangled in and hooked upon.

This is not obvious! Why aren’t “files” those items? Informally, because files can be seen as different “faces”, “views” of the same “thing”.

In fact, you never know when things that you are experiencing in your life are going to grow in abstraction, and turn from a file into a directory. It is better to always start from directories. (WWW Consortium agrees with me: https://www.w3.org/Addressing/)

But the point is – you never know where they will go. If you are going to a dancing party and making a directory for a ticket purchase, it may later turn into a directory for dancing textbooks and videos, or maybe into a directory of cocktail recipes, or a directory of cool dancing places.

But you would still want to also keep this directory in the “tickets” catalogue.

(This is what you need symbolic links for.)

Yes! And this is a problem!

I am trying to use both symbolic links, and hardlinks in order to make the system VFS (virtual file system) match my brain, not the distribution of data on hard drives and clouds.

It works not very well! Suggestions welcome! But so far I have created a fairly reasonable structure from symlinks, bind mounts, and regular copies of file trees with rsync.

Even if your file system structure is decent, you will forget where you put stuff, and you will find yourself exploring your mind map as if it is alien to you. (Sometimes this is also exciting.)

Thus… help your “future self”. Annotate everything that can be annotated, you will thank yourself a million times later.

Context will also help your automatic tools be more productive. I will say a bit more of that later.

The most obvious place to add context to your files is their name. Yes, it is not very flexible, and frankly quite bodgy, but it is the only place that is at least remotely reliable in computing.

There are other places, but they are more specialised.

One more place that is worth considering – is your file headers. You can ofter put the vital context information there.

Context includes:

1. Creation date.
2. Modification date.
3. Refiling date.
4. Publication date.
5. Author.
6. Language.
7. Category. (One per file)
8. Tags. (many per file)

How does a human’s brain work?

We have “Projects”, “Events”, and “Categories” . Projects are limited in time and scope. Events are limited time only. Categories are limited in scope only.

There are also “tags”.

Suppose you are studying Chinese. This gives you a category “Chinese”, under which you would be creating your stuff.

Suppose you are joining the University of Edinburgh. This would give you a category “Uni”.

In year 2014, autumn semester, you are joining an introductory course in Chinese, in Edinburgh.

This course is definitely a project.

You’re studying badly, pass your exam so-so, and get the artefact, the diploma.

You leave Edinburgh, but still keep studying Chinese. In your spare time you are working on the exercises from the same textbook.

Can you write your solutions into the same project? Apparently, no, as the project is already closed.

A file system is a tree. A git repo is a DAG, Directed Acyclic Graph.

You can traverse a tree naively. You can traverse a DAG in a smart way.

However, human brain is not a tree, and not even a DAG. It’s a general-purpose DG.

You can traverse a DG too, but you need to be much smarter than usual.

How would you like to organise your brain? Keep in mind that there should be some data structures available for shared usage with other people and robots.

Let’s take a simple example.

You have a directory called ~/Music , where you put your music. You have a directory called ~/People/Mom , where you put things that are related to your mom.

For example, your Mom likes a band called Gogol Bordello, and you also like the same band. Would you put it into ~/Music/Gogol-Bordello, or into ~/People/Mom/Gogol-Bordello? The problem is exacerbated by the fact that you may need to update the names of directories.

In the file system visualiser, I am using soft links to associate directories. But in general there seems to be no good solution yet.

The heuristic here is: first build your hardware/software synchronisation, later build the semantic harness.

Things to consider:

1. Pictures and “daily data chronology”.
2. Other People. You will be sharing some subtrees of the file system, as if you are having some common parts of the brain.
3. Downloaded vs Personal files. You usually do not care about loosing downloaded files that much.
4. Repositories.
5. Raw data. It will be badly displayed on you “file system map”, so you have to think in advance how to store it.
6. Projects by time, status, and class.
   1. By time: these are usually projects you do not expect to last long, for example, buying a theatre ticket.
   2. By status: these are projects that occupy so much of your effort that you cannot just put them into a category. You would have <7 open ones, and would make your reminder system remind you about them as often, as posible. As they are being closed, you would reduce their level of annoyance and move either into the “topical project directories”, or into the “projects by time”. You can make symlinks too.
   3. By area/category. Some areas of life naturally occupy some significant volume of your life. Put your projects there.
7. Official documents and government interaction. You don’t want those just laying around your file system, as those are more sensitive.
8. Your medical infromation. This is as important as a “by status” project, but by nature is a “by category” project. This data is also sensitive.
9. Financial information. Same as above.
10. “Incoming” directories with stuff to read and digest. Those tend to take a lot of space, so be careful.

The main difference between 3.2.2 and backups is that backups are restorable objects.

Data Dumps are file system subtrees or, sometimes, archives, that usually appear as a result of using a non-specialised tool for “saving” some data in a dangerous situation, instead of using a special-purpose backup tool.

They are usually non-restorable.

Git and friends. Try to store all your text files in a VSC, it pays off.

Reconciling the differences between two copies. Often used as opposed to merging (two conflicting copies).

Taking two versions of the same file, developed separately, and combining to create a single one.

Storage that is frequently emptied. For example, a tmpfs.

For a personal laptop, every non-resumable operation is limited by a 8 hours time window. Because realistically, every operation should be done at most overnight. Every resumable operation is limited by 7*8=56 hours, as that is the amount of time available during the week. Practically, a backup that is more than a week old is useless.

When you have two “more or less similar” copies of a single directory tree, you are in a big trouble. Now you have to combine them somehow, and get a “master copy”. Not easy.

A well-tuned computer need to run tasks for self-maintenance. On Windows, many people were used to defragmentation and disk checking. On Linux we still need disk checking, file system checking, and several other upkeep operations.

Some things cannot be done by a machine. For example, when you need to connect a backup HDD. Those tasks you need to plan in advance and enforce yourself. This is hard, but worth learning how to do.

A not so bad tool to find which of your directories take up too much space.

A console version of 3.3.1

Do not use fdupes.

Do not use rdfind.

Checks your file system for errors.

#switches to fsck.ext4 mean the following: -c run_badblocks_ro -c -c run_badblocks_nondestr_rw -C 0 show_progress -f force_check -k keep_old_badblocks_list -y auto_repair_yes -t -t print_time -v verbose
echo time fsck.ext4 -c -c -C 0 -f -k -y -t -t -v /dev/sdc1

<2020-11-20 Fri 13:55> I found rmlint recently. It is a bit weird, but at the end of the day turned out to be more reliable and tunable.

It is an excellent tool to use for Subtree Merge. Highly recommend.

https://www.speedtest.net/apps/cli

exists for ARM 64, and has a huge database of servers

speedtest -s 26850 would do a test to some server in Wuxi, China

If you still do not use it – it is time to start. Learn it well, and it will help you a lot when you “kinda” know where your file should be.

The excellent “regular expression search tool” to use for content search within files you “kind of” know where they are.

Learn it and start using it. It’s a great tool for super fast search of “stuff that was out there somewhere”.

It’s an amazing, very fast and efficient desktop search tool. It takes time, maybe, days to index your drive, but contrary to Gnome’s Tracker and KDE’s beagle, it actually works. The database is huge and you probably need an SSD for it.

I do not use it that much, because with a good FS structure you can be doing find/grep many times more often, and with good context you can just get by with “locate”. But in those cases when you “do not really remember”, recoll helps you “recoll”.

Rsync is an extremely versatile tool with an extremely fragile syntax

The following will copy everything from root to the backup root.

Combined with rmlint, it can be used as a 3.2.8 tool. In general, it is hard to use, but much-much better than just cp or scp. Lets you resume your transfers, do incremental backups, getch backups from remote machines, and a lot of similar things.

echo time rsync --links --partial --fuzzy -arHAXyh --info=progress2 /
echo time rsync -v --archive --hard-links --acls --xattrs --inplace --one-file-system --del --fuzzy --human-readable --info=progress2 --partial --dry-run from/ to

Is a very versatile tool for downloading all kind of stuff. I recommend it. It can download through ssh too! sftp is actually ssh

time a2 –max-tries=0 –ftp-user=username –ftp-passwd=<scrubbed> sftp://server.lockywolf.net:22/mnt/hd/file.txt

Where a2 is an alias for alias a2=’aria2c -l /tmp/RAMFS/2021-01-06T13:37:11+08:00-aria2-download.log -x120 –min-split-size=148576 –split=120 –auto-file-renaming=false’

You need an aria2-nitro patch to allow 120 connections with 100k splittings.

Important! by default, sshd has a built-in DDOS protection setting MaxStartups 10:30:100

You want to set it to 120:30:220 or something. But be wary of a real ddos.

Keep all you literary work in git. I use magit on Emacs, console git in console, and mgit on Android for my diary synchronisation.

A kinda fragile, but still extremely useful tool for synchronisation of machines, and it can also protect you from a bit of regrets after deleting things automatically. Put in Syncthing things that you cannot put on git.

It is that tool that lets you extract all the valuable from your photographs.

This is not really about organising files, but rather about creating files, but I cannot avoid mentioning it here, because org-mode is very versatile tool, and you can build a lot of your personal information management system on top of it. You can add to your files with ease. You can also add cross-references without much difficulty.

Lets your run tasks periodically. Worth learning.

Those relatively nice tools that let you donate all your data to an oligach in the name of his business interests. It is also quite convenient and works as an additional backup for your files. The killer-feature is making the files available on your phone without synchronisation.

If you can, avoid it with the help of NextCloud. But probably you won’t be able to.

A replacement for Dropbox.

A way to keep your email locally and read it without internet. Is there a way to use it as a file system? Or visualise?

A way to keep your contacts on the local disk just the same as they are on your phone. Is there a way to use it as a file system?

A way to keep your diary records on your local disk for indexing an search. Is there a way to use it as a file system?

Lets you backup Skype.

A relatively unified format for keeping archives of things that have a natural order in time. Consider exploring RSSBridge or other anything-to-rss portals to avoid Internet Giants’ trickery.

A tool to create a 2.3.3 on your local computer. Useful for outputting the scripts that check that all of your synchronisation machinery works.

Lets you backup your /etc with the help of 3.3.14.1.

A way to add valuable to your files. The tools are called setfattr and getfattr.

You must have them configured. They are the only thing that can give you at least some thing of a warning before you HDD dies.

https://phoenixnap.dl.sourceforge.net/project/adbextractor/android-backup-tookit-20180521.zip

are actually tar files, and can be unpacked as tar xvf

https://stackoverflow.com/questions/36819474/how-can-i-attach-a-vhdx-or-vhd-file-in-linux https://download.libguestfs.org/1.43-development/libguestfs-1.43.3.tar.gz

It needs hivex and supermin. Damn, a lot of work. I build hivex from github tarballs, but redhat disapproves this.

guestmount –add yourVirtualDisk.vhdx –inspector –ro /mnt/anydirectory

I didn’t manage to mount this with qemu 5.0. Trying to build 5.4.0rc4. Also failed. And Windows 10 in vbox failed. Perhaps, the file is actually broken.

Living without a VPS is hardly possible nowadays. You need it for every task which needs a public address.

You may have to keep some data there, but it is usually expensive, and it is “another person’s machine”. So ideally, the file system should be encrypted (from the hosting company’s technicians).

You need it, because you cannot keep all your data with you all the time. This is where you will keep most of your data, as well as backups.

You have to be prepared that you may drop it and it will die. Or you root ssd will die. Or your home ssd will die. But you will still keep your useful data there.

It is worth designing your system in such a way that losing your phone is sad, but not too much. You need to backup or synchronise at least:

1. Contacts (see 3.3.22)
2. Logins (you can use some special service, but Android built-in is probably enough)
3. Data (look at 3.3.15)

Encrypting your phone will likely make it a huge pain to extract data from your phone if the screen is broken. I used to belive that having a regular backup (a nandroid or a dd image) is a good idea, but not any more. Keep your stuff on your laptop or NAS, not your phone.

If it has locked forever – just break it with a hammer and destroy the memory.

Is usually integrated with a messaging/scheduling service, and is thus convenient. You will need it as an interface to your friends who are controlled by the machine anyway.

For root drives, it is enough to keep a backup root drive in every machine and do an rsync backup every night. You can just plug the drive instead of the main one if the main one dies. For the laptop, you have to carry a drive with you, because laptops usually don’t have 4 drive bays.

You need at least two backup drived for your laptop, because ssd’s tend to die “suddenly”. So you will have an ssd drive to backup your home partition every day quickly, and a more reliable hdd to backup the data overnight.

Those tend to die extremely quickly, so have a few in your pocket, ideally, identical. It’s worth having them linux-bootable, and have a few root directories:

1. Windows installers
2. Windows portable software
3. Portable documents (which you delete at each plug-in)
4. Permanent documents (your passport photo and such), that you may urgently need to show to the police and other officers.

Doesn’t work yet for “all apps”, only root ones

#!/system/bin/sh
while [ "$(getprop sys.boot_completed)" != 1 ];
do
sleep 1
done


su -mm -c mount -v none -t tmpfs -o size=4g,nosuid,nodev,noexec,noatime,context=u:object_r:sdcardfs:s0,uid=0,gid=9997,mode=0777 /mnt/runtime/write/emulated/0/Download/tmpfs-cleared-on-reboot > /sbin/.magisk/img/.core/mount.stdout.magisk.log 2> /sbin/.magisk/img/.core/mount.stderr.magisk.log

I have a separate “howto”, which is more of an example, on how to draw a data flow diagram for your cloud. Visit the article.

Unfortunately, such a file is hard (if not impossible) to generate automatically, and updating it is a pain.

But even so, getting a high-level overview of what your digital brain is like, is priceless.

I generate the file system map and print it on a giant three-by-one metres poster on a wall.

tmpfs            /tmp/RAMFS       tmpfs  nosuid,nodev,noexec,sync,dirsync,size=4G,mode=1777         0    0

Note: I mount /tmp as tmpfs, and /tmp/RAMFS as tmpfs-noexec. /tmp needs to be exec for building packages.

Why would you want that?

Because files that you download steal the space on your HDD, and, what is worse, they steal your attention when you are browsing your disk and/or searching in it.

You do not want to keep any useless files for less than needed.

Transmission->Edit->Preferences->Downloading->Automatically add files from-> /tmp/RAMFS

Transmission keeps .torrent files in ~/.config/transmission/torrents

This is the same idea as in the previous paragraph.

Do not store anything worthless, and save time on navigating Transmission’s interface.

I didn’t find how to do this, I basically mounted /tmp as ramfs. Firefox creates a directory called “${usename}_firefox0” and downloads stuff there.

echo find . -iname 'desktop.ini' -delete

Worth doing after 3.5.9

find . -type d -empty -delete

echo rmlint -T "df" -c progressbar:fancy --progress --no-crossdev --match-basename --keep-all-tagged --hidden --must-match-tagged ~/Incoming/ // ~/good-dir

Run on about 400 GB.

lockywolf@delllaptop:~/Incoming$ rmlint -T "df" -c progressbar:fancy --progress --no-crossdev --match-basename --keep-all-tagged --hidden --must-match-tagged ~/Incoming/ ~/BACKUP/ //  ~/books/ ~/Data/
⦃⌿⌿⌿⌿⌿⌿⌿⌿⌿⌿⌿⌿⌿⌿⦄              Traversing (585350 usable files / 0 + 0 ignored files / folders)
⦃⌿⌿⌿⌿⌿⌿⌿⌿⌿⌿⌿⌿⌿⌿⦄                  Preprocessing (reduces files to 107623 / found 0 other lint)
⦃⌿⌿⌿⌿⌿⌿⌿⌿⌿⌿⌿⌿⌿⌿⦄   Matching (59120 dupes of 44798 originals; 0 B to scan in 0 files, ETA: 50s)

==> In total 585350 files, whereof 59120 are duplicates in 44798 groups.
==> This equals 81.36 GB of duplicates which could be removed.
==> Scanning took in total  1h 29m 10.406s.

Wrote a sh file to: /home/lockywolf/Incoming/rmlint.sh
Wrote a json file to: /home/lockywolf/Incoming/rmlint.json

You can use it to share files over your VPS right from your phone.

Rewrite it with a batch like xargs or smth.

With pkgtools-15.0-noarch-41, the installpkg can be modified to obtain the $SUBJ behaviour by adding

( # line 670
    cd $ROOT/
    grep -v '^install' "$TMP/$shortname"  > "$TMP/$shortname"_noinst
    xargs --arg-file="$TMP/$shortname"_noinst --delim='\n' setfattr --name=user.slackware_v1.installpkg.package_name --value="$shortname"
    rm -f "$TMP/$shortname"_noinst
)

to line 670 of /sbin/installpkg

and makepkg can be modified by adding

find ./ -type f -exec setfattr --name=trusted.slackware_v1.makepkg.package_name "--value=${TAR_NAME}" {} + # line 414

to line 414 of /sbin/makepkg

Oftentimes I have the following task: I am working on some task that can be published later. For example, I am reading a book and writing a review, which produces two files. These files I keep in a git repository for “book reading”.

There is also a repository for the blog posts. I do not want to synchronise the same file in two different repositories. I also do not want to check in pictures into a git repository, because git is not very efficient at working with binary data.

So I make a hard link of the review file. This way the git-versioned file gets all the changes instantly, even if the file is modified on a different machine. Moreover, if I update the file from a git perspective (adding something to the review), the changes automatically get into the Google Drive directory.

There is a huge caveat here: Some programs, notably, Emacs, by default make backup files by renaming. That is, a file is renamed to be called filename.bak, but it is still hard linked with the old “primary brother”. In order for Emacs to not be bad here, you need:

(set-variable 'version-control nil) ; should be t, but breaks hardlinks
(set-variable 'backup-by-copying-when-linked t) ; --//--

• by the package it comes from
• by the program that has created it
• by the user who last used it: uid and username

Should be doable with a kernel module, or something.

Kprobes? uprobes? systemtap?

Supposedly, tracker should be tagging stuff automatically. Didn’t try it.

declare DISK
DISK=sdc
cd /root
# -b block_size -w destructive -s show_percentage -v verbose
#echo time badblocks -b 4096 -n -s -v -o /root/"$(date --iso)"-sdc.badblocks "/dev/sdc"
echo time badblocks -b 4096 -w -s -v -o /root/"$(date --iso)"-"$DISK".badblocks "/dev/$DISK"

With USB2.0 estimated time – ~75 hours? Hm…

Didn’t write down the USB3 speed.

Esata speed 0.1% – 1:50. Which makes is ~33 hours to just write one pattern?

Manual TWRP without “sdcard”, all partitions, via TWRP to an OTG disk takes:

3389 seconds, that is roughly ~1 hour, in total 41Gb.

Storage is not yet measured Started <2020-06-19 Fri 20:28> – <2020-06-20 Sat 00:10> – Not finished 4 hours is not enough, but the phone battery is dead.

You will likely have to backup everything that is listed in the “software” above. When doing backups you have to consider the following:

1. When to backup. You cannot fetch data when you are offline, and too often you cannot guarantee that you may be online.
2. How to backup. You do not want to clone your hard drive over a metered connection.
3. How to rotate backups. If your backups occupy too much space, this space is lost.
4. How to make sure that your backups succeed. A good backup is, by definition, unnoticeable, you shouldn’t remember it exists, unless you need it. But this means that you don’t have a reminder (by default), when your backup fails.

I generally try to support each backup procedure with two auxiliary subprograms.

One run together with the backup itself, and notifies me if something goes wrong. The backup itself does not notify me, because the backup often goes wrong. E.g., some backups I run every 5 minutes, but they succeed only once or twice a day, because only at that time the required device is nearby. But there should still be a service that sends you an email if backups are failing for too long.

The second proceduree updates my Dashboard and paints bright red things that have obsolete backups.

I have a “BACKUP” directory, on my laptop, which is roughly 2-level structured, as in: class/application, e.g. 01_Messengers/ICQ. This directory is getting “scheduled backups” every time it can, with regularity ranging from 5 minutes to 1 day.

My rough list:

• 00_Etc-Git-All-Machines
• 01_Browser-settings-sessions
• 02_Hardware-Data-Stock-Setups-Manuals
• 03_Contacts
• 04_Organiser
• 05_Lists-of-Apps-If-I-Forget-Where-Data-Was-Stored
• 06_SMS-Message-Databases
• 08_Chat-Logs
• 10_News-Feeds
• 11_Voice-Message-Recordings
• 12_Call-Logs
• 13_Directories-Lists
• 15_Diaries-Paper
• 99_Data-Dumps

Additional operation to do overnight:

1. Reindex locate
2. Reindex recoll
3. Update softlinks on the basis of xattrs
4. Run BOINC (in the name of the greater good)
5. Update file tags semantically – run text OCR on PDFs, images, entitity detection, all the cool “NN-related” stuff.

Not everything can be done automatically. The easiest example is the portable SSD that you backup your laptop to. You need to plug it in, as modern laptops just do not have a bay for a separate drive. This is getting even more important as new laptops have the NVMe storage soldered onto the motherboard, so if you break your laptop, you cannot even take the drive out.

This section has a rough outline of what is probably worth doing.