How to make a good feed reader.

1. TODO Abstract ignore

This file is about how to create a good feed reader in 2023.

Recently I had a discussion with my friend, about what makes modern messengers to be an example of (possibly, intentional) bad design.

This file is a place where I want to write down my wish list for a good feed reader, that would be worth implementing eventually.

In this document, the "TODO" marker at the beginning of the headline signifies that this headline is not finished. In this document, the "TODO" tag (at the end of the line) signifies that this line can be interpreted as a task in a design document.

Not just the way they are served is unreliable, but the content is unreliable too. It might appear and disappear.

2.1.2. What is less obvious is that proprietary services are also unreliable.

Authors delete posts and alter them at a later time to present a better version of themselves to the audience.
Self-destruction of messages is a humiliation for all adult human beings.

2.1.3. Feeds are the source of information, but the reader should: TODO

clean them up
maintain consistency database
indicated deleted and altered entries
retry failed entries gracefully

2.2. Clusterable and cacheable.

2.2.1. It is fine to have a single source of truth, but all data should be also cached locally on all devices. TODO

all servers
laptop
phone
tablet

There is a limited form of hierarchical organisation in Discord called "servers", but it is very limited.

2.3.3. "Folders" have existed in email for ages.

2.3.4. There should be a programmatic way to create folders, based on keywords, regexps, hooks, or something like that. TODO

2.4. Messages should have tags. TODO

2.4.1. Gmail has had tags for years.

2.4.2. Tags must be supported as coming from at least the following sources: TODO

the feed itself
manual tagging
regexp tagging
LLM tagging
hook tagging
full feed tag
randomly (say, with probability 0.1, if you want to shape a feed with too many messages)

2.4.3. Tags must be updated periodically.

A tag might be "a feed which had no update for 1 year".

2.4.4. There should be a way to detect and gracefully process "invalid" tags, that is if a tag meaning has been changed by the user later, or if a manual tag is later found to be conflicting with an algorithmic tagging rule.

2.4.5. Tags should be not just words, but key-value pairs.

Two very important tags which really need the <k,v> structures, are:
1. Interest
2. Importance

2.5. The interface must support content filtering and querying.

2.5.1. Based on categories or tags, the messages/entries must support the following ways of noise reduction TODO

collapsing a message to a topic with a plus-sign [+]
greying-out a message to reduce visual noise
completely hiding a message

2.5.2. Querying should be at least the following. TODO

full-text search
structured querying, based on tags, categories, stored procedures, authors

2.5.3. Message sorting and prioritisation should be supported.

Sorting channels by inverse arrival date is one of the worst possible ways of prioritising messages. (Even though it should also be supported.)
Facebook-style "smart-feed" is even worse that sorting by inverse arrival date.
A typical "view" should be a structured query to the database, tailored for the user. TODO
Heuristically, messages should be the more prioritised, the less their authors post. TODO
There should be a way to indicate feeds which have not been updated for too long. TODO
There should be a way to detect duplicate messages sent to different channels (FB+VK). TODO
There should be a way to detect links to "ground truth" posts references by secondary sources.

2.6. The database should be tightly integrated with an Address Book (possibly a different piece of software).

2.6.1. When I am adding a user into my address book, I want their accounts to be automatically added to the reader.

SMS
RSS
hEntry/hFeed
Email Lists
Facebook
WhatsApp
Twitter
Tumblr
YouTube
Vimeo
WeChat
QQ
Renren
WeiBo
Viber
Kakao
Slack
xmpp
matrix
mastodon
Telegram
Discord
Skype
Zoom
VK
OK
LiveJournal
lj.rossija.org
substack
medium
diary.ru
darkdiary.ru
traditional media websites
custom hooks

2.6.2. Anonymous message (badly signed messages in public channels) should be processed for author detection heuristically. TODO

2.6.3. I want to have some match between an author and an organisation they work for. TODO

This probably is as hard as establishing authorship.

2.6.4. The Address Book should take into account that authors sometimes switch "account IDs", by, say exchanging their phones. TODO

There is no way (or it is very difficult) to explore data silos with automated means.
The system should support adding feeds/authors/while using those silos, such as YouTube normally.
1. a browser extension
2. a phone screen reader
3. a clipboard monitor

If such a need arises, there should be a way to delete messages with heavy unimportant content.
The system should support a way to try and restore deleted messages from original sources if those exist.

2.9.2. The system should support a way to combat adversarial (but standard-compliant) behaviour, such as sending huge amount of worthless heavy data.

2.10. The system should have some integration with instant and asynchronous messaging. TODO

2.10.1. Instant messaging is fast and convenient.

2.10.2. Instant messaging might be one source of messages/posts in a feed, if we find a way to cluster messages by topic, and, possibly, summarise them.

2.10.3. It might be possible to leave a comment to a post, when needed. But this is not well thought-through.

3. TODO Possible implementation details.

4. TODO References.

Completely TODO.

1. TODO Abstract ignore

2. TODO Requirements.

2.1. Reliable (including heuristics) work with sources. TODO

2.1.1. Many people who worked with RSS readers notice that feeds are unreliable.

2.1.2. What is less obvious is that proprietary services are also unreliable.

2.1.3. Feeds are the source of information, but the reader should: TODO

2.1.4. There should be a way to write feed-fetchers manually, for sources which have no obvious feed structure. TODO

2.1.5. There should be a way to poll feeds via obfuscated means, such as TOR.

2.1.6. There should be a way to pretend to be a naive (official) feed fetcher.

2.1.7. There should be a way to work around CAPTCHAs, or offload them to Mechanical Turk or myself.

2.1.8. Images and videos should be downloaded, cached, re-coded, and analysed.

2.2. Clusterable and cacheable.

2.2.1. It is fine to have a single source of truth, but all data should be also cached locally on all devices. TODO

2.2.2. It should be possible to work with the reader completely offline. TODO

2.3. Feeds should be organised hierarchically.

2.3.1. "Folders" appeared in Telegram after 5 years of existence.

2.3.2. "Folders" do not even exist in a reasonable way in WhatsApp, WeChat, Discord, and such.

2.3.3. "Folders" have existed in email for ages.

2.3.4. There should be a programmatic way to create folders, based on keywords, regexps, hooks, or something like that. TODO

2.4. Messages should have tags. TODO

2.4.1. Gmail has had tags for years.

2.4.2. Tags must be supported as coming from at least the following sources: TODO

2.4.3. Tags must be updated periodically.

2.4.4. There should be a way to detect and gracefully process "invalid" tags, that is if a tag meaning has been changed by the user later, or if a manual tag is later found to be conflicting with an algorithmic tagging rule.

2.4.5. Tags should be not just words, but key-value pairs.

2.5. The interface must support content filtering and querying.

2.5.1. Based on categories or tags, the messages/entries must support the following ways of noise reduction TODO

2.5.2. Querying should be at least the following. TODO

2.5.3. Message sorting and prioritisation should be supported.

2.6. The database should be tightly integrated with an Address Book (possibly a different piece of software).

2.6.1. When I am adding a user into my address book, I want their accounts to be automatically added to the reader.

2.6.2. Anonymous message (badly signed messages in public channels) should be processed for author detection heuristically. TODO

2.6.3. I want to have some match between an author and an organisation they work for. TODO

2.6.4. The Address Book should take into account that authors sometimes switch "account IDs", by, say exchanging their phones. TODO

2.6.5. The Address Book should take into account that sometimes different authors use the same "account ID". TODO

2.6.6. The Address Book might have a lot of metadata on authors.

2.6.7. Adding authors should be possible from external tools. (But this is not about the Feed System feature.)

2.7. TODO The reader should have a rudimentary system for content analysis.

2.7.1. It should be able to generate summaries for channels which have too many posts, using LLM or rewriting. TODO

2.7.2. It should generate channel/author statistics at will. TODO

2.7.3. It should generate an analysis of my brain. TODO

2.7.4. It should generate image descriptions with LLM. TODO

2.8. TODO The system should support content sharing.

2.8.1. It should be easy to share a link to a message and its saved copy and edit history. TODO

2.9. The system should support smart message storage.

2.9.1. Ideally, there should never be an need to delete any old content.

2.9.2. The system should support a way to combat adversarial (but standard-compliant) behaviour, such as sending huge amount of worthless heavy data.

2.10. The system should have some integration with instant and asynchronous messaging. TODO

2.10.1. Instant messaging is fast and convenient.

2.10.2. Instant messaging might be one source of messages/posts in a feed, if we find a way to cluster messages by topic, and, possibly, summarise them.

2.10.3. It might be possible to leave a comment to a post, when needed. But this is not well thought-through.

3. TODO Possible implementation details.

3.0.1. TinyTinyRss.

3.0.2. Rss-Bridge.

3.0.3. Selenium.

3.0.4. Llama2

3.0.5. Lisp

4. TODO References.