A record of a meeting about news, their sources and development. (Attendees: Vladimir Nikishkin, David Wood, Ilya Vorontsov)

Examples: a tsunami, a piece of software released, a president elected, I made a photo of my cat.

Examples: NVidia publishing a press-release, Government office issuing a press-release, A website issuing an update, I uploaded the cat’s photo online.

Example: Bloomberg, Reuters, a blog, a Facebook profile.

They are usually in about 90% of the cases acting as news providers, but in about 10% of the cases acting as news sources, when they have “special correspondents”.

Examples: The BBC, The First Channel (of whatever country), The Washington Post, The Echo of Moscow Radio.

Examples: Cnet.com, TechRadar.com, Habr.{com,ru}, opensource.com, medium.com.

Examples: Google News, Yandex News, Yandex Zen, news.ycombinator.com, Perl Planet

Examples: Email client, Facebook website, Facebook App, AtomFeed Reader, WeChat messenger, Telegram

A hard to define precisely thing, since it is context and consumer dependent. The length of a text/video/audio can be an estimate, but a bad one.

As extreme examples, scientific papers can easily demand 8 hours per page, whereas pulp fiction can be consumed at a much higher rate. Vladimir’s personal record is the SICP book, which took 9 months to read.

• Text
• Audio
• Graphic
• Video
• Digitally-native (3D/Program)
• N-media (e.g. a text with illustrations)

Google News is a service that gives you news headlines, as well as full article bodies fetched from news providers. Google does quite a good job at identifying duplicates. Since one news source is usually later used by many news providers aiming at delivering the content, enriched and post-processed, to their users.

Google News gives you access to the full article body, fetched from one of the providers, but usually not the news source.

Google News uses a sophisticated recommendation method, that is presumably fully algorithmic (no human involvement), to recommend news to the consumers. This algorithm is heavily based on the data Google knows about the users, collected implicitly, the most used data provider being user’s search queries.

The problem with Google News is that it is hard (impossible) to “force” Google to show you more of something. It’s just not possible to make it directly subscribe to something.

Vladimir gets his news by subscribing to individual RSS/Atom feeds via an email-gateway. He gets ~50 news-related emails daily and is quite overwhelmed with them.

This solution has a difficulty in that not all news providers (or news sources) have an Email or Feed gateways. For example, Facebook disabled their gateways circa 2011.

The filtering problem could be solved by crafting various data-collection “sensors”, such as a Chrome extension or a context-sensitive keylogger, and than training a local filtering tool, but Vladimir has so far been extremely far from doing that.

That’s what most people do. They just regularly visit, say, Habr.com, and try to tune the news feed in a way that is as personalised as possible.

1. problem with this approach is that it requires most of the time spent on deliberately reading news.
2. Another problem is that the user is mostly limited to the news from a single provider. There are ways to break this limit (LiveJournal, for example, lets you introduce RSS feeds into your news feed), but that is not a frequently used feature.

Most of the ways above are annoying.

• Too little control.
• Too bad recommendations.
• Bad coverage.
• Hard to tune interface.
• Lack of API.
• Preferences data leaks likely.

What follows will present various thoughts about the news data structures, algorithms and pipelines.

News creation can be:

• Automatic : a CCTV camera finished recording for today, uploads a video file, and updates an Atom feed.
• Manual : I take a photo of a beautiful flower-bed and upload it to Instagram, by creating a new post.
• Semi-automatic : I just upload a new software release .tar.gz on a web server (I am making an event), and some other bot or a human spots this and makes a piece of news.

Often the original piece of news is very terse. There is a process that is called “augmentation” in this document, that makes that piece of news more understandable, more readable, and richer.

Example: Vladimir wrote SRFI-203, which is a technical document. Later, Vladimir wrote an article on Habr.ru, in order to announce the existence of SRFI-203, and in order to provide more context on why it is needed, and to give some examples of its usage.

Augmentation is usually done by the “news providers”, and often is tailored for their audience. This is one of the places where bias is introduced. On the other hand, leaving out augmentation entirely seems not viable, as the readers are often lacking the context.

Naturally, news providers are many, and it is hardly possible to subscribe to each of them individually, especially since many of them do not have any web pages at all, let alone feeds, especially RSS feeds.

The news, therefore, have to be aggregated.

Aggregation can be:

• Automatic
  • Pushed (RSS)
  • Pulled (parsing bots)
• By the employees of a news provider (pulled)
• By crowd-sourcing (pushed)

Naturally, filtering is crucial for any news-related ecosystem, since the amount of noise is giant.

• Implicit : the news that are missed by the collection systems are naturally filtered out. This is not always bad, but many golden nuggets are lost this way too.
• Automatic : Regexps, Natual Language Processing, stop-words, Sieve spam-filters, etc…
• Human : Direct censorship, editorship, class selection by customers, etc.
• Collaborative : That is a mixture of automatic and human. Some bootstrap is made by humans, and then we try to extrapolate the same filtering mechanism on the other news pieces.

Formatting is more important than it is usually seen. Some people are happy with just headlines. Some people prefer abstracts. Other people are into full-length articles, extended articles (long-reads), or even series of articles (a thing that is hard to define!).

A (hypothetical) perfect piece of news supports all the aforementioned levels of abstraction.

The problem here is usually that news are manufactured at a single level of abstraction. We are therefore, met with a problem of up-scaling and down-scaling information.

Since this section naturally deals with the problem of news “weight”, apart from up-scaling and down-scaling, we should mention same-scaling, or re-wording a piece of information.

Note that re-wording is tightly connected to lossless compression. Lossless compression reduces the length of a piece of news, while preserving its weight. However, its practicality seems to be not very self-evident.

In the same section, I have to discuss medium conversion. Medium conversion

• Up-scaling
• Down-scaling
• Re-scaling
• Compression
• Generation (faking)
• Medium conversion
• Language translation

Classifying the news is also important, but a little bit difficult to define. Classification is not entirely the same thing as filtering, although the class of a piece of the news can be a basis for filtering it out or letting it go through.

News pieces have to be stored somewhere. News sources often do not care about making the pieces persistent to any degree.

News providers usually care a bit more about that, but even they often neglect permanence of web-links.

It is not by definition clear which news are worth storing for a long time. CCTV systems are usually wiped very soon after recording, perhaps about half-year maximum.

Indexing and searching is also a difficult question. Progress exists for text search (e.g. elastic), and there are things like “search by image”, but the state of the art I do not know.

These kinds of monetisation may be present at any stage of the pipeline.

• Cost-less/Enthusiasm
• (Sub-)Service subscriptions
  • Time-limited
  • Lifetime
• Pay-per-piece
• Advertisement
  • Explicit
  • Native
• Pay-per-publication (from authors, e.g. OpenAccess)
• Information selling (e.g. selling customer data)
• Sponsorship
  • Government
  • Commercial
• Donations
• Merchandise
• Archive access fee