Reading and referencing in Emacs Org-Mode.

Seems to be the first found reference for bibliography is org-mode. Its most prominent feature seems to be importing bibliographic data from webpages via org-protocol capture.

It generates org-headings in a prescribed file, with bibtex-code entry pasted into the heading body, and the same metadata being saved as org properties of the same headline.

It uses Zotero as a link between Firefox and Bibtex, since, apparently, at that time Firefox did not support xdg-style links “org-protocol://”. Or, maybe, the big idea is that Zotero has some code for automatically extracting some metadata from webpages via its database of shims with different paper databases.

Kind of interesting, if the “ground truth” of your reading list is an org file.

Not developed since 2009, so now is probably only of historic interest.

The workflow there is surprisingly similar to Fireforg. Zotero is still used as the tool to manage books and articles, which are then exported through org-roam-bibtex to an org-roam node with bibtex properties encoded as org properties.

I guess they are then “transcluded” using org-transclude to the final document?

In any case, I consider org-roam’s approach of using a separate SQLite database and mandatory IDs to be fatally flawed. (But I should check, maybe something has happened since my last time looking at org-roam.)

Also, some plugins are mentioned (added to Literature Review), with which I had terrible experience.

The important thing here is the difference between ox-bibtex and ol-bibtex. They are not the same thing. ox-bibtex defines a cite:foo link, for using citations in org documents.

ol-bibtex does import-export to and from actual bibtex text files. I am not sure how ol-bibtex links are exported when exporting to html, I need to check how it works when reviewing packages independently. But the main use for this package is, seemingly, to keep a list of papers in org, and export into bib files when assembling a paper.

I suspect that one would make an org-file with a “reading list” for a project, populate it with books as they appear on the horizon. Those books might be pointing toward, say org-noter files with review?

The idea here is, seemingly, to have “ground truth” in org files, not in bibtex files. Each heading is a book, which has bibliographic data recorded as org properties. Importing and exporting is done semi-manually, in the sense that you can export the required headings into a bib file, which, presumably, you would only do when compiling a paper, but it does not see to be possible to set an org file as a bibliographic database with exports organised completely automatically.

Importing stuff into org is also done semi-automatically. The code will help you to yank a piece of biblatex as an org-heading, but not much more.

Citation exports seem to work using ox-bibtex, that is, using the cite: format. The format using “ordinary links” is not yet mentioned.

This is a howto by Arne Babenhauserheide. He is using a specific LaTeX style, which he is adding to org-latex-classes, as well as a few custom packages.

He is explicitly loading reftex-cite-only, not full reftex, but it is clear that he is going to use reftex for citation inclusion.

He is using org-mode-reftex-search, which is an old function found on the org-mode mailing list, https://list.orgmode.org/3613.1329506279@alphaville/T/. This function, if I am not mistaken, should make a jump to the notes for an original cited document. The code is missing, I guess, Arne copied it from the mailing list into his .init.el.

It is interesting that there we are encountering the concept of “notes file” once again.

Also, it is interesting that he is using minted for org code listings, which, in turn, uses Pygments. I never bothered to make them work, and nowadays there is a whole new machinery for org, called engraved, which should make it possible to colourise both latex, and html, using Emacs means only.

Another useful thing that this HOWTO has is #+BIND: variable value syntax. It lets one override some variables for exports, which is especially useful when there is no #+KEYWORD: syntax for this variable, and when using file-local variables is imperfect, such as when you need different values for editing and for exporting the file. (You also need to set org-export-allow-bind-keywords to t.)

In section https://orgmode.org/worg/org-tutorials/org-latex-export.html#sec-17-1, there is an interesting trick on how to make references (not citations) work:

(setf org-export-latex-hyperref-format "\\ref{%s}") will make intra-document references work correctly.

Otherwise, they define a strategy that is basically like ox-bibtex, defining custom links for each citation type.

What I have learnt from his article.

1. He also uses Zotero for article management. I guess, I need to study it once more. He use the same BetterBibTex that https://emacsconf.org/2020/talks/17/ is recommending.
2. He faithfully uses org-ref (makes me doubtful)
3. He has a single file with Bibtex metadata in PROPERTIES, and with annotations in bodies. I guess, things are imported there using ol-bibtex. How exactly that page and the bib file are kept in sync I do not know.

4. He is having three blogs on his site: blog, journal, log and a “wiki”.

   1. Blog is for “longreads”
   2. Journal is for shitpost
   3. Log is for habit tracking
   4. and there is also what I call “howtos”, that he puts in a “wiki”.

   This seems more complicated than my setup of just two categories, “notes” and “howtos”. I have tried to switch from paper into keeping my records in a journal for a very long time, but always failed.

This is Dennis Ogbe’s setup. He is using ebib+helm-bibtex+org-ref

The interesting bits in his setup are the following:

1. He uses multiple .bib files merged into a single database.

(defun do.refs/update-db-file-list ()
  "Update the list of bib files."
  (interactive)
  (let ((db-list (do.refs/get-db-file-list)))
    (setq reftex-default-bibliography db-list)
    (setq bibtex-completion-bibliography db-list)
    (setq ebib-preload-bib-files db-list)))

So, his “ground truth” is still biblatex, but he has a way to group “Points of Knowledge” into categories by different bib files.

1. He has separate dirs for PDFs, and notes.

(defvar do.refs/db-dirs nil "A list of paths to directories containing all my bibtex databases")
(defvar do.refs/pdf-dir nil  "The file for the entry with key <key> is stored as <key>.pdf")
(defvar do.refs/notes-dir nil "The note for the item with key <key> is stored as <key>.org")
(defvar do.refs/pdf-download-dir nil "The path to directory we download PDF files.")

Some things I immediately dislike here. Firstly, rigid notation for naming PDF files. I like calling my files with full names. In general, keeping as much info as possible in file names is good. For example, I have PDF files on my drive called like 2023-09-01_Various-Authors_GNU-Maxima-manual-for-version-5.47.11_2023.pdf, and I like it this way. Moreover, I kind of like it where it is, I do not want to specifically put it somewhere.

1. He uses autokeying.

And I do not like the default authoryear autokeying of bibtex.el, because it is too easy to forget what jackson2001 means. I want my keys to be full names with dates: various2023maximaManualForVersion5.47.11. Since I am going to use some automated machinery to cite those papers, long keys should not matter.

As a side-note, there should be a way to make autokeying better:

(setq bibtex-autokey-year-length 4)
(setq bibtex-autokey-titleword-separator "-")
(setq bibtex-autokey-name-year-separator "-")
(setq bibtex-autokey-year-title-separator "-")
(setq bibtex-autokey-titleword-length 16)
(setq bibtex-autokey-titlewords 8)

1. But I like that I am, once again, seeing a repeated pattern: PDFs, notes, bib-database.

There is one more interesting bit there:

(defun do.refs/ebib-add-annotated (arg)
    "Advice for `ebib-import-file' that automatically creates a
  copy of the imported file that will be used for annotation."
    (interactive "P")
    (let ((filename (ebib-get-field-value "file"
                                          (ebib--get-key-at-point)
                                          ebib--cur-db 'noerror 'unbraced)))
      (when filename
        (let* ((pdf-path (file-name-as-directory (car ebib-file-search-dirs)))
               (orig-path (concat pdf-path filename))
               (annot-path (concat pdf-path
                                   (file-name-sans-extension filename)
                                   "-annotated"
                                   (file-name-extension filename t))))
          (unless (file-writable-p annot-path)
            (error "[Ebib] [Dennis] Cannot write file %s" annot-path))
          (copy-file orig-path annot-path)))))

  ;; add the above after the original call is done.
  (unless (and (boundp 'do.refs/add-annotated) (not do.refs/add-annotated))
    (advice-add #'ebib-import-file :after #'do.refs/ebib-add-annotated)))

See! He also has a file which will be used for annotations only.

1. The rest of his ebib configuration is fairly straightforward. Note that he is using the “note” field, but not the “annotation field”. (But maybe that is an old version of ebib?)
2. He uses bibtex-completion. While bibtex-completion deserves its own chapter, I need to instantly write down something here:

(setq bibtex-completion-find-additional-pdfs t)

This snippet means that apart from key.pdf, bibtex-completion will also consider PDF files named key-*.pdf for completion.

1. He also uses org-ref, and from his code I did not understand how!

And, in fact, he does not even explain much about hit org-ref use-cases at all. The one thing that is worth noting, however, is that he hooks org-ref to use ivy-bibtex to insert citations.

1. He uses a self-written Emacs function to extract citations from a document and generate a one-shot bib file for publication.

This is where reftex comes into play. He really only uses reftex to extract all citations from a latex project sort-uniq them, and generate a bib file.

This document has not been written linearly. In particular, this blog post is praising org-ref, which, at the time of writing this sentence, I have already reviewed. Nevertheless, as a Russian proverb is saying, “repetition is the mother of study”.

Let’s start:

(org-babel-load-file "org-ref.org")

Wow, okay, he is using org-based loading. I remember @wasamasa converting his dotEmacs to org, but I always doubted this approach. But okay.

There are a few more things mentioned in this article:

1. Citation styles. What he is calling a “citation style”, is what I call “citation formatting” in the text of the article. That is, is the item cited by name, or by author, and is year included, or not. So, basically, irrelevant stuff, which, I guess, must have given him a lot of pain when submitting research articles.
2. List of tables.
3. List of figures.

What is not mentioned, although, I expected it to be, is formatting of the bibliography. That is, how exactly items are presented at the end of the paper.

An article, which exhibits technologically nothing new, same old bib:key, and note:key org-links, which open either a bib file, or a notes file. But what is valuable in this file, is his description of his workflow; it is written with exceptional clarity, and describes the knowledge acquisition process in great detail. Let me try to repeat it here by myself:

1. He has a .bib file for each paper, downloaded from a bibliography database, or a journal website. (These files are concatenated together to make one large database file.)
2. He has a “notes.org” file, a single file, where each heading is either a category, or a paper. Bodies of paper nodes contain, well, notes about those papers.
3. He creates headings in that org file with reftex, using a clever citing format.
4. Entries in the org file are tagged meaningfully.
5. His files have the name of key.pdf, and he also changes keys in .bib files to key.
6. He also heavily relies on reftex.

A workflow that is similar to both the one used in Emacs Conference talk of 2022.

The question I see here is the following: How do you structure your “projects” when doing science? Papers “database” can consist of objects of different granularity, and “projects” also can be of varying granularity.

If a paper is read and understood as a part of a project, shall time invested be contributed to that project or to reading in general? Copying logbook entries in org-mode is annoying, even if possible.

The interesting property of this setup is that rather than keeping paper metadata in org properties, it is kept in a babel block.

Which makes it easy to tangle the bibliography into a single file, which can be included into latex.

In general, his setup is quite similar to the setup from Emacs Conference talk of 2022 .

1. He does use Zotero as a reading-list management tool. First red flag :D, but anyway.
2. He also used org-roam for a “personal wiki”.
3. He seems to export the Zotero database into a single .bib file with “Better Bibtex”.
4. His “innovation” seems to be using “deft” for searching notes in a directory.
5. He is using org-noter, which, again, I am starting to like less and less.

Okay, his setup is consistent, but I keep getting annoyed with a few things, that, in my opinion, mar all of those setups.

1. Using an sqlite database for backlinks is insane.
2. Using a single directory for notes is insane.

At least he manages to conjugate notes and references, so in some sense his setup is almost the most consistent from among what I have seen.

This is a bibtex-completion (helm-bibtex / ivy-bibtex) based setup. Its prominent feature is the the use of citar as a citation-management tool.

Otherwise it is not too remarkable.

This blog has another interpretation of the “rtcite:” link.

One more suggestion of using reftex, with not much detail. The interesting bit is using latexmk for building the pdf, which is now a must-have for me too, but I already have it.

They guy has discovered org-ref, and is frustrated by it. How familiar.

What he does right, however, is mentioning that org-ref is for formatting citations. Of course, he only cares about LaTeX, so the HTML part is missing.

He has an excellent suggestion of recording all readings in an org file. So that each entry would be a boot, and the bodies would be his comments.

The problem is that I cannot process audiobooks too much, they are too quick for me. And for text books I usually need a much-much more detailed notes file.

Okay, this is interesting and has some “meaty” stuff.

The first meaty piece is this script: https://github.com/novoid/extract_pdf_annotations_to_orgmode

It lets you extract annotations from a PDF file into org-mode. This script is, seemingly, not round-trip, but even one way is useful as a source of inspiration.

The more and more I am thinking that PDF metadata should be stored where it should not be lost, ideally. In the pdf file itself.

This sketch is very short, but it is nice to see that people are considering more or less the same options for citing and referencing that I do.

A fairly standard setup of org-ref and bibtex-completion, with an attraction point of making a dedicated list of books to read, along with page count in org columns.

Fun? Maybe, but not for me, I guess.

After all, I like structuring my life according to projects/tasks, of which books would be parts. Having a dedicated reading list sounds contrived. Also, what about web pages there?

This is a good intro to oc.el from a developer point of view.

I will definitely need to re-visit it when implementing my own bibliography system.

A classical link that is the father of this document.

This is the way I have been using citations in org for a long time, and, I think, this is what is implemented in ol-bibtex.

“bib” links are exported as proper citations, and the bibtex file needs to be carried around manually.

This is how my setup used to work before considering this review.

This StackOverflow question has a nice MWE for the new org citation machinery.

The most important quote from there “It took me several hours to get the system working”.

Interesting.

The “source of truth” here is bibliography.org, which is exported to bibliography.bib using ox-bibtex.

org-bibtex-extras lets one annotate the reference.

Citations are inserted with reftex, via org-reftex-citation, which even has some sort of “intelligent completion”.

Links should be done, presumably, with ol-bibtex.

I guess, this setup has a lot to learn from. In particular, it would be, maybe, nice to tweak it slightly in the following way:

Make a “review.org” file for each review, done with “org-noter”. The top heading would be compliant with ox-bibtex, and would produce a tiny bibtex file. Those files would be joined together to make a bib bibliography, which could be used with, say, reftex, or bibtex-completion.

Possibly, the “new” citation machinery would be even able to jump to those “notes” by the means of the “activate” processor.

One extra feature from there worth learning is the ox-extras, which allows ignoring headlines but not children. This allows making fake headlines for, say, bibliography and list of figures.

Definitely worth re-visiting.

This is a giant, and also very interesting description of one’s life in org.

I am very impressed by his approach, thoroughness, and meticulousness.

However, I don’t think that his way of working really fits me, for several reasons.

1. His task state machine is too complicated. I just won’t be able to track all those state changes.
2. His capture machinery requires being able to capture all tasks that might occur, and I just don’t think I can implement that.
3. His machinery does not seem to be able to integrate well with distributed PIM. I do need to do a lot of stuff on my phone, and sometimes even on the “cloud”.
4. I am spending a lot of time on actually doing the planning. Like, I am trying to decide how to progress further in my life, and which steps would be more valuable, and which urgent. His way seems to expect a fairly standard way of progressing through tasks.
5. I am often not in the state when I can do a task. Like, very often. If I may (very roughly) say that I have “capability X± δ X”, and tasks have “difficulty Y”, a lot of my tasks have difficulty “X+(δ X/2)”.

I am finding it extraordinary that he is actually billing his clients based on org clocking data. This is very impressive.

I don’t really believe that his bbdb stuff works. It is just not advanced enough.

His guide on abbrevs and skeletons is also something to learn from.

To sum up: I strongly recommend reading and learning from his example.

However, I did not learn much about either reading or referencing from his treatise.

The key thing in this setup is using elfeed to fetch a list of new papers from arxiv.

It seems tremendously useful, not having to deal with the arxiv interface, and such, however, I am not reading a lot of arxiv even though I do use some paper from there. I guess, it is really useful for people where a single paper can be read in a few hours, not weeks?

After the introduction of the arxiv fetching machinery, his setup essentially converges to a “bib-file with file: field”, and uses bibtex-completion to open those pdfs when needed.

An interesting, although a very bloated configuration.

Moreover, he even suggests using org-roam-bibtex, which is a king of all of them emacs org-related packages, and tries to integrate so many of them that my head is spinning.

What is interesting is that his “ground truth” seems to be coming from org-roam, which is not what most people do.

What makes this article stand out is the introduction of the citar package, which is one of the newer oc.el citation processors, and can probably be used instead of helm-bibtex.

He also uses Zotero, which I, again, probably need to study.

He mentions embark and marginalia, which are probably worth looking at, even though, maybe, not for referencing or researching directly.

Again, the most important contribution of this essay is probably the availability of external oc.el processor citar.

Okay, this is a slight variation on the subject of making notes.

Still requires a fixed directory, but now at least it does not require sqlite.

A nice intro into oc.el. Recommends using Zotero for actual bibliography management, with export to biblatex with the “Better BibTeX” plugin.

Okay, I clearly would never use Kitchin’s setup verbatim, but there is no reason not to learn from him.

1. Snippets. So far very few examples from here mentioned snippets, perhaps only one, which used abbrevs. I also do not see why the author uses yasnippet instead of org-tempo or abbrevs.
2. Many custom keybindings. I never got used to them, so whatever.
3. Hydras. A pop-up interface for hotkeys. As usual, hydra is competing with, say, transient. But I really think that such a functionality must be built into Emacs, not provided externally.
4. Spell checking. He just has some tweaks to hunspell, which is better than nothing, but, as usual, only works for English, and only for a single language in a document. Also only works for single words (because English!). So this is important, but will not work in a straightforward way.
5. He is keeping his database in a bib file. He adds records automatically, based on DOI, using a custom function, which, I suspect will not work in my case. But maybe that is what zotra should do.
6. He is also using Elfeed to subscribe to feeds. Noteworthy. A bit like: https://koustuvsinha.com/post/emacs_research_workflow/
7. He has an “org-db”, a, seemingly, full-text search for org-mode files. Interesting, if he is so keen on bloated stuff, why not just org-roam? But, on the positive side, it’s good that that org-db of his does not depend on a single directory for all notes.

I am not convinced of the benefits of using MySQL for bibliography management.

RefDB is a standard used by libraries to exchange bibliographic data. After brief skimming, and mentioning that the most recent version is from 2008, I suspect that unless you are really running a library, it is not worth using.

Okay, now we are getting somewhere.

Pure biblatex files are weirdly formatted database files with entries for “papers”, which are called “entries”. I am tempted to call them “entities”, because why would not I add there any kind of vaguely related stuff, such as theorems?

Let us see some examples:

Inlinetodos are added with C-c C-x t, and are really nice. It is good that I watched that video.

Reftex is built into Emacs, and is a mode for managing references, citations, labels, and index in LaTeX. Let us see if it can be abused for helping us do research in org-mode.

Seemingly, it assumes that you are editing a LaTeX project, and it can index files in that project, look for labels, and offer them for auto-completion, as well as lookup citations in the bib file.

In order for that to work, we need to agree on a certain pattern on using labels, and on what goes into the bib file.

As a side-note, the previous paper I did in org-mode didn’t have any labels.

C-c ( creates a label. C-c ) lets you choose a label, and inserts it. C-c [ inserts a citation with a key from a bib file.

I believe, there are some customisation variables in Emacs, which would let us insert org targets, links, and, for example, cite:bla links, although I am not sure it is the best way to do it.

Reftex has features for quick navigation within the project, C-c =, which in org can be partially substituted by (1) imenu, (2) collapsing org trees with TAB, (3) grep and isearch.

Reftex has support for index. Use C-c < to index an entry.

I think that indexing is quite underused in 2023. I have never seen scientific papers include an index, and I seldom use index in a PDF book, if I can search instead. However. I am thinking that an index, or, rather, a glossary, is that tool of data organisation that can be best used to establish cross-document relationships. As an example, let us assume someone is studying a certain subject, or, more realistically, tries to break through a difficult paper, and is trying to build his own way to have a solid foundation needed to understand a paper. One would necessarily read several books on the same subject, as well as subjects building on top of previous subjects, and establish inferential links.

A book has a natural representation of a graph with nodes-headings, paragraphs, and theorems. However, wandering over many not isomorphic graphs is stressing for the brain. Marking certain places as “interesting” would help to establish relationships between the graphs, and a certain “concept” node could server as a point of link concentration for them. This is why I am saying about a glossary (with bodies), not just an index.

So, the idea of ol-bibtex is that you keep a reading list, together with bibtex metadata, in an org file.

The nice thing here is that you can

1. use org’s search machinery to filter headings by properties and tags,
2. group books/articles by subtrees, say:

* Differential Equations
** Paper 1 :interesting:
** Paper 2 :boring:
* Measure Theory
** Paper 3 :interesting:
** Paper 4 :boring:

Not sure how org-bibtex machinery will work with tree-like headings. org-bibtex does not seem to export org tags as a “keywords” field in the resulting org file. (There seems to be a parameter for that org-bibtex-tags-are-keywords)

Bodies are also not exported neither as a “note” field, nor as an “annotation” field.

In general, I am doubtful that having all annotations for all papers in an org file is practical.

On the other hand, org-noter seems to integrate with this approach not too badly, as it expects a heading to keep all annotations for a file.

Okay, I still have not understood what this package does. It should somehow improve org exports to html, but I am still not sure how exactly.

This is a very straightforward package. It defines org links in the format of cite:key, which are later formatted either as html links, or as latex citations. You can also add the #+bibliography: plain t code piece, which will either use latex machinery to format bibliography, or use bibtex2html.

I have to stress, it expects a bibtex-file bibliography, not an org-file. Perhaps, when if we want to keep all our “ground truth” as org-files, we can add a hook to convert those files into bib files with org-bibtex.

Also, maybe, even a three-stage process might be useful: keep a bibliography in a file, use org-transclude to only include the required bibliographic entries into an org file, then export this file into a bib file and then org-export will be able to build both an html, and a pdf from it.

This section comes from org#Citation handling. This machinery in Org is relatively new, circa 2021.

So, firstly, we are getting one more NIH way of adding citations. That is, instead of relying on standard Emacs’ reftex, org developers introduced yet another key combo to insert citations, C-c C-x @. Why exactly is beyond comprehension, since reftex was written by Carsten Dominik, the guy who wrote the actual org-mode. It surely would have been much easier to update reftex, rather than write one more extra system doing the same thing.

It is looking at a file pointed to by #+bibliography: File.bib, which is at least a little consistent with the older ways.

Citations look like [cite:@key] links, and why exactly they need the @ symbol, I do not know, but in this case it works as expected, as we can keep our older ox-bibtex’s cite:key links without opening function conflicts. C-c C-o works, and opens an entry in File.bib.

oc.el supports some kind of “citation styles”, which are, I guess, useful to some people.

Exporting citations is handled by the keyword #+cite_export: basic author author-year.

Using the csl processor makes the exporter format everything manually, both in html, which is, I guess, okay, or, at least, I have not found it to be much worse than bibtex2html, but in LaTeX it looks super weird.

Writing #+cite_export: biblatex makes more sense, at least it is writing out \addbibresource{File.bib} and replacing citations with \autocite{key}. Exports to html, however, are not supported, and the exporter just prints LaTeX commands in place of proper references. I guess, it might not be too hard to patch it to do basically what ox-bibtex did in the past, exporting via bibtex2html.

Note (!) export to html, conflicts with ox-bibtex. If your config still includes ox-bibtex, you might want to remove it, or somehow tweak and debug.

So, in general, I have mixed feelings about all this new citations machinery in org. It is good that there is now a system with pluggable backends, but so far, documentation is lacking, and making it do what you expect, is not straightforward.

In particular, I wanted to write a disappointed comment here, but looking in the source code for oc.el, found the variable: org-cite-export-processors. This variable does not do what your expect it to do from the name. You might actually want to set it to something like this:

(setf org-cite-export-processors  ((beamer natbib)
                                (latex biblatex)
                                (t csl)))

Which means that beamer will process beamer files using pure TeX, latex-pdf will use biblatex, and html and all the seldom used formats will use csl. Looks almost as good as the “old” approach with ox-bibtex and bibtex2html.

A backend for org-bibtex, seemingly, still needs to be written. I can understand why they rejected reftex. After all, reftex is bibtex-only. What I do not understand is why they introduced a new kind of link, inserted with a separate keybinding, C-c C-x @, rather than just reusing org-link machinery. For example, there could be a cite@: link, and, depending on the value of #+bibliography:, C-c C-l would insert a citation from that database, using normal tab completion.

But anyway, it seems that with the new version, it should be possible to rewrite the old citation mechanism using just oc.el. This would make bibtex2html not needed, as well as ox-bibtex.

~bibtex-completion∼ lives here: https://github.com/tmalsburg/helm-bibtex

Org-bibtex users can also specify org-mode bibliography files, in which case it will be assumed that a BibTeX file exists with the same name and extension bib instead of org. If the bib file has a different name, use a cons cell ("orgfile.org" . "bibfile.bib") instead:

Really? org-bibtex? Not ol-bibtex? I asked a question https://github.com/tmalsburg/helm-bibtex/issues/438

Apparently, bibtex-completion is neither bibtex, nor completion. It is an abuse of the completion framework to be used as a GUI to the bibliographic database.

Okay, org-ref is, on the first glance, hugely over-engineered, but since it is omnipresent in so many org setups, I need to study it too.

1. The intro is here: https://github.com/jkitchin/org-ref
2. The manual is there: https://github.com/jkitchin/org-ref/blob/master/org-ref.org

From the intro it basically follows that org-ref was designed to facilitate citing and cross-referencing in Org-Mode. The intro itself now even says the citing part is, seemingly, outdated, as org-mode has a new citation mechanism.

What about the cross-references (intra-document links)?

Firstly, let us remember that org-mode has “targets”, that is pieces of text in double-angular brackets. <<target>>. That “target” will be interpreted as a “label” in LaTeX. The links to those targets can be created like target. Here is the link: 1.2.11.

Let us see how org-ref improves on it.

I will try to avoid copying all of the Ebib’s manual. In this subtree I will try to outline the main scenarios for it.

So, ebib supports adding citations to org, markdown, and latex, from within its own database.

This is a big drawback. Ebib has to be running in background.

It supports importing bib data using a package called “ibiblio”, and, I guess, if you have some other backend to fetch biblatex sources, into the bib file itself, it can deal with that too.

It has all the papers in the same giant list, with no categories, but it does support tags and keywords (Of which I do not remember the difference. I guess, tags are non-thematic, and keywords are thematic?)

It also supports one reading list, so I can imagine a workflow of being “a little bit” strict to oneself, and adding all reading material to the bibtex database, and interesting items to the reading list.

But how would I make org-exports work painlessly? Well, I guess, I can point org to the same bibtex file?

If I were not violently cross-referencing stuff I am reading, and if I were not violently reverse-engineering files of books, I could be able to force myself to do this kind of rigour.

A variation on the theme of “org file is ground-truth, and I export to .bib what I need”.

The interesting thing here is using pdf drag-and-drop, as well as a “map” for the org file, “org-imenu”.

Since there is at least another option for mapping org files, org-sidebar, I am confused why this is even a thing.

Small, simple, straightforward. Do (org-link-set-parameters "cite" :follow 'org-ebib-open) to make cite: links open in ebib. Nice if you use ebib.

Denote is a package whose main purpose is, seemingly, to keep backlinks to org files without the use of external databases.

Amsreftex is a way to keep a bibliography database right in LaTeX, with no bibtex and similar stuff involved.

Great news, but late for about 40 years.

Moreover, we want a reading list in org, or a vendor-neutral database, not a tex-specific thing.

On the other hand, amsrefs maybe still should be considered as a “ground truth” for the bibliography database.

Not sure it is good.

Okay, org-transclusion…

So, the idea is very noble, you should be able to include different pieces of org files into other org files.

Imagine the following:

You have a book on, say, a university course of Probability Theory. And you are reading a book on post-grad Probability Theory.

You might want to reverse-engineer the first book, and wherever you see a reference of, like “see Bla page 1 theorem Y”, you could write a transclusion command and obtain that theorem in your own text.

What might be even more fun is writing custom bibliographies, while transcluding entries from a “big database”.

So, suppose your setup is the following: you have many small org files with citations for various projects… and you org-transclude them into a single org file for a “large database”?

And then you use citar to import data from that “large” database.

Then each “project” will have “own” papers, that is, crushed during its progress, and “alien” papers, crushed during making other projects.

But you still need periodic upkeep for this “transcluded” database.

This package ended up here not so much because it is relevant to referencing, but because it is relevant to general task management.

If in your setup one org file corresponds to one project, and if it is possible to keep track of this project by just working with this file, and if you have upcoming events, and similar things, associated with this project, it might be worth having a look at it.

It provides a “sidebar” of two buffers, with tasks and scheduled events, relevant to current org buffer.

I suspect that using “speedbar” would make more sense?

Well…

org-roam certainly seems to have been done by people who do research.

It is basically a personal wiki with links and backlinks. p It does rely heavily on the ID property of the headings, to create an sql index, and backreferences. It is also incompatible with a lot of standard org’s machinery.

On the positive side, it does seem to do what I have always wanted to do: generate a map of links.

When I started deciphering mathematical papers, I really wanted to have a system to link different concepts together. The problem is (1) when I am reverse-engineering a paper, I am most likely not doing it in the org-roam directory, (2) I seldom invent my own concepts, and when I do, they are uploaded to my website, and hence are also not getting into the org-roam directory.

On the other hand, it seems that org-roam can really be used as a backend for the citation machinery of org. I already mentioned the following “ground truth” backends:

1. A single bibtex file.
2. Many bibtex files in a directory.
3. Many scattered bibtex files over many directories.
4. Single org-file (exportable into bibtex with ox-bibtex)
5. Many org-files, also exportable, in one directory.
6. Many scattered org files.

The option number 5, seemingly, can be extended to have all those files in an org-roam directory, and, I guess, bodies can have annotations for the papers, created with org-noter.

“Doctor, when I do it like this, it hurts!” “Then don’t do it. Next!”

After reviewing 17 out of 50 documents, I started to get some thoughts about what a research system should do.

I am tempted to say that a basic unit of the system is an “article”. An article can have several forms:

1. PDF
2. HTML
3. TeX
4. Semantic markup, such as org or markdown, which is the most useful option.

The interesting thing is that these kinds of content can be transformed one into another. Semantic markup can be freely compiled into PDF or HTML, and PDF can be converted into Semantic markup using OCR (mathpix.com does amazing things).

We need a system which can track at least those three kinds of content.

But that is not enough.

In addition to “different presentations”, a piece of knowledge has metadata. Supposedly, we can deal with biblatex’s fields to “describe” a document exhaustively.

Among the metadata, at least “annotation” is a very useful field which need to be written for each processed paper. You might call it “lightly embedded” into the brain, because often the depth needed to write a decent annotation is still not profound enough to understand all of the paper. You can speculate that an “annotation” is what “Mathematical Reviews” or “ZbMATH” are doing. I guess, if a written annotation is not to be openly published, you can just type in a URL into the annotation field.

If we have a database of “articles”, a database of pieces of knowledge, we, quite naturally, might want to interlink them.

This would be mimicking the Web, or human (or, rather, artificial) brain, or some other semantic network.