1. https://source.android.com/docs/setup/download/source-control-tools
2. https://www.akshaydeo.com/switching-branch-with-repo-for-android-source-code/
3. https://source.android.com/docs/setup/reference/repo

repo is a tool to fetch Android. It is essentially a different (to git submodule) approach to multi-repository source trees.

You initialise it by running

repo init -u https://mirrors.tuna.tsinghua.edu.cn/git/lineageOS/LineageOS/android.git -b lineage-23.2  --git-lfs

The same command should also work when switching versions. In theory it is enough to just replace the branch name.

There are tricks, however. repo is not error-resistant. When it fails, it does not take care to restore the original state of the source tree, it just fails, often leaving individual git repos in an inconsistent state. You might have to repair those repos manually, usually by aborting merges, committing uncommitted files, or something like that.

repo is basically just helps maintaining a list of git repositories, revisions, and remotes, and calls git commands for individual repos.

The files to study to understand it are: ./repo/manifests/default.xml and ./repo/.repo/local_manifests/local_manifest.xml.

The first one is the list of repos for the main tree of Android, and the second one is for you to add references to repos specific to your device.

Remote server definitions are quite self-explanatory, except, perhaps, a special path called "..". This is a parent directory with respect to the default repo.

There is an XML tag for “default settings”, called default:

< default  revision= "refs/heads/lineage-23.2"
          remote= "github"
          sync-c= "true"
          sync-j= "4" />

Official build system documentation is so terrible that I am not even linking it here, as it’s completely pointless.

It is convoluted and consists of several mutually duplicating parts.

You might hear the words “Soong”, “CKati”, “Jack”, and “Bazel”. There are also shell functions (pay attention, they are not files!, they are bash functions), which are expected to control the build, and they also seem to have bit-rotted a bit.

First of all, go to build/make, and add a pair of commands to enable ccache when building Android.

diff --git a/envsetup.sh b/envsetup.sh
index 12a69fc77b..d20f01aac7 100644
--- a/envsetup.sh
+++ b/envsetup.sh
@@ -15,6 +15,10 @@
 # gettop is duplicated here and in shell_utils.mk, because it's difficult
 # to find shell_utils.make without it for all the novel ways this file can be
 # sourced.  Other common functions should only be in one place or the other.
+
+export CC_WRAPPER=/usr/bin/ccache
+export USE_CCACHE=1
+
 function _gettop_once
 {
     local TOPFILE=build/make/core/envsetup.mk

Or you can add the same lines to your own bash build script, which is expected to source the aforementioned source build/envsetup.sh. This is documented somewhere, but the page mentioned does not seem to have this any more.

The bash functions you might find are the following:

1. m
2. mm
3. make
4. mka
5. breakfast
6. brunch
7. lunch

In theory they were expected to have their own meaning, in particular, breakfast is expected to select the device configuration for building. However, in my experience you usually only have a single device in the source tree anyway, so you do not need it.

There should be a way to switch between debug, userdebug, and release configurations with these:

lunch lineage_--

Where codename is your device code name (you have to find it out!), and it is not the same code name as board code name. For example for my device the code name is xigua (Chinese for Watermelon), and the board name is kalama (a city name).

But in my case just typing brunch xigua was enough.

In any case, in order to just run the build with a single command, I wrote a bash script:

 export  CC_WRAPPER=/usr/bin/ccache
 export  USE_CCACHE=1
( reset
rm out/target/product/xigua/lineage-23.0-*-UNOFFICIAL-xigua.zip*
source  build/envsetup.sh
(  cd  ~/Lineage-Camera/cam-oplus-15/
 ./mine_repack-cam.bash ||  exit 1
  exit 0 ; )
(  cd ./vendor/oplus/camera/
  ./extract-files.py --no-cleanup  ~/Lineage-Camera/009_15.0.0.860 ||  exit 1
   exit 0 ;  ) ||  exit 1
printf  "mine:camera files extracted\n"
(  cd device/oneplus/xigua
  ./extract-files.py --only-target --no-cleanup  ~/Lineage-Camera/009_15.0.0.860 ||  exit 1
   exit 0 ; ) ||  exit 1
printf  "mine:device files extracted\n"
 export  CC_WRAPPER=/usr/bin/ccache
 export  USE_CCACHE=1
brunch xigua  ||  exit 1
 exit 0 ; )
 RETVAL=$ ?
 if (( RETVAL == 0 )) ;  then
  espeak  'compilation finished with success'
 else
  espeak  'compilation finished with failure'
 fi

This build script is more complicated that I mentioned, but there are reasons for that.

1. There is a need to extract binary drivers from the original firmware.
2. There is a need to modify the camera app so that it would run on LineageOS.

It might happen that you do not actually need (2), but you will probably need (1).

This is documented here quite decently: https://wiki.lineageos.org/extracting_blobs_from_zips_manually.html

1. Android Studio :: https://developer.android.com/studio
2. Android Studio for Platform :: https://developer.android.com/studio/platform
3. Gradle :: https://gradle.org/
4. Emacs + jdtls

I need to mention these, because they exists. I did not use them much, except for writing a small polyfill library, so I cannot produce a meaningful critique. In general, Android Studio is quite a decent tool, it is a version of JetBrains IDE, so, basically, the best IDE available in the ecosystem. It is a safe choice, unless you are good enough of a developer to assemble your own tool-belt, based on Emacs.

If you want to setup Android Studio for Platform, see the howto file in the LineageOS repo: ./lineage/wiki/pages/how-tos/import_to_android_studio.md

I use Emacs for development, but even though Emacs supports Eclipse JDTLS LSP server, it does not support Android java, so I ended up doing find+grep (fd+rg), like in the old times.

1. https://source.android.com/docs/setup/build/building-kernels

Okay, I know nothing about building the kernel for LineageOS, except that it is built with bazel, and is shipped in three source tree modules:

1. kernel//sm8550
2. kernel//sm8550-devicetrees
3. kernel//sm8550-modules

Kernel is one of the most interesting things in any Linux system, but I did not need to deal with it yet.

The only thing to note is that Android is not using udev to determine hardware configuration in runtime, it requires that all system devices be declared in advance in a data structure called “device tree”, which I know nothing about except that it is compiled and that it is also used in Linux on Raspberry Pi.

About modules I also know nothing except that, as far as I understand, the kernel has to be Open Source, as it is under GPL, but modules do not, and a lot of phone manufacturers are providing drivers for their devices only as pre-built modules.

The entry to the kernel build system is vendor/lineage/build/tasks/kernel.mk.

This is another incredibly badly documented part of Android, even though it is central to its architecture. ( https://source.android.com/docs/core/architecture/vintf)

So, Android is an operating system, and it needs to talk to hardware. But it does not want to talk to hardware directly, querying the drivers, or udev or /proc or /sys, it wants to talk to userspace daemons over Binder, and those daemons can implement the userspace<->kernelspace interaction in any way they want. So it collects a list of requirements , for some obscure reason called compatibility matrix into an XML file, and expects to compare that XML with a device manifest, which is also an XML file, in a similar format, which collects all capabilities of a particular device, which is called a manifest. The same thing also happens in another direction, so we have framework manifest, framework compatibility matrix, device manifest, and device compatibility matrix.

The most confusing thing is that both of those things are happening in the same source tree!

All four XML files are compiled by the build system, and compared to each other to find if some services are not provided.

I did not have to write them myself, but you probably will, if you want to port Lineage to your own device. I can only recommend looking at how this VINTF is declared and requested in device/oneplus/xigua.

Device manifest files are usually placed into “ vendor/etc/vintf”, or “ odm/etc/vintf”, I don’t know which one goes where, and I don’t know how to tell the build system that “these XML files are for the VINTF, and these are not”.

This part is also documented poorly and I did not understand it.

As mentioned above, Binder services have to be declared in the “VINTF” XML files, and there should be services, implemented as Linux binaries, started by init from /{vendor,odm,system,system_ext}/etc/init/servicename.rc, which provide their services via Binder calls.

Binder has to do with memory mapping, but it’s not just a place in memory into which you can write some garbage, it’s more like Sun-RPC, there are two versions of the library, and there is a serialising-deserialising server between them. ( https://en.wikipedia.org/wiki/Sun_RPC) In Sun-RPC, there is an RPC compiler, rpcgen, which creates .c-files from .x files, which can be compiled and linked into the client and server programs, so that they would call functions and do not deal with serialisation and network transfers themselves.

Android Binder does something similar, but instead of rpcgen, there are hidl and aidl.

But I cannot actually give you an example of a three-stage compilation: AIDL, Server, Client, because I have not found it, and I didn’t actually need it for anything.

But AIDL services can be called both from C++, and from Java, and somehow even from apps. I wonder, how AIDL interfaces are getting to apk writers?

In any case, services not just have to be declared in the VINTF, but also registered when they start, with “service manager” or “hw service manager”. I don’t know how this is done, but I have seen a mention of this thing.

But there is more to it. There is such a thing as SP-HAL. SP stands for Same Process.

This means that you can just link to a library, or dlopen a library, and use its functions. They do not have to actually do anything with hardware… I think.

In any case, I have no idea whether all libraries supplied by Android are a part of SP-HAL, or not. Again, all of this is very confusing and poorly documented. If you can explain this to me in a consistent way, feel free to email me.

So, as mentioned in the previous section, Android cares a lot about abstraction barriers between different parties involved into the creation of a device. In most cases there are at least three parties:

1. Google (framework)
2. BBK/OnePlus (ODM)
3. Qualcomm (vendor)

Google thinks that the interactions between them three should be if not tightly controlled, then at least well declared, and hence introduces VINTF.

Now believe me or not, but each of those parties has a partition allocated specifically for itself, and since Project Treble even two (suffixed _a and _b).

Wait, I didn’t say anything about partitioning on Android before. This should be a whole section dedicated to it, but so far I can tell you that “vendor”, “odm”, and “system_ext” are separate partitions. Moreover, at least on my phone, both vendor and odm are read-only images of an ext2 file system, and cannot be remounted read-write, only rebuilt.

Why is that? Well, because the three parties do not communicate to ensure common library interface, and end up with API and ABI incompatibility issues. So, the decision ended up being: give each party its own set of basic Android libraries, and make the dynamic linker ensure in runtime that programs associated with vendor, say, the camera provider services (a process run by init, which is allowed to open /dev/video0 and which services captured frames via a Binder service registered with the servicemanager) can only link dynamically and dlopen .so files from /vendor/lib64, even if it is given an LD_LIBRARY_PATH or LD_PRELOAD.

Linker also enforces restrictions on which libraries can be accessed by .so libraries shipped with apk files.

Now the important question: Which libraries can be used where? Does anybody have a list?

1. https://source.android.com/docs/core/architecture/configuration/add-system-properties

So, Android has a unified configuration system called “properties”. Properties are essentially a global key-value database, accessible through the /proc filesystem, where they are represented as files, which allows them to have SELinux labels. Properties are accessible to console programs with getprop, setprop, and __android_property_get, and through Java API.

They are stored in

1. /default.prop
2. /system/build.prop
3. /system/default.prop
4. /vendor/build.prop
5. some other *.prop files

This part, surprisingly, actually has a decent HOWTO about it written by Google, so I will just link it here:

https://source.android.com/docs/core/architecture/configuration/add-system-properties

I will note that properties are divided into “system” properties and “vendor” properties, distinguished by specific prefixes.

ro. properties are writable only by init, or by the build system. persist properties survive a reboot.

In the build system, properties are defined either with PRODUCT_{PARTITION}_PROPERTIES (in the Makefile), or in TARGET_{PARTITION}_PROP (in separate files).

The biggest question is “why are properties different by partition?”. I don’t know, but if your program or app cannot read or write a property, probably it needs to be moved to a different partition.

Properties have SELinux labels, and those are defined in the files called property_contexts, for example ro.vendor.oplus.camera.frontCamSize u:object_r:vendor_camera_prop:s0

The full list of property contexts is probably somewhere under system/sepolicy or in Qualcomm directories.

Now the good idea would be to make an exhaustive list of all properties with explanation what those properties do. The list can be obtained by running just getprop, but what the meanings… Maybe at least make a wiki page?

I know nothing about this, except that there is a data structure called AHardwareBuffer, and you can manipulate it a lot.

SurfaceFlinger is the Android GUI system, I guess, the only one which can read/write /dev/drm or /dev/dri.

I don’t know anything about them.

1. https://lineageos.org/engineering/HowTo-SELinux/
2. https://source.android.com/docs/core/architecture/aidl/aidl-hals#sepolicy
3. https://stackoverflow.com/questions/36790794/what-is-c512-c768-of-selinux-process
4. https://rtx.meta.security/reference/2024/07/03/Android-system-apps.html
5. https://github.com/SELinuxProject/selinux-notebook/

SELinux is decently documented by “SELinux Notebook”, and the basic principle is not too hard, but the implementation is very convoluted.

SEAndroid (a modification of SELinux for Android) is decently described here: https://lineageos.org/engineering/HowTo-SELinux/

Basically, SELinux works like this: source x target x operation => allow, and everything else is denied.

Firstly, the language is very confusing. Essentially, there is not difference between labels on source and target, so I am calling them “labels”. This is the most visible in the following example: suppose a project A wants to kill project B, then a check would be scontext=label_a tcontext=label_b class=process acton=kill. When a process B wants to kill process A, the action would be the same, but scontext and tcontext would be reversed.

However, you may often see the words “domain” and “context”, which are basically the same thing – labels.

In Android, many things have labels:

1. processes/apps
2. properties
3. files
4. pseudo-files

Apps are labelled in files called seapp_context, properties are labelled in property_context, and files in file_context. Permissions are given in /.te files.

Of course, writing all permissions and re-flashing the system is a pain, so there is a tool called sepolicy-inject, which allows injecting policies into the running kernel. File contexts can be changed with chcon, but process/app contexts have to be rebuilt.

SELinux has essentially seven types of statements:

1. allow :: permit an operation and print its denial in log
2. neverallow :: prohibit an operation
3. auditallow :: allow an operation but print log
4. dontaudit :: prohibit an operation and do not print its denial in log
5. define :: define a macro with arguments which will expand to something
6. type :: define a new label
7. typeargument :: add property to a label

define is tricky, because it usually allows a lot of permissions, while prohibiting (with neverallow) some actions which can only be allowed via this macro This make a naive policy of “Enable SELinux, look for denials in the log ( dmesg), allow as needed” impractical, because while you know that you need to allow something, you do not necessarily know the correct macros for allowing that operation, and allowing the actions directly will be prohibited via neverallow.

For the list of macros, see:

1. system/sepolicy
2. find hardware -name '*.te'

A convenient pattern for debugging SELinux is to find an object you are interested in (say, a camera app), give it a separate label (say mycamera_app) in seapp_contexts, and look at dmesg for the actions denied to it.

As a last topic in this chapter, I want to touch partitions and Android boot process.

In fact, in most operating system manuals this is the first chapter, combined with “how to install an OS”, but Android is a little different in that:

1. most people do not to install Android themselves,
2. installing Android is difficult, the whole chapter 1.4 is dedicated to this,
3. boot process and partition details vary significantly between different manufacturers

In principle we know how a system should boot:

1. BIOS finds a bootloader
2. bootloader finds an OS kernel and root file system
3. bootloader boots the OS kernel, passing the root file system location as a command-line
4. kernel mounts the root filesystem, sometimes additional file systems, and starts the init, which does all the initialisation work

But Android is different. Indeed, there are partitions, and in theory they are on the MMC storage, there is a bootloader and a kernel, but how exactly they are implemented, seems to depend on the manufacturer a lot.

It should be possible to see the full list of partitions on a device … But the method how see them differs from a manufacturer to a manufacturer.

And what is also bad, their amount is significantly greater than on a normal OS.

/dev/block/by-name  #  ls
ALIGN_TO_128K_1    fsc             oplusdycnvbk    storsec
ALIGN_TO_128K_2    fsg             oplusreserve1   super
DRIVER             hyp_a           oplusreserve2   toolsfv
abl_a              hyp_b           oplusreserve3   tz_a
abl_b              imagefv_a       oplusreserve4   tz_b
aop_a              imagefv_b       oplusreserve5   tzsc
aop_b              init_boot_a     oplusstanvbk_a  uefi_a
aop_config_a       init_boot_b     oplusstanvbk_b  uefi_b
aop_config_b       keymaster_a     param           uefisecapp_a
apdp               keymaster_b     persist         uefisecapp_b
apdp_full          keystore        qmcs            uefivarstore
apdpb              last_parti      qupfw_a         userdata
bluetooth_a        limits          qupfw_b         vbmeta_a
bluetooth_b        limits-cdsp     qweslicstore_a  vbmeta_b
boot_a             logdump         qweslicstore_b  vbmeta_system_a
boot_b             logfs           rawdump         vbmeta_system_b
cdt                mdcompress      recovery_a      vbmeta_vendor_a
connsec            mdm1oemnvbktmp  recovery_b      vbmeta_vendor_b
cpucp_a            mdtp_a          rtice           vendor_boot_a
cpucp_b            mdtp_b          rticmpdata_a    vendor_boot_b
ddr                mdtpsecapp_a    rticmpdata_b    vm-bootsys_a
devcfg_a           mdtpsecapp_b    sda             vm-bootsys_b
devcfg_b           metadata        sdb             vm-data
devinfo            misc            sdc             vm-persist
dinfo              modem_a         sdd             xbl_a
dip                modem_b         sde             xbl_b
dsp_a              modemst1        sdf             xbl_config_a
dsp_b              modemst2        secdata         xbl_config_b
dtbo_a             multiimgoem_a   shrm_a          xbl_ramdump_a
dtbo_b             multiimgoem_b   shrm_b          xbl_ramdump_b
engineering_cdt_a  multiimgqti_a   splash_a        xbl_sc_logs
engineering_cdt_b  multiimgqti_b   splash_b        xbl_sc_test_mode
featenabler_a      ocdt            splash_odm
featenabler_b      oplus_sec_a     spunvm
frp                oplus_sec_b     ssd

The _a/_b partitions are an implementation of a system called “Project Treble”, which was expected to help phone manufacturers deliver updates to their phones faster. The idea was that system, odm, and vendor partitions could be updated independently, and if an update fails, the phone reboots using a dual partition _b if _a failed, and vice-versa.

In theory this would allow booting separate operating system images while keeping the driver partitions odm and vendor intact. I am not sure this actually works anywhere except Google’s reference devices (Pixel).

So, in theory, a Treble-enabled phone has at least two system “root” partitions to boot, but there are actually more. Many devices are able to boot into

1. bootloader
2. fastboot
3. fastbootd
4. edl
5. recovery
6. sideload
7. system

The bootloader, in theory, should be independent from the MMC storage and be bootable regardless of a system installed. Fastboot is a “debugging interface”, usable through a command-line tool “fastboot”, which should allow re-flashing the system or individual partitions. It is often combined with Fastboot into the same interface. Usually it is also not kept in the main MMC.

Recovery is a stripped-down version of Android, using the same kernel and loadable into ram filesystem. It usually has some kind of interface, but its main purpose is to be able to repair Android in case it is broken. Together with recovery, there is also “fastbootD” (note the D). It is a system implementing the fastboot protocol, but running from a booted system, such as recovery. It is used to flash partitions which are not real partitions, but are read-only files (file system images) mounted loopback.

Note that the list of partitions above does not have a system partition. This is because the system partition is not real.

For some reason, Android really likes having different components of the boot process as separate partitions.

For example, on many phones you install Android like this:

fastboot flash boot boot.img
fastboot flash init_boot init_boot.img
fastboot flash vendor_boot vendor_boot.img
fastboot flash dtbo dtbo.img
fastboot flash recovery recovery.img
fastboot reboot sideload-auto-reboot
adb sideload lineage*.zip

fastboot reboot sideload might be replaced with fastboot reboot recovery and tapping “install update”.

1. boot is essentially the kernel
2. init_boot is the ramfs with initial userspace for the booted kernel
3. vendor_boot is a similar ramfs with vendor files
4. dtbo is the “device tree” describing the system you are running on
5. recovery is also a ramfs with a the few recovery options and an interface to install the main system

Android is not Windows, and ARM is not x86.

In general, ARM is a much worse of a system, restrictive, repressive, and limiting. While on x86, it is possible to query the system for a lot of things about itself, ARM systems have not ACPI, or BIOS, or EFI. Well, something like an EFI does exist, but is not widespread.

As a result, firmware (like, real firmware, not the Android system) is often only accessible through secret tools sold by the manufacturer for a lot of money.

Sometimes such tools are available to download though, and they are not user-friendly.

These tools are often used by repair shops to restore completely broken devices. The boot mode used by such tools is sometimes called EDL. Some bootloaders allow booting into EDL manually.

1. /system :: system files
2. /system_ext :: emm… more system files?
3. /odm :: brand-related (say OnePlus) things
4. /vendor :: chipset manufacturer (say, Qualcomm) related things
5. /data :: data
   1. /data/data/ :: app data. Usually only one sub-directory there is available for an app to write files, say /data/data/com.termux
   2. /data/data/adb :: directory to place adb-related files
   3. /data/tombstones :: crash minidumps
   4. /data/vendor/camera :: camera intermediate directory and debug dumping directory
6. /apex :: sub-tree to mount specific sets of versions of system libraries, for compatibility reasons
7. various *etc* directories :: configuration files
8. various *lib64* directories :: shared libraries
9. various *bin* directories :: binary executable files

1. Background reading: https://www.kernel.org/doc/Documentation/filesystems/sharedsubtree.txt
2. https://android.stackexchange.com/questions/203951/how-can-i-make-a-symlink-or-equivalent-inside-storage-emulated-0/
3. sdcardfs : https://www.xda-developers.com/diving-into-sdcardfs-how-googles-fuse-replacement-will-reduce-io-overhead/
4. https://android.stackexchange.com/questions/197959/why-partition-gets-unmounted-automatically-after-some-time/200449#200449
5. https://android.stackexchange.com/questions/214288/how-to-stop-apps-writing-to-android-folder-on-the-sd-card/
6. https://android.stackexchange.com/questions/217741/how-to-bind-mount-a-folder-inside-sdcard-with-correct-permissions/217936#217936
7. https://developer.android.com/training/data-storage#scoped-storage

Android has suffered a series of transformations from one build system to another, and the current one is sort of a mess. I have not personally met Bazel, even though it is believed to be somewhere inside there too, but there are Android.mk and Android.pb to be aware of.

Both are a variation of classical UNIX Make, you define targets, connect them with dependencies, and the system traverses the dependency tree and builds the resulting object. Android.mk is essentially a Makefile, but it is parsed by ckati into a Ninja script rather than executing the statements directly. Android.bp is also parsed into Ninja, but by a different tool.

You probably will not have to tweak Ninja scripts at all, but Make and Android.bp files are interesting, and you will probably also not call ckati or soong (the proper name of the build system) directly.

Good for us, Android.bp kinds of targets are well documented:

1. Syntax :: https://source.android.com/docs/setup/reference/androidbp
2. Modules :: https://ci.android.com/builds/submitted/13288697/linux/latest/view/soong_build.html

Android.mk is not so well documented, especially since it is expected to be phased out eventually.

Nevertheless, Make files are still very important, as they are used for device definitions, and you will have to write our own one (or adapt someone else’s) in order to successfully port Lineage.

1. device :: mostly things related to a particular device, say, Samsung Galaxy S10
   1. device/oneplus/sm8550-common :: source files related to all devices by OnePlus built on Qualcomm sm8550
   2. device/oneplus/xigua :: source files related to a particular device. Will most likely inherit from a common-phone.mk profile
2. vendor :: mostly related to things extracted from original firmware
   1. /vendor/oneplus/sm8550-common :: binaries from original firmware, which are chipset-related
   2. /vendor/oneplus/xigua :: ditto, concrete device related
   3. /vendor/lineage/... :: despite being in vendor, not really binaries, actually I don’t know what it is, it makes little sense
3. kernel :: kernel-related directories
4. frameworks :: system libraries available to Java apps
5. packages :: pre-built apps
6. hardware :: em… some more drivers, but not chipset-related? I am confused
7. external :: things used in Android transparently, for example, Linux libraries, such as libxml2
8. system :: system libraries and services unavailable to Java apps

In general, you need three components for a device:

1. Device module (say, device/oneplus/xigua.mk)
2. Device binaries (say, vendor/oneplus/xigua.mk), usually extracted from an official build
3. Device kernel
   1. Kernel itself kernel/oneplus/sm8550
   2. Modules kernel/oneplus/sm8550-modules
   3. Device trees (dtb files) kernel/oneplus/sm8550-devicetrees

Kernel, modules, and dtbs is often published by the manufacturer, for example:

https://github.com/OnePlusOSS/android_kernel_modules_and_devicetree_oneplus_sm8550/tree/oneplus/sm8550_b_16.0.0_ace_2_pro/

But do not be so naive to believe that just plugging those into your LineageOS tree will make it work instantly.

Let us have a look at the files in a typical device module: device/oneplus/xigua/.

 #  ls
Android.bp
AndroidProducts.mk
BoardConfig.mk
board-info.txt
device.mk
extract-files.py
lineage.dependencies
lineage_xigua.mk
odm.prop
overlay
overlay-lineage
proprietary-files.txt
proprietary-firmware.txt
setup-makefiles.py
system_ext.prop
vendor.prop

As far as I understand, their meaning is the following:

1. AndroiProducts.mk :: defines a “device” for the purposes of the Android build system.
2. lineage_xigua.mk :: defines a module, which actually defines the device
3. BoardConfig.mk :: defines … well, I don’t know what it defines and what is the need for it to be separate from lineage_xigua, but it includes Makefile from vendor/oneplus/xigua, which is the module for pre-built drivers, so it is obviously very important.
4. device.mk :: seemingly also important, because defines a lot of device properties on part with lineage_xigua.mk and BoardConfig.mk. I have no idea why there are three files.
5. lineage.dependencies :: tells the system that your device is actually designed on the basis of a System-on-Chip which is used for other devices as well and some drivers can be shared
6. board-info.txt :: I have no idea what it is, but it includes the code name of my SoC
7. extract-files.py :: extracts binaries from an unpacked original firmware and post-processes them.
8. setup-makefiles.py :: just calls extract-files.py
9. proprietary-files.txt :: a file from which extract-files.py takes information on which files to extract
10. proprietary-firmware.txt :: I have no idea what it is, but it is probably somehow connected to the partitions on the device which are not Open Source, such as modem firmware
11. {overlay,overlay-lineage} :: no idea what they are for
12. Android.bp :: empty

If you want to port Android to a device which is to an extent similar to some other device already supported, chances are that you only need to find the true board name to inherit in BoardConfig.mk, and adjust the parameters in the files to match your device.

Extracting the correct binary files is very important for making the device run.

In order to do that, it is important to understand which files serve which purpose and correspond to which packages. As has already been said, there is no way to identify that using normal Android means, although I suspect that hacking something on the basis of xattrs should be possible.

Nevertheless, if you look at proprietary-files.txt, and extract-files.py, you will see that those files do more than just copy files into a temporary directory.

There are a lot of examples on how to rewrite files documented in tools/extract-utils/templates/single-device/extract-files.py, but I will note the few most important here.

1. .regex_replace(from, to) replaces a string
2. .binary_regex_replace(bytes.fromhex('cafebabe'),bytes.fromhex('deadbeef')) replaces a binary sequence
3. .replace_needed('libfoo.so', 'libbar.so') replaces a dependency
4. .fix_soname() fixes an SONAME if the .so metadata

The last two are very important for the case when you need to link an app to an original version of a library, and copy it into /odm/lib64/libfoo_vendor.so.

1. https://docs.qualcomm.com/doc/80-70017-3/topic/features.html
2. https://wiki.postmarketos.org/wiki/Device_Tree_(dtb)

I don’t know what to write here, I am ignorant, but I never had to write a device tree (dtb), or overlay it with some customisations with a device tree overlay (dtbo).

Look at the examples of real device trees in kernel//-devicetrees

You can try decompiling your own phone device trees:

 # !/bin/ bash 
mkdtboimg dump dtbo.img -b mydtb
 for i  in mydtb.* ;  do
  dtc -I dtb -O dts  "$i" -o  "${i/mydtb/mydts}"
 done

But the result will be megabytes in size.

1. https://github.com/Heydarchi/AIDL-HAL-Service

Missing place.

Presumably you need to write an “*.aidl” file, add it to Soong, then write the server and the client for the service and include the headers generated by Soong. But that is as much as I know.

But there seems to be at least one working example: https://github.com/Heydarchi/AIDL-HAL-Service

To begin is always the hardest thing for a programmer. You change one line in a huge project, and the crystal edifice shatters into myriads of tiny shards. At best itn starts to fail building. At worse, it builds, but begins to misbehave in a subtle way. At worst, it works, seemingly, as intended, but either leaks memory or becomes horribly insecure.

This is why having an introductory manual is so important.

1. https://lineageos.org/engineering/HowTo-Debugging/

Of course, when you are writing code, everything can go wrong. You need to debug it.

Basically, in software there are just two ways of debugging: logging and debuggers.

The following are the main log sources in Android:

1. dmesg
2. logcat
3. /data/tombstones
4. /tmp/recovery.log (in recovery)

Quoting Lineage’s guide:

For apps crashing, logcat is your best choice. For native code, look at the “tombstones”, at least they are going to have stack traces.

1. https://github.com/NationalSecurityAgency/ghidra/issues/6386#issuecomment-2455257933
2. Debugging Android aarch64 with Ghidra :: https://blog.nviso.eu/2024/01/15/deobfuscating-android-arm64-strings-with-ghidra-emulating-patching-and-automating/

Okay, I do not really understand this, but there are some compatibility issues between different versions of Android, SDK and NDK. This is so convoluted that I do not know how to understand it, so you have to search.

When you want to port an app for LineageOS, you usually only have two options: (1) modify the app, (2) modify the Operating System.

In both of those cases you want to make your development iterations as short as possible.

I already mentioned this, but it is never hurts to repeat:

1. apps can be signed with the platform key and copied
2. native libraries can be copied to /data/local/tmp and mounted over
3. selinux permissions can be injected using sepolicy-inject

First of all, you need to make sure that working with code is convenient, and stupid useless security theatre is not getting in the way.

So, the first thing you need to do is turn off ADB authentication to avoid tapping stupid permission windows on each re-flash.

go to vendor/lineage and apply the patch.

diff --git a/config/common.mk b/config/common.mk
index abc2b9ec..69aff098 100644
--- a/config/common.mk
+++ b/config/common.mk
@@ -33,7 +33,7 @@ ifdef WITH_ADB_INSECURE
 PRODUCT_SYSTEM_DEFAULT_PROPERTIES += ro.adb.secure=0
 else
 # Enable ADB authentication
-PRODUCT_SYSTEM_DEFAULT_PROPERTIES += ro.adb.secure=1
+PRODUCT_SYSTEM_DEFAULT_PROPERTIES += ro.adb.secure=0
 endif

 # Disable extra StrictMode features on all non-engineering builds

Also, I added the following lines to my device/oneplus/xigua/device.mk:

PRODUCT_PROPERTY_OVERRIDES += \
    ro.debuggable=1 \
    persist.service.adb.enable=1 \
    persist.service.debuggable=1 \
    persist.sys.usb.config=adb \
    ro.adb.secure=0

Again, this is making a lot of fiddling with the settings after each system reboot unnecessary.

Some debugging does not require complicated techniques.

Very often you can see what an app needs right into the adb logcat output.

In order to make your life simpler, you can edit the Android log-printing functions to produce more information.

For example, when I was debugging “Vendor Tags”, I added the following line to the code:

  status_t  VendorTagDescriptor:: lookupTag( const  String8&  name,  const  String8&  section,
                                          /* out */  uint32_t*  tag)  const {
+  ALOGE( "my-own:%s: Section: '%s', name: '%s'.", __FUNCTION__, section.c_str(), name.c_str());

This line appears in “ADB logcat” and is easy to filter. ALOGE is essentially Java’s Log.e(TAG, message);

Install your app, run it, look at logcat, dmesg and /data/tombstones.

Often it will tell you what is missing.

Another “naive”, but useful and practical tools are grep (or ripgrep) and strings. Those let you search binary files for the presence of content.

Look for something in the log, grep for this string, find out which binary contains it.

A lot of problems with an app running on Lineage are coming from Lineage just missing some libraries that the manufacturer is shipping in his firmware.

Firstly, you can unpack the apk with apktool and examine its shipped JNI and native libraries:

look at the lib directory and see which libraries the apk is carrying.

You can examine those libraries with objdump, readelf, and ldd. Unfortunately, Android’s ldd is unreliable, and it is listing all possible dependencies, transitively.

But readelf -d libfoo.so | grep NEEDED will give you a list of dependencies.

I have written a small script in Scheme to plot those dependencies as a tree:

#!/usr/bin/chibi-scheme -q
( import (chibi shell))
( import (chibi filesystem))
( import (chibi show))
( import (srfi 69))
( import (srfi 113))
( import ( only (scheme base) when))
( import (chibi pathname))
( import ( only (srfi 1) zip filter))
( import (srfi 27))
( import (srfi 95))
( define  to-string
  ( lambda (num)
    (substring (show #f (written num)) 0 4)))
( define  graph-file  "dep-graph.dot")
( define  stopfiles (set equal?
                   "/apex/com.android.runtime/lib64/bionic/libdl.so"
                   "/apex/com.android.runtime/lib64/bionic/libc.so"
                   "/apex/com.android.runtime/lib64/bionic/libm.so"
                   "/system/lib64/libc++.so"
                   "/system/lib64/liblog.so"
                   "/system/lib64/libz.so"))
( define  processed (set equal?))
( define  processed-vertices (set equal?))
( define ( get-dep-names file)
  ( map ( lambda (x) (substring x 1 (- (string-length x) 1)))
   (shell->string-list ( "adb"  "-s" a740e4ea shell readelf -d ,file) (grep  "Shared") (awk  "{print $5;}") )))

( define ( get-dep-paths filepath)
  ( let* ((extractor ( lambda (x)  ;; :/apex/com.android.i18n/lib64  /vendor/lib64/hw:/vendor/lib64
                      (shell->string-list ( "adb"  "-s" a740e4ea shell ,(string-append  "LD_LIBRARY_PATH='/apex/com.android.i18n/lib64"  "'"  " ldd  " filepath)) (grep  "-v"  "vdso") (awk ,(string-append  "{print $" x  "}")))))
         (deps (extractor  "1"))
         (paths (extractor  "3")))
    (zip deps paths)))

( define  systemp ( lambda (target)
             (equal? (substring target 0 (string-length  "/system"))
                      "/system")))

( define  max-depth 7)

( define  walk-tree
  ( lambda (path depth)
    (show #t  "Processing " path  " at depth " depth  "\n")
    (set-adjoin! processed path)
    ( if (< depth max-depth)
       ( let* ((deps (get-dep-names path))
               #;(_ (show #t "deps:" (displayed deps) "\n") )
              (paths (get-dep-paths path))
               #;(_ (show #t "paths:" (displayed paths) "\n") )
              (pairs (filter ( lambda (t) (not (equal? #f t)))
                             ( map ( lambda (dep) (assoc dep paths)) deps)))
               #;(_ (show #t "pairs:" (displayed pairs) "\n") )
              (targets (filter
                        ( lambda (t)
                          (not (set-contains? stopfiles t)))
                        ( map cadr pairs)))
               #;(_ (show #t "targets:" (displayed targets) "\n") )
              )
          #;(map (lambda (x) (display x) (newline)) pairs )
         ( map ( lambda (target)
                ( let ()
                  ( if (not (set-contains? processed-vertices target))
                     ( begin
                       (shell (echo ,(string-append  "\"" target  "\" "
                                                    ( if (systemp target)
                                                        "[color=red]"
                                                        "[color=green]"))) (>> ,graph-file))
                       (set-adjoin! processed-vertices target)))
                  (shell (echo ,(string-append  "\"" path  "\" -> \"" target  "\"
                               [color=\""
                               (to-string (random-real))
                                "+"
                               (to-string (random-real))
                                "+"
                                "0.5\"]")) (>> ,graph-file))))
              targets)
         ( map ( lambda (target)
                ( if ( and (not (set-contains? processed target))
                      (not (systemp target)))
                   (walk-tree target (+ 1 depth))
                   'already-processed))
              targets))
       (show #t  "depth of " depth  " reached\n"))))
 ;;  (define my-deps (get-dep-names "/vendor/lib64/libcamerapostproc.so"))
 ;;  (define my-paths (get-dep-paths "/vendor/lib64/libcamerapostproc.so"))
 ;;  (map (lambda (x) (display x) (newline)) my-deps)
 ;;  (map (lambda (x) (display x) (newline)) my-paths)
( define  main1
  ( lambda ()
    (shell (echo  "strict digraph {
ranksep=\"1.2 equally\"" ) (> ,graph-file))
     #;(walk-tree "/vendor/lib64/libcamerapostproc.so" 0 )
     #;(walk-tree "/vendor/lib64/vendor.qti.hardware.camera.postproc@1.0-service-impl.so" 0 )
     #;(walk-tree "/vendor/bin/hw/vendor.qti.camera.provider-service_64" 0 )
    (walk-tree  "/vendor/lib64/hw/com.qti.chi.override.so" 0)
    (shell (echo  "}" ) (>> ,graph-file))
    ))
(main1)
(shell (dot -Tpdf  "dep-graph.dot") (>  "dep-graph.pdf"))
( define  main2
  ( lambda ()
    ( let* ((get-rel ( lambda (p) (list->set equal? (filter systemp ( map cadr (get-dep-paths p))))))
           (deps-camera (set-union (get-rel  "/vendor/lib64/libcamerapostproc.so")
                                   (get-rel  "/vendor/lib64/vendor.qti.hardware.camera.postproc@1.0-service-impl.so")
                                   (get-rel  "/vendor/bin/hw/vendor.qti.camera.provider-service_64")
                                   (get-rel  "/vendor/lib64/hw/com.qti.chi.override.so")))
           (deps-libAlgoProcess    (set-union
                                    (get-rel  "/odm/lib64/libAlgoProcess.so")
                                    (get-rel  "/odm/lib64/libAlgoInterface.so")))
           (difference (set-difference deps-camera deps-libAlgoProcess))
           (shower ( lambda (e) (show #t (written e)  "\n")))
           (mysort ( lambda (s) (sort (set->list s) string"deps-camera:\n")
      ( map shower (mysort deps-camera))
      (show #t  "\ndeps-libAlgoProcess:\n")
      ( map shower (mysort deps-libAlgoProcess))
      (show #t  "\ndifference:\n")
      ( map shower (mysort difference)))))
(main2)

Just look at how beautiful it is. It is generating a graphviz tree, and you can compile it into a searchable pdf.

When you find missing dependencies, just add them to proprietary-files.txt.

When simple dependency and log-based methods stop working, it is time to do dig a little bit deeper.

The previous chapter already mentioned decompiling an apk with apktool. An extremely nice property of apktool compared to traditional decompilers is that it is possible not just to decompile an apk, but to also re-compile it, and the resulting apk will be as good as the old one, except, maybe, losing the platform key, but since we are building our own Lineage, we have the key.

Some adaptations might require you to just adjust the file AndroidManifest.xml and recompile the apk, but sometimes you have to actually edit the code in order to make the app compatible with your platform.

Here comes Smali, the Java assembly.

This is a decent howto on Smali: https://sallam.gitbook.io/sec-88/android-appsec/smali/smali-cheat-sheet See the chapter 1.6.7 to see how actual smali patches work.

apktool will decompile the code successfully, but trawling through megabytes of Smali is tedious, so we have JADX at our disposal.

JADX is a decent decompiler. You can rename variables, add comments, and search nicely.

It is not almighty, and it cannot compile the apps back into apk, so when you find a place you want to modify, you have to edit the original smali code produced by apktool.

Luckily for us, Dalvik does limit the amount of registers per function, so we can add as many as we want, and call Log.e() whenever we need.

const-string v4, "mine-parseMenuSetting"
const-string v5, "17"
invoke-static {v4, v5}, Landroid/util/Log;->e(Ljava/lang/String;Ljava/lang/String;)I
invoke-interface {v0, v2}, Ljava/util/List;->add(Ljava/lang/Object;)Z

As I mentioned, JADX can theoretically connect to a running app and become a debugger, but I failed to make work.

1. ELF Format specification :: https://refspecs.linuxfoundation.org/elf/elf.pdf
2. Learning Linux Binary Analysis by Ryan “Elfmaster” O’Neill :: https://github.com/PacktPublishing/Learning-Linux-Binary-Analysis
3. The Ghidra Book The Definitive Guide by Chris Eagle and Kara Nance :: https://nostarch.com/GhidraBook
4. https://blog.k3170makan.com/2018/09/introduction-to-elf-format-elf-header.html
5. https://en.wikipedia.org/wiki/Executable_and_Linkable_Format
6. https://thinkingeek.com/categories/aarch64/

If Java decompiling does not work, and you need to use heavier artillery, have a look at Ghidra.

1. https://www.youtube.com/watch?v=RoNppaG1re4
2. https://github.com/iddoeldor/frida-snippets?tab=readme-ov-file
3. https://cilynx.com/vulnerabilities/exploring-native-functions-on-android-and-runtime-analyses-using-jadx-ghidra-and-frida/1565/
4. https://github.com/Hamz-a/frida-android-libbinder

Ghidra is programmable in Java and Python, in theory you can run Ghidra scripts in Ghidra debugger, but it is a bit inconvenient, because Ghidra works by pulling memory to your machine via lldb, which sometimes happens to be slow.

Frida goes into the opposite direction. It embeds a JavaScript interpreter into the target process and allows you to run arbitrary JS code at trace points. You can use this, for example, to trace important functions calls.

Frida is instantaneously fast.

I did not use Frida a lot (just traced a few function calls), but generally a programmable debugger-tracer is a very mighty tool.

For example, you can monitor Binder transactions. https://github.com/iddoeldor/frida-snippets?tab=readme-ov-file#binder-transactions

1. https://www.jemarch.net/poke-elf.html

This section is not developed, because I did not do it.

So, the idea is that one might want to write a C function, compile it into an .o file and all-link it to an .so binary. The goal is to be able to call that function from within some other functions, for example inserting a call with ddisasm.

I suspect that GNU Poke-ELF ( https://www.jemarch.net/poke-elf.html) should be able to do that, but I am not sure.

1. Working features
   1. Normal photos
   2. 50 MPx photos
   3. “Livephoto”
   4. 1080@60Hz video
   5. 4k@30Hz video
   6. “Pro” photos
   7. Panorama
   8. SELinux
2. Partially working features
   1. 4k@60Hz video
   2. slomo videos
   3. RAW images
   4. RAW+ images
3. Broken features
   1. Portrait/Bokeh
   2. AI ID scanning
   3. 10bit+HEIC images

I started, of course, standing on the shoulders of giants, which in my case are

1. https://gitlab.com/pjgowtham/proprietary_vendor_oplus_camera
2. https://gitlab.com/ThankYouMario/proprietary_vendor_oplus_camera

I did not really care whether I am getting an “OnePlus Camera” (the international build), or the “OPlus Camera” (the Chinese build), as I only needed the offline features, and the Chinese version turned out to be easier to port.

A few points deserve attention.

1. Binary extraction
2. Init script
3. Mounting the files to override selinux context
4. Main Camera project name

Extracting blobs is documented here: https://wiki.lineageos.org/extracting_blobs_from_zips_manually

The important thing is that at some point LineageOS has moved from extract-blobs.sh to extract-blobs.py, which seems to be more functional, but is a little weird to use.

I was extracting blobs from an unpacking of the most recent version of the OnePlus original operating system, done as usually done with most Lineage builds, with a single caveat.

My system used a single “APEX” package to provide alternative versions of the binary libraries to (at least some of) camera .so libraries as dependencies.

So in addition to unpacking the first layer of firmware into separate directories ( system, system_ext, vendor, odm), I had to unpack that APEX file and add an additional directory to the unpack.

It is very important that some binaries are “pinned”, that is that they are not expected to be extracted and are provided with the camera module itself. In my project those are the files of the camera and gallery themselves, because they are actually re-compiled from Smali sources.

Init script, mount, and camera project name come from my predecessors, and I did not try removing them, and I do not know if they still serve any purpose.

It is surprising how far one can go without any specific debugging tools, just by reading logs and following the intuition.

Essentially I kept reading adb -s logcat and adb -s dmesg for errors, and tried fixing those which were appearing.

Most of the problems were of these kinds:

1. Missing Java classes :: those I implemented in the oplus-fwk
2. Missing start-time .so libraries :: those I copied from the original firmware.
3. Missing dlopen .so libraries :: those were also usually reported in the log, except when they were not.

At some point I used a script written by myself in Scheme in order to plot the dependency tree of the camera .so files.

This is just a fragment, the full graph is a PDF, and it kinda helped me to track which dependencies are already installed and which are not related to the camera.

At some point I found that the requirements for the apk files changed with between Android 13, 14, 15 and 16.

Those changes, luckily, were primarily those which can be accommodated with one of the two methods:

1. Adding some statements into the AndroidManifest.xml to declare using new permissions.
2. Adding some statements to the /system_ext/etc/permissions/*xml and /system/etc/sysconfig/*xml /system_ext/etc/default-permissions/*xml to grant those permissions to the app without asking. See the files in the camera project repo.

But if changing the xml files in on the system is easy, changing the xml file in the apk requires recompiling the apk.

Here is my script for it:

 cd ~/Lineage-Camera
(
   set -x
   cd OplusCamera-15 &&  \
  rm -rf ./dist/OplusCamera-15.apk  &&  \
  apktool b &&  \
  java -jar ~/Lineage-Build/out/host/linux-x86/framework/apksigner.jar sign --in ./dist/OplusCamera-15.apk -out ./dist/OplusCamera-15.apk.signed --cert ~/Lineage-Build/build/make/target/product/security/platform.x509.pem --key ~/Lineage-Build/build/make/target/product/security/platform.pk8 --v3-signing-enabled false &&  \
  mv ./dist/OplusCamera-15.apk.signed ./dist/OplusCamera-15.apk &&  \
   mysha=$(sha1sum ./dist/OplusCamera-15.apk | cut -d  ' ' -f 1) &&
  printf  'Opluscam sha1sum=%s\n'  "$mysha" &&
  sed -i  "/my_product\/app\/OplusCamera\/OplusCamera\.apk/s/|.*$/|$mysha/"   ~/Lineage-Build/vendor/oplus/camera/proprietary-files.txt &&
  cp ./dist/OplusCamera-15.apk ~/Lineage-Build/vendor/oplus/camera/proprietary/system_ext/priv-app/OplusCamera/OplusCamera.apk ||  exit 1
) &&  \
(
   cd OppoGallery2 &&  \
  rm -rf ./dist/OppoGallery2.apk &&  \
  apktool b &&  \
  jarsigner -storepass  "android" -verbose -sigalg SHA1withRSA -digestalg SHA1 -keystore ~/.android/debug.keystore ./dist/OppoGallery2.apk androiddebugkey &&  \
   mysha=$(sha1sum ./dist/OppoGallery2.apk | cut -d  ' ' -f 1) &&
  printf  'Oplusgallery sha1sum=%s\n'  "$mysha" &&
  sed -i  "/my_stock\/priv-app\/OppoGallery2\/OppoGallery2\.apk/s/|.*$/|$mysha/"   ~/Lineage-Build/vendor/oplus/camera/proprietary-files.txt &&
  cp ./dist/OppoGallery2.apk ~/Lineage-Build/vendor/oplus/camera/proprietary/system_ext/priv-app/OppoGallery2/OppoGallery2.apk
) ||  exit 2

 if ! diff -u OplusCamera-15/dist/OplusCamera-15.apk ~/Lineage-Build/vendor/oplus/camera/proprietary/system_ext/priv-app/OplusCamera/OplusCamera.apk  ;  then
  printf  'Cameras do not match!\n'
   exit 1
 fi

 if ! diff -u OppoGallery2/dist/OppoGallery2.apk ~/Lineage-Build/vendor/oplus/camera/proprietary/system_ext/priv-app/OppoGallery2/OppoGallery2.apk ;  then
  printf  'Galleries do not match!\n'
   exit 1
 fi

Note that I am actually signing the camera apk with the platform key. This is superfluous when building Android once, but if you want to avoid re-flashing the image each time you rebuild the apk, you can sign the apk with the “platform key” and just copy it to the target directory.

1. https://xdaforums.com/t/edge30ultra-moto30xpro-camera-research.4492159/page-3

There was a third, “bonus” file to edit, /etc/camera/camxoverridesettings.txt, but it only required two edits.

 raiserecoverysigabrt=0
 disableFDHWProcessing=1

The first one does not crash the process when ASAN ( https://clang.llvm.org/docs/AddressSanitizer.html) failures occur, and they will occur because, come on, the project is not built from one source.

The second disables hardware face detection (FD is Face Detection), because there were some crashes connected to it? (I guess, I actually forgot).

See the link to xda-forums for more details about using camxoverridesettings.txt. In particular, by setting enableFeature2Dump=1 it is possible to get not just DNG RAW, but full-sized Bayer matrix dumps into /data/vendor/camera.

The full list of parameters is in the project repo, but I don’t know what they mean.

I had to stub a lot of oplus-framework.jar functions, mostly blindly making them return false, null, throw an exception, or do nothing.

Perhaps this is the part of my work most likely to be accepted to LineageOS’ upstream. Not all of the code is written by me, a lot of it is written by my predecessor, sx75 from crdroid.

In total, this is:

git log HEAD~2..HEAD --patch -- oplus-fwk/ | wc -l
16099

in

git log HEAD~2..HEAD --stat -- oplus-fwk | grep -F  '|'  | wc -l
135

I had to add the following exceptions to the declared “allowed framework classes” into build/soong/scripts/check_boot_jars/package_allowed_list.txt

diff --git a/scripts/check_boot_jars/package_allowed_list.txt b/scripts/check_boot_jars/package_allowed_list.txt
index 983650039..7c5ed0ca2 100644
--- a/scripts/check_boot_jars/package_allowed_list.txt
+++ b/scripts/check_boot_jars/package_allowed_list.txt
@@ -280,6 +280,12 @@ com\.nvidia\..*
 # OPLUS adds
 com\.oplus\..*
+oplus\..*
+com.coloros\..*
+vendor.oplus\..*
+com.heytap\..*
+net.oneplus\..*
+com.color\..*
 # QC adds
 com.qualcomm.qti

Even though I have introduced Smali in 1.5.5, I actually did not need to write much Smali to solve concrete issues. Essentially, I added just two patches: one fixed a race condition deadlock, another one removed a hanging check.

The deadlock is fixed like this, and I am afraid to touch it, because it is just making the correct thread “win” the race and is not actually solving the bug.

diff --git a/smali/com/oplus/camera/common/config/CameraSettingsConfig.smali b/smali/com/oplus/camera/common/config/CameraSettingsConfig.smali
index e80eaccf..befe96b8 100644
--- a/smali/com/oplus/camera/common/config/CameraSettingsConfig.smali
+++ b/smali/com/oplus/camera/common/config/CameraSettingsConfig.smali
@@ -1383,7 +1383,7 @@
 .end method
 .method public static declared-synchronized initialize(Landroid/content/Context;)V
-    .locals 3
+    .locals 6
     .line 1
     const-class p0, Lcom/oplus/camera/common/config/CameraSettingsConfig;
@@ -1472,6 +1472,9 @@
     .line 45
     sput-boolean v0, Lcom/oplus/camera/common/config/CameraSettingsConfig;->sbInit:Z
+    const-string v4, "mine-initialize"
+    const-string v5, "after sbInit=true;"
+    invoke-static {v4, v5}, Landroid/util/Log;->e(Ljava/lang/String;Ljava/lang/String;)I
     .line 46
     .line 47
@@ -1487,23 +1490,23 @@
     .line 50
     :cond_0
     :goto_0
-    const-string v0, "CameraSettingsConfig"
+#    const-string v0, "CameraSettingsConfig"
     .line 51
     .line 52
-    new-instance v1, Lcom/oplus/camera/common/config/a;
+#    new-instance v1, Lcom/oplus/camera/common/config/a;
     .line 53
     .line 54
-    const/4 v2, 0x0
+#    const/4 v2, 0x0
     .line 55
-    invoke-direct {v1, v2}, Lcom/oplus/camera/common/config/a;->(I)V
+#    invoke-direct {v1, v2}, Lcom/oplus/camera/common/config/a;->(I)V
     .line 56
     .line 57
     .line 58
-    invoke-static {v0, v1}, Ll7/a;->r(Ljava/lang/String;Ljava/util/function/Supplier;)V
+#    invoke-static {v0, v1}, Ll7/a;->r(Ljava/lang/String;Ljava/util/function/Supplier;)V
     :try_end_0
     .catchall {:try_start_0 .. :try_end_0} :catchall_0
@@ -1513,6 +1516,9 @@
     monitor-exit p0
     .line 62
+    const-string v4, "mine-initialize"
+    const-string v5, "after before return-void;"
+    invoke-static {v4, v5}, Landroid/util/Log;->e(Ljava/lang/String;Ljava/lang/String;)I
     return-void
     .line 63

The second is just as simple:

--- a/smali/com/oplus/camera/common/config/CameraSettingsConfig.smali
+++ b/smali/com/oplus/camera/common/config/CameraSettingsConfig.smali
@@ -3683,32 +3683,37 @@
     .line 157
     .line 158
     .line 159
-    invoke-static {}, Lv6/k2;->c()Z
+#    invoke-static {}, Lv6/k2;->c()Z
     .line 160
     .line 161
     .line 162
-    move-result v0
+#    move-result v0
     .line 163
-    if-eqz v0, :cond_5
+#    if-eqz v0, :cond_5
     .line 164
     .line 165
-    iget-object v0, p0, Lcom/oplus/camera/common/config/CameraSettingsConfig;->mMenuSettingList:Ljava/util/List;
+#    iget-object v0, p0, Lcom/oplus/camera/common/config/CameraSettingsConfig;->mMenuSettingList:Ljava/util/List;
     .line 166
     .line 167
-    const-string v2, "pref_spruce_function_key"
+#    const-string v2, "pref_spruce_function_key"
     .line 168
     .line 169
-    invoke-interface {v0, v2}, Ljava/util/List;->add(Ljava/lang/Object;)Z
     .line 170
     .line 171
     .line 172
-    :cond_5
+#    :cond_5
     invoke-static {}, Lcom/oplus/camera/configure/CameraConfig;->C()Z
     .line 173

That is just it. I just removed some hanging function calls.

This issues is not entirely solved, but at least is somehow progressed. When I was trying to find out why the “240Hz slomo” mode is not opening in the camera, I encountered a message about not enough buffers being allocated for a stream.

This is related to an Android feature called “HAL3 buffer management API.” ( https://source.android.com/docs/core/camera/buffer-management-api)

requestStreamBuffers is a function which is sometimes called by the “Camera HAL” client to receive … something. I guess, some buffer pointers, for stream data.

The log was complaining about the code requiring too many buffers, more than had been allocated in advance. I added a hack to allocate more buffers, but the issue is probably that BBK ships a patched version of their cameraserver and its dependencies, which somehow manage buffers differently. It was also complaining about wrong frame numbers… whatever that is.

diff --git a/services/camera/libcameraservice/device3/Camera3OutputUtils.cpp b/services/camera/libcameraservice/device3/Camera3OutputUtils.cpp
index 7a53847765..ddcf59c096 100644
--- a/services/camera/libcameraservice/device3/Camera3OutputUtils.cpp
+++ b/services/camera/libcameraservice/device3/Camera3OutputUtils.cpp
@@ -654,10 +654,15 @@ void processCaptureResult(CaptureOutputStates& states, const camera_capture_resu
	 std::lock_guard l(states.inflightLock);
	 ssize_t idx = states.inflightMap.indexOfKey(frameNumber);
	 if (idx == NAME_NOT_FOUND) {
-            SET_ERR("Unknown frame number for capture result: %d",
-                    frameNumber);
+          if (states.inflightMap.size() != 0) {
+            idx = states.inflightMap.size() - 1;
+            goto mine_out;
+          }
+          SET_ERR("Unknown frame number for capture result: %d",
+                  frameNumber);
	     return;
	 }
+    mine_out:
	 InFlightRequest &request = states.inflightMap.editValueAt(idx);
	 ALOGVV("%s: got InFlightRequest requestId = %" PRId32
		 ", frameNumber = %" PRId64 ", burstId = %" PRId32
diff --git a/services/camera/libcameraservice/device3/Camera3Stream.cpp b/services/camera/libcameraservice/device3/Camera3Stream.cpp
index ae76e603bd..0a0360e306 100644
--- a/services/camera/libcameraservice/device3/Camera3Stream.cpp
+++ b/services/camera/libcameraservice/device3/Camera3Stream.cpp
@@ -382,7 +382,9 @@ status_t Camera3Stream::finishConfiguration(/*out*/bool* streamReconfigured) {
	 std::string name = std::string("Stream ") + std::to_string(mId);
	 mStatusId = statusTracker->addComponent(name);
     }
-
+    if (camera_stream::max_buffers >=6 || camera_stream::max_buffers <= 24 ) {
+      camera_stream::max_buffers = 25;
+    }
     // Check if the stream configuration is unchanged, and skip reallocation if
     // so.
     if (mState == STATE_IN_RECONFIG &&

I suspect it should increase memory usage significantly.

Somehow this made the “slomo” work, even though in some kind of crippled way. It is “slow” (whatever that means for a slo-mo), and crashes in the dark. I guess this issues is still awaiting its champion.

Is suspect it is somehow related to “stream metadata”, the same issue as in 1.6.11.

When the app began starting successfully, I discovered that it crashes in the native code when making photos.

Native code means that disassembling and binary debugging now became mandatory.

The first steps in debugging the camera were a detour, but I still found some useful stuff.

That is, I found some system properties which enable debugging logging for the Oplus Camera, and some for the Qualcomm backend:

ro.control_privapp_permissions=log  \
persist.sys.state.is.otest.running=1  \
vendor.oplus.apsSN.algo.enable=0  \
ro.control_privapp_permissions=log  \
sys.gcsupression.optimize.enable=0  \
persist.camera.aps.core.warningMask=0xffffffffffffffffffffffffffffffff  \
persist.camera.aps.core.infoMask=0xffffffffffffffffffffffffffffffff  \
persist.camera.aps.core.debugMask=0xffffffffffffffffffffffffffffffff  \
oplus.autotest.camera.debug.forcelog=true  \
persist.sys.camera.lao.enable=true  \
persist.oplus.aps.trace=true  \
com.oplus.camera.pw.debug.capture.flow=true  \
ro.build.release_type=false  \
ro.build.version.ota=PRE  \
ro.version.confidential=false  \
persist.camera.rotate.dump=true  \
persist.camera.cfr.debug=1  \
vendor.aps.logic.snapshot.metadata.debug=1 \
persist.sys.assert.panic=true   \
persist.camera.assert.panic=true  \
persist.vendor.camera.oplus.enableLogging=true  \
persist.sys.dump.aps=1  \
persist.vendor.camera.jpeghwencode=1

Not all of them enable debugging, but the names are quite self-explanatory.

The “dumps” usually go to /data/vendor/camera/, the logs go to logcat. There are some logging parameters in camxoverridesettings.txt, but I failed at using them, too difficult.

As I said, some parameters are actually misleading and led to a detour. In particular, persist.vendor.camera.jpeghwencode=1 does make the post-processing flow go into a different direction, but it turned out to be harder to debug than the “hardware” one.

At the end of the day, the camera supports hardware post-processing. (The name of the parameter is misleading, it is not just activating hardware JPEG.)

1. https://developer.android.com/ndk/reference/group/a-hardware-buffer

One of the crashes that I had to debug for the camera to work was a crash related to a wrong version of AHardwareBuffer_lock ( https://developer.android.com/ndk/reference/group/a-hardware-buffer#group___a_hardware_buffer_1gaf10c050f1ddfe2a074ab0f80660b2925)

I found that a piece of code was crashing the program by accessing a non-allocated memory, as so-called “segmentation failure”.

Of course, I had had many segmentation faults before, but never studied in depth where they may be coming from. It turns out that program memory space is fixed for a program when it is loaded into memory and how in-process addressing works in modern operating systems.

To keep the long story short, the code was crashing in an completely innocuous function, which was just manipulating some bits, i.e. no complicated system calls, stack unwinding, or any other magic.

I started investigating why it crashes at accessing some memory, and found out that neither my debugger (lldb), nor Ghidra debugger can access that memory and it is shown as “unmapped”. (Note that in Linux, “unmapped” my mean both “not mapped to a different process”, and “not allocated even in the current process space”.)

Neither the Ghidra Book, nor “Learning Linux Binary Analysis” helped me with solving this mystery, but at some point I encountered a StackOverflow question asked by a programmer who could not read memory-mapped region with his debugger, and I guessed that this might be my case.

To this day I do not know whether this memory is from Android Binder or from just a simple mmap, but it did not matter for my case.

I started to look around that code, and found that the mapped memory region was produced by a call to a function in the “AHardwareBuffer” family.

I looked at the code for extracting binaries from the vendor’s firmware, and found the following lines ( https://github.com/xigua-dev/device_oneplus_xigua/blob/183e8f24df43b390f8e26a8d7c6f40bcc99f65e5/extract-files.py#L55):

( 'odm/lib64/libHIS.so',  'odm/lib64/libOGLManager.so'): blob_fixup()
      .clear_symbol_version( 'AHardwareBuffer_allocate')
      .clear_symbol_version( 'AHardwareBuffer_describe')
      .clear_symbol_version( 'AHardwareBuffer_lock')
      .clear_symbol_version( 'AHardwareBuffer_release')
      .clear_symbol_version( 'AHardwareBuffer_unlock'),

“What the hell?” – I thought. “These are the same functions which are used near the place where my code crashes. Why are symbols there even versioned?”

Also, symbol versions for those functions were weird. They were not what we usually have in Linux, as in @GLIBC_2.24, but just @LIBNATIVEWINDOW.

It is worth noting that in Linux linking, symbols with symbol versions can be linked with libraries requiring functions without symbol versions, but not the other way round. I.e. if you library requires some symbols, versioned will do, but if it requires versioned, un-versioned will not work.

Then I added the copy of libnativewindow.so to the list of extracted binaries from the vendor’s firmware, and also a set of its dependencies transitively, and had to rewrite some “sonames” and replace dependencies, and boom, the camera started to start up successfully and not crash when trying to make photos. The photos still did not appear in the DCIM directory, but at least the camera did not crash.

I assume that the different versions of the AHardwareBuffer_* functions are creating different mappings of the P010 raw pixels arrays in the memory, and the code in the camera was expecting some particular mapping.

After I made the camera successfully try to make photos, I found that they do not appear in the DCIM directory. Luckily, at the same time some crashdumps, “tombstones” started to appear in the /data/tombstones/ directory.

Those crashdumps indicated that the crash was happening in the process called vendor.qti.camera.provider-service_64.

This seems to be a process written by Qualcomm, which is responsible for managing the camera devices in the Linux kernel ( /dev/video*) and wrapping their usage into the Android HAL API.

I am still not exactly sure whether the camera app is interacting with the HAL process through the Binder calls directly, or through a different process running under the same cameraserver user, inventively called cameraserver.

In any case, the crash was happening because the function called PostProc was querying the “stream metadata” for the Logical Camera ID and the Physical Camera ID, and since both were missing, it was using some inappropriate heuristic to get the physical ID, and was accessing unmapped memory.

I started to investigate why exactly they were missing, but failed and never found the cause. However, grepping and searching for various relevant symbols create a strong impression in me that BBK has somehow noticeably patched the process called cameraserver, and the library called libcsextimpl.so, so that they would be writing more metadata into the streams, which the Qualcomm process, in turn, would be relying upon.

As far as I understand, this is not a polite way of writing code. The code of the “framework” should be unchanged, and only “ODM” code should be tweaked by the phone manufacturer, but hey, who am I to say what should and what should not be done outside of my area of expertise.

Initially I tried to do the same thing with libcsextimpl.so, but found that in the current Android version Google has turned it into a static library instead of a dynamic one, and even though making it dynamic again and trying to replace the open source version with a binary one, but it turned out to require way more dependency rewriting than I had expected, and I gave up.

However, it turned out that by using a small binary patch I can set that “Logical Camera ID” to always be zero, and it was enough to placate the PostProcess function to do its function correctly. I generated the patched binary in Ghidra (by using the “Patch instruction” feature), and then used diff -u and xxd to generate a two-line binary diff, which I added to the extract-files.py, and voila, it works.

.binary_regex_replace(  #  metadata error in in ChiOfflineIPEUpscale::PostProcess(PostProcSessionParams*)
     bytes.fromhex( '6299009400060035844F40F962034039940040B982051836460100B0C6F444F9'),
     bytes.fromhex( '0000805200060035844F40F9620340391400805282051836460100B0C6F444F9'))

1. https://source.android.com/docs/core/camera/camerax-vendor-extensions
2. https://source.android.com/docs/core/camera/camera3

So, in addition to capturing CCD pixel intensity, the camera also measures a lot of parameters, such as aperture, shutter delay, flash, timestamp of each frame, et cetera.

Those are saved into a data structure called “stream metadata”, which is basically a map/dictionary from “vendor tags” to metadata values. That is, metadata keys are binary numbers, not text strings, and each participant in this trio: Google, Qualcomm, BBK, is having its own system for converting strings such as "com.oplus.LogicalCameraID" into a code for querying metadata.

There is an issue about it https://issuetracker.google.com/issues/261760717

I suspect that in the current setup a lot of metadata is not getting into the pipeline.

Tags can be dumped using the property vendor.aps.logic.snapshot.metadata.debug=1, I think. The file called /data/vendor/camera/metadata/metadatatable.txt.

Making the camera work with SELinux set to “Enforcing” was the last part of my debugging process.

Setting SELinux to “Enforcing” produced quite a few denial errors in the dmesg, but fixing all of them with sepolicy-inject and chcon did not really make the camera work, or even start.

02-15 20:16:53.495  8722  8722 I PreviewReceived: type=1400 audit(0.0:287): avc:  denied  { open } for  path="/dev/__properties__/u:object_r:vendor_default_prop:s0" dev="tmpfs" ino=464 scontext=u:r:platform_app:s0:c512,c768 tcontext=u:object_r:vendor_default_prop:s0 tclass=file permissive=1 app=com.oplus.camera
02-15 20:16:53.495  8722  8722 I PreviewReceived: type=1400 audit(0.0:288): avc:  denied  { getattr } for  path="/dev/__properties__/u:object_r:vendor_default_prop:s0" dev="tmpfs" ino=464 scontext=u:r:platform_app:s0:c512,c768 tcontext=u:object_r:vendor_default_prop:s0 tclass=file permissive=1 app=com.oplus.camera
02-15 20:16:53.495  8722  8722 I PreviewReceived: type=1400 audit(0.0:289): avc:  denied  { map } for  path="/dev/__properties__/u:object_r:vendor_default_prop:s0" dev="tmpfs" ino=464 scontext=u:r:platform_app:s0:c512,c768 tcontext=u:object_r:vendor_default_prop:s0 tclass=file permissive=1 app=com.oplus.camera

and

06-10 00:05:59.224  1026  1026 E SELinux : avc:  denied  { find } for interface=vendor.qti.hardware.camera.postproc::IPostProcService sid=u:r:platform_app:s0:c512,c768 pid=8558 scontext=u:r:platform_app:s0:c512,c768 tcontext=u:object_r:vendor_hal_camera_postproc_hwservice:s0 tclass=hwservice_manager permissive=0
06-10 00:05:59.264     0     0 E [T201019] SELinux: avc:  denied  { find } for pid=11375 uid=10184 name=oiface scontext=u:r:platform_app:s0:c512,c768 tcontext=u:object_r:default_android_service:s0 tclass=service_manager permissive=0
06-10 00:05:59.265     0     0 E [T201019] SELinux: avc:  denied  { find } for pid=11375 uid=10184 name=oiface scontext=u:r:platform_app:s0:c512,c768 tcontext=u:object_r:default_android_service:s0 tclass=service_manager permissive=0

audit2allow was always meeting “ neverallow” failures, and all of this stuff is so terribly documented that I was almost in despair.

However, two ideas came to my mind. The first one is that it does make a lot of sense to give the app process its own SELinux context (domain, label). It took me a while to find out how to do that, as doing it in a naive way (just adding a type opluscamera_app and adding a line to seapp_contexts) was causing a boot-loop.

However, at the end of the day I did something, and this something satisfies me.

I added a type opluscamera_app, and literally copied the definition of this type from platform_app.te. This made sure that my app has all the same permissions as a normal “platform app”, but is running at a different label, and I can filter the logs by this label. It also meant that I can add permissions to this label, without interfering with the main SELinux policy.

The drawback is that I had to declare this type not in the camera module but in the main sepolicy module (so my could would not be taken upstream), and add exceptions to the neverallow rules, which I do not mind, as there is only a single app running as this label, mine, and I do not intend to upstream it anyway.

I had to add neverallow exceptions like this:

system/sepolicy:

diff --git a/private/app.te b/private/app.te
index e4ac05b9c..441fb1889 100644
--- a/private/app.te
+++ b/private/app.te
@@ -521,7 +521,7 @@ allow appdomain zygote_tmpfs:file { map read };

 # Superuser capabilities.
 # bluetooth requires net_admin and wake_alarm. network stack app requires net_admin.
-neverallow { appdomain -bluetooth -network_stack } self:capability_class_set *;
+neverallow { appdomain -bluetooth -network_stack -opluscamera_app } self:capability_class_set *;

 # Block device access.
 neverallow appdomain dev_type:blk_file { read write };
@@ -746,7 +746,7 @@ neverallow {
 }:file no_x_file_perms;

 # Don't allow apps access to any of the following character devices.
-neverallow appdomain {
+neverallow {appdomain -opluscamera_app} {
     audio_device
     camera_device
     dm_device
@@ -758,6 +758,7 @@ neverallow appdomain {
 # needs access to /dev/video* for interfacing with the host
 neverallow {
     appdomain
+    -opluscamera_app
     -device_as_webcam
 } video_device:chr_file { read write };

device/qcom/sepolicy_vndr/sm8550/:

diff --git a/generic/vendor/common/domain.te b/generic/vendor/common/domain.te
index 0f4f0539e..2e256d693 100644
--- a/generic/vendor/common/domain.te
+++ b/generic/vendor/common/domain.te
@@ -85,6 +85,7 @@ neverallow { domain
 - hal_contexthub_default
 - hal_sensors_default
 - hal_camera_default
+- opluscamera_app
 userdebug_or_eng(` -vendor_usta_app')
 userdebug_or_eng(` -vendor_ustaservice_app')
 userdebug_or_eng(` -vendor_sysmonapp_app_test')
@@ -111,6 +112,6 @@ neverallow { domain
 userdebug_or_eng(` -vendor_sysmonapp_app_test')
 } vendor_qdsp_device:chr_file *;
 neverallow { domain -init -vendor_init - ueventd } vendor_qdsp_device:chr_file ~{r_file_perms};
-neverallow { appdomain - shell userdebug_or_eng(`-vendor_sysmonapp_app_test') } vendor_qdsp_device:chr_file ~{ioctl read};
+neverallow { appdomain - shell userdebug_or_eng(`-vendor_sysmonapp_app_test') -opluscamera_app } vendor_qdsp_device:chr_file ~{ioctl read};
 neverallow mediacodec vendor_qdsp_device:chr_file ~{ioctl read};

This allowed me to study the behaviour of the camera app without interfering with the workings of the “platform_app” label.

Another revelation, and I cannot call it by anything else, came when I was painfully and hopelessly staring through these two documents:

1. https://source.android.com/docs/core/architecture/aidl/aidl-hals#sepolicy
2. https://android.googlesource.com/platform/system/sepolicy/+/refs/heads/master/public/te_macros

The first one is written so badly that having it written in Swahili and read by a non-Swahili speaker would not hurt much. The second is just “the source”, you cannot expect much from it.

In any case, some of my SELinux denials were about vendor_hal_camera_postproc_hwservice, and none of the statements like hal_client_domain(opluscamera_app, hal_camera_postproc_hwservice), or hal_client_domain(opluscamera_app, hal_camera_postproc), or hal_hwclient_domain(opluscamera_app, hal_camera_postproc) were working.

Finally after spending a long time studying hardware/oplus/sepolicy/qti/vendor/, something clicked in my head and I wrote hal_client_domain(opluscamera_app, hal_camera). This worked. After looking once more at https://source.android.com/docs/core/architecture/aidl/aidl-hals#sepolicy , I decided that with a lot of mental gymnastics and imagination stretching, it is possible to see how the official document is claiming that this is actually what should be written, but, honestly, this approach to documentation is just horrible.

In any case, even though I had to add a type and edit the main tree, I am satisfied with this solution, because the app is running confined.

As a side-note, the camera app is actually requiring selinux_check_access(opluscamera_app), that is, its behaviour is different depending on whether SELinux is Enforcing, Permissive, or off. Which is a bad coding practice.

I do not have time to work on the camera any more, but I hope that someone can take over this honourable task.

I also did not find a good way to identify thread pools and find out which places are placing the tasks on the queue, so the code for me looks like “functions starting at libc pthread_start”.

It would be nice if someone submits a fragment to this howto on how to track task entry into thread pool.

This section is speculative, I am by no means an expert on Android Cameras.

I saw three processes interaction over Binder:

1. com.oplus.camera :: app
2. cameraserver :: Android camera HAL service process
3. vendor.qti.camera.provider-service_64 :: Qualcomm device process

The camera app had basically two big image-processing parts: preview and shooting. Both of those were implemented in the libAlgoProcess.so, which was dlopen from some of the camera built-in .so files. libAlgoProcess.so interacted with the cameraserver (?) somehow to obtain the image (buffer array), or several images as AHardwareBuffer_Planes.

After a buffer was obtained, it did post-processing on this buffer, either inside itself (depending on a key persist.vendor.camera.jpeghwencode), or passing the buffer to the vendor.qti.camera.provider-service_64 to do post-processing in hardware.

All of this was followed by “CameraMetadata”, indexed by “VendorTags”, which contained some important information about streams and buffers.

The camera process had some kind of “collection of typical pipelines”, called “usecases”, and in fact, a lot of Qualcomm code is named “CHI::usecase” – Camera HAL Interface Usecase.

I cannot believe that vendor.qti.camera.provider-service_64 is used only for post-processing. A camera process is naturally expected to also do capturing. But I did not find where capturing happens.

Вы, наверное, слышали, что программы для Андроида пишутся на Java. Наиболее продвинутые люди слышали что-то про Kotlin. Но на самом деле программы для Андроида пишутся на XML.

В файле AndroidManifest.xml нужно описать "структурные элементы" программы, которые будут затем определять то, как она взаимодействует с другими программами. Но не надо думать, что в AndroidManifest.xml нужно перечислить список путей, по которым программа может создавать named pipe, или список файлов, к которым разрешён доступ по аналогии с AppArmor или SELinux.

AndroidManifest.xml намного сложнее.

https://developer.android.com/guide/topics/manifest/manifest-intro

Самое забавное – это то, что он невероятно хуёвый. Например, он не поддерживает комментарии внутри тэгов.

Вот так можно:

< application
     android: allowBackup= "true"
     android: something= "meaningless"
     tools: targetApi= "31">
    < activity
        bla-bla
activity>
application>

А вот так нет:

< application
     android: allowBackup= "true"
     
    tools:targetApi= "31">
    < activity
        bla-bla
activity>
application>

Действительно, верно, что валидный xml не допускает комментарии внутри тэгов, но уж в Android такую чушь могли бы и исправить.

R означает "ресурсы", и генерируется из каталога res, процессором xml. Это в каком-то смысле похоже на Qt ресурсы, или даже на cmrc ( https://github.com/vector-of-bool/cmrc).

https://developer.android.com/guide/components/activities/intro-activities

https://stackoverflow.com/questions/77307867/screen-capture-mediaprojection-on-android-14

TODO: uses-feature TODO: uses-permission

Bundle – это какая-то такая штука, которая должна помогать нам восстанавливать состояние программы в случае перезапуска.

С какой-то стороны, это полезно, потому что Андроид часто убивает программы для экономии памяти или батареи. С другой стороны, возиться с этим муторно, и если нам хочется сгенерить простейшую обвязку для консольной программы, с этим не хочется возиться.

В этой главе будет краткий мануал по WebRTC, потому что это супер нужная штука для работы с потоками видео и аудио.

1. MediaStreamTrack = { AutioStream, VideoStream }
2. VideoCapturer/VideoSource/VideoStream
3. Transceiver/Transmitter/Receiver
4. ICE

In 1996 he went to Russia, as a part of the FúJiàn government delegation. They went to Saint-Petersburgh, Sochi, and Moscow. SPb impressed him by its grand and ceremonious architecture, but he does not tell any stories about what happened there, except that they were greeted by a Duma official.

In Sochi they studied a kolkhoz, apparently, not privatised, and he tells a few anecdotes.

1. They caught a few sea turtles for lunch, and locals were baffled by that.
2. The kolkhoz secretary invited them to build a mill near the kolkhoz, and when they asked for a “feasibility study” and a business plan, the secretary was baffled, he had never heard about anything like that.
3. The kolkhoz secretary told them the story about how Russian economy works: “There is a big gas pipe from Russia to Europe, the government sells gas and buys cheap Chinese goods. In fact, gas fees lets the government pay salaries to half of the country.”

(Souns fresh.)

In Moscow they also mostly did sight-seeing, and he was very disappointed by the quality of the hotel, which did not even allow their hosts to join the meal, because “it is not in the plan”. They met with the local Chinese trade representatives, who told them that the “protection fees” in Moscow are very high, a single ice-cream stand mush pay 5000 USD each month.

In general, however, he expressed positive thoughts about Russian future, saying that if the reform succeeds, a bright future is awaiting Russia due to a highly developed heavy industry and a lot of resources.

In 2001, FúYào and other Chinese car glass manufacturers were sued for selling their products “below the market price” and thus damaging the US economy.

This lead to a court case, which lasted years, in which FúYào tried to defend its position. Cáo DéWàng claims that they had prepared more than a ton of documents to substantiate their position.

In 2002 they were also sued in Canada, by a glass manufacturer PPG, also for dumping, but this case proved easy to win, just by proving that PPG as a US company, not a Canadian company.

They finally won the US lawsuit in 2004, and even had their pre-paid “anti-dumping tariffs” refunded.

This story impressed Cáo DéWàng so much that he even organised a research department, associated with a law school, which would be collecting data on anti-dumping cases all over the world, and funding anti-anti-dumping lawyers and researchers.

In 2010 he visited Kaluga, on an invitation from the governor, suggesting that he build a glass factory there, because Kaluga was trying to become a “Russian Detroit”, and had already domesticated Ford, Renault, and Honda.

He is saying that in 2010 Kaluga only had a single “pocket” hotel, built and run by the Germans for their own factory.

He tells a wonderful story on how expensive is building a factory in Kaluga.

1. Land: 5$
2. Design: 30$
3. Construction works: 200$
4. Concrete: 300$ per tonne.

He was surprised that regulations do not allow importing neither a design plan nor a construction team from China. He talked to the governor and said that under such conditions he would withdraw from the project, and they agreed that a Chinese design can be “converted”, and a construction team can be invited from China, as long as they register a company in Russia to organise the works.

In 2011 the project was underway, but the Russian part required that the agreement be signed in the presence of Hú JǐnTāo, in the Kremlin. It took them 2 months to make an agreement on when the signing ceremony can be organised. The Russian made the Ministry of Foreign Affairs send invitations three times (!), the first two being rejected.

Cáo DéWàng is saying that this was one of the most nervous times in his life. After all, his company is not that large, and not state-run. But after all, the Russian MFA succeeded, and found time when the Chinese leader was visiting Moscow.

After his factory in Kaluga had been completed, they organised a reception ribbon-cutting event, which, since the contract was witnessed by the top leaders, received an all-Russian attention.

Cáo DéWàng prepared a speech for this event, in which he was promising Russia a bright future, was describing several Russian problems and ways to solve them and hence make some money.

This speech is so interesting that I will eventually translate it into Russian, I suspect, but I will not copy it here.

After the reception he met with two representatives of the states of Michigan and Ohio, who were invited to the event as a part of the international community, and they invited him to invest in the USA.

In 2013 they purchased a factory in Ohio, which had used to belong to General Motors, and was later purchased and refurbished by a Jewish guy.

FúYào bought a half of this factory from him, and he also told them a story on how Rockefeller sold his skyscraper to the United Nations for a single dollar.

Of course, the building had cost much more than one dollar, but Rockefeller also owned most of the buildings nearby, and giving one for free to the United Nations make all of those properties rise in price and make him much more money than the price of a single building.

It was 2014, and, since he was planning to invest in the USA, he was considering pros and cons.

This chapter is quite long and full of insights, so I will not re-tell it here, but generally he describes the following pluses and minuses of the US position.

Pluses:

1. A good Constitution and a legal system which enforces property rights
2. American Dollar is the world’s reserve currency
3. A united political culture, which gets to every citizen
4. Outposts all over the world, including Hawaii
5. Quick access to international politics through United Nations headquarters
6. Simple lifestyle of its citizens, signified by Coca-Cola
7. Incessant strive for progress, signified by Apple
8. Sports culture, which is as important for Americans as food culture is for Chinese
9. Wall Street and the concentration of the world capital

Minuses:

1. Too much influence of the Wall Street, whose interests are wildly different from ordinary Americans
2. Decaying manufacturing, caused by the over-reliance of the economy on “services”
3. Weird political system which is not bringing in the people who can address issues, in particular, neither party represents the society as a whole and sees the full picture
4. The exceptionalist attitude of most Americans, who believe that their achievements are mostly due to their own industriousness and skill, whereas in fact it is due to the fact that they won the WW2 without taking much part in it.

This is 2014, and he is still quite positive about investing in the US manufacturing, but is still saying that the US economy has not fully recovered.

This chapter is actually not that much about his own investments in the USA, but rather about finding a correct stance to the USA in general.

The story he is telling there is about helping the University of Dayton buy a building in China, where they had a branch, but could not buy a building due to legal difficulties. At the end of the day, he donated the price of the building to the university, or rather made his company donate those money, and thus established “good friendship” in the USA, and the university thereupon helped him a lot to “merge” into the American society.

His point is “when making a business in China, I tried to work for the Chinese nation, when investing in the USA, I need to learn how to work for the American nation”. Not sure how truthful he is, but nevertheless.

A huge number of punishers!

Stats are what Cyberpunk authors consider to be innate, or at least hard to modify traits.

9 to 90 by a dice roll.

1. Intelligence (generalised “cleverness”)
2. Reflexes (generalised “dexterity”)
3. Cool (generalised “resist”)
4. Technical ability (generalised “skillfullness”) :: while I certainly see why this is not Int, I don’t understand why this is a Stat, not a Skill
5. Luck :: add these points to the dice roll
6. Attractiveness (generalised “charisma”)
7. Movement :: Why is this not the same as “reflexes”?
8. Empathy
   1. Humanity
9. Body (generalized “strength”)
10. Body type modifier

To be honest, all this looks like a complete crap. This traits are so ill-defined and not independent, that I am baffled at trying to fit normal people into this skeleton. The body description is particularly stupid and hard to assess even for myself, let alone other people.

How I would model inborn characteristics myself?

Okay, first of all, I am tempted to partition the traits into physical and mental. However… However, it is not clear whether psychological traits are really “software” and not “hardware”. And even my favourite Eric Berne was mentioning an (extremely dubious!) hypothesis that people are divided into endomorphic, mesomorphic, and ectomorphic.

Anyway, I would try to mention the following immutable characteristics:

Hardware

1. Height
2. Constitution (bone weight)
3. IQ
4. Luck

Software

1. Openness
2. Conscientiousness
3. Introversion
4. Agreeableness
5. Emotional stability

Does this look like OCEAN? Well, yes, because I literally took them from there.

And then I would add “first order skills”, which are, in some sense “impossible to forget”.

1. Muscle weight (STR)
2. Muscle flexibility (AGI)
3. Fat percentage (LOAD)
4. Willpower (WILL)
5. Memory (MEM)
6. Empathy/perception (EMP)
7. Attractiveness (ATT)
8. Leadership (LEAD)

First order skills would be limited by the immutable characteristics. But even the immutable characteristics are not totally immutable, and are evolving in time. So, in a good model we would have the system adjust them on each turn.

When you are writing your own new Facebook, why don’t you check people on these traits on registration?

Background is both interesting and not very much. It is interesting, because we like to remember things, and it is filling our roster and gives us the sense of our life being fulfilling.

However, for the purposes of modelling roles and scenarios, RPG-style, background is not terribly important, it is essentially a “historicist” misconception.

Nevertheless, when you are writing your own “Facebook” for keeping files on people, this “background” thing indicates a few database columns to prepare for data mining people’s connections.

So, Cyberpunk’s model of “skills” is that they are added to inborn stats when doing something. They give a lot of skills for teenagers to enjoy, but overall, this surprisingly resembles skills as they are indicated on LinkedIn. I wonder if it is hard to make a crawler for LinkedIn to collect all those skills and find correlations.

I will not list the skills here, it’s not useful, but finding a full list of human skills would be interesting.

“You will own nothing and you will be happy.” The game has a simplification here, but “everything you own you have to carry with you” approach is surprisingly similar to what I see around. Having a lot of stuff is inconvenient.

Property does not even appear in the book as something to own. “Nomads” have bikes and caravans. “Normal people” rent something to sleep at “per night”, which is radically different from how we rent flats now.

Regarding body types and inventory. According to the book, a 6-point body “average body type” should be able to carry about 60kg. To me this seems too generous to a person, but this surprisingly mathches the “marching load” of modern soldier. This probably makes my own body type 3-4, and a typical woman even less.

The guns part I am going to skip, because this part of the model is designed for being spectacular, not predicative.

The interesting thing here is “smart goggles”, which are definitely worth exploring.

One more fun thing is that “cybernetic” versions of ordinary stuff seem to cost “more” than old-school ones, whereas nowadays we know that the opposite is going to happen – buying a non-cyber things becomes harder and more expensive.

有人说精神压力大的时候他们吃更多并更容易变胖。 这个假设比较难证明，但是好像大部分人会承认，是这样。

这个现象是心理性的还是生理学的，我不确定。 有可能，承受精神压力的大脑，会误认为存在饿死的可能性，所以想要储存多一点资源，以防万一。 也有可能精神压力导致身体激活存储脂肪相关的基因表达，所以身体容易变胖。

不过，减轻精神压力肯定是有助于保持合理的体重。

肯定“减轻精神压力”是容易说的，但是不容易做的，但是有些措施可以执行，即使如果外面的压力不变：

1. 保证睡眠时间。 对，这个很普通，但是一定的程度之内，睡眠对大脑来说和吃饭一样重要。 所有的人有不同的睡眠需求，但是9个小时是猜测大部分人需要的睡觉时间的起点。 据说，睡觉3*x小时比别的长度更好。 就是说，睡觉6个小时比睡觉7个小时更好。 大部分人一般地会把8个小时贡献给睡眠，所以这个说明书推荐尝试睡觉9个小时。
2. 早睡。 毕竟，你入睡的时间不是很重要，所以我推荐日落之后立刻睡：这个对身体是最自然的，而且没有必要设置闹钟。 家务事和娱乐，早上或晚上做没有区别。 而且你可以调用早上的，更安静的，没有车辆喧嚣的时间做自己的事情。 并且早上你的大脑运转的更敏捷，利于完成自己的任务而不是老板的任务。
3. 散步。 散步很重要：它输送淋巴，因为淋巴系统没有别的泵，不如血液系统。 找散步时间对大部分忙的人不那么容易，但是我成功了。 我早上上班的时候，早下一个地铁站，就走一个地铁站的长度。

4. 规划你的休息时间。 这个建议也是很平庸，但是我请读者再思考：计划休息时间有很明显的好处，就是你宁愿安排高质量的娱乐，也不要抓住身边随机存在的娱乐。 真的，娱乐也区分高质量和低质量。 读书比刷抖音更好。 去旅游一天比40个小时连续地打电脑游戏更好。

   以上的措施不会直接导致减肥，但是它们会帮助你更容易节食。

为了减肥你应该吃得少。 所有的人都知道这个常识，但是按照我跟别人聊天的经验，很多人不知道这到底是什么意思。

首先，很多人好像认为，知道你吃多少卡罗里是足够的。

虽然这个态度对某些人有可能有效，但是对我来说它没有成功了。

为什么？ 你看，卡罗里测量的是能量，但是人类吃的是食物，而不吃能量。 食物是使用克（或者公斤）测量的。 尽管按照消耗的食物容量计算相等的能量，但是大部分这种计算很强的依赖个人身体的属性，所以我很怀疑这种计算的准确性。

相反，计算被摄入的碳水、脂肪和蛋白不仅是可以的，而且是比较可靠的。 肯定这个计算也是大概做得，因为不同的碳水，脂肪和蛋白都不是完全一样的，但是至少它们的容量是可以用化学方式测量，不需要依赖可以的公式。

此外，不灭定律，对普通人来说，比能量守恒律更容易理解：输入等于输出。 如果每个消耗循环当中（一天左右），你的身体消耗比摄入更多，体重就会减少。 （以上的逻辑不考虑核反应，因为在这个情况下它们不存在。）

（此节之后我会常用“碳脂蛋”缩写表达“碳水化合物，脂肪，蛋白质”。）

这个节确实很主观的，但是算法本身是很常见的，好像适合大部分人，有可能需要微调。

首先，你应该找到你的标准体重.

这个任务并不是很简单，因为所有人是不一样的，男和女是不一样的，而且甚至有同样高度的人，也有不同的骨头密度和直径。 幸运的是，你只需要一次完成这个任务。 如果你不是孩子，你的标准体重不太会变。

（如果你认真地健身就会。那你不需要我的建议。 在这种情况下推荐你去咨询你的医生。）

如果你只想要设计你的计划一次，然后一直沿它执行，找到理想体重应该是足够的。 我不能告诉你一个准确的公式，你应该自己找到，互联网有很多。

找到理想体重之后，你应该计算每日最低碳脂蛋摄入量。 此量是：为了保持体重需要摄入的量。

每公斤体重每日需要：

1. 碳水 ：4克
2. 脂肪 ：1克
3. 蛋白 ：1+克

让我们以重60公斤的人为例（男和女之中）：

碳水240克，脂肪60克，蛋白60克。

以上的数字表达身体的 _整个消耗_，由两个成分组成的：

1. 维修
2. 能量

理论上，三个资源都能对这两个成分发挥作用。 就是身体需要各个资源（碳脂蛋），才可以自我修复，以及它可以为了取得能量燃烧这个资源。 （这不是很重要，但是燃烧者碳水身体会制造“腺苷三磷酸”，脂肪成为“酮体”，而蛋白…我不知道，反正燃烧蛋只会非常差情况下发生。）

事实上，身体当然会优先将资源分配给修复工作而不是给能量。 毕竟，人类身体的物质转换能量的效率很高，但是修复能力…比较差，否则我们都会成为健美运动员。

如果你只消耗以上的资源数量，你还是会减轻体重，因为以上的数字表达 “最小的足够数量”，所以如果你的身体更重，以上的数量会小于维持较重体重的消耗。 你会非常慢的减肥，但是对很多人这个是足够的。

那现在我应该陈述第一个打破幻想的常谈： 减肥过程是永无止境的 。 确实，你找到正确的食物数量之后，必须终生保持消耗此数量。 为什么？你猜！ 是因为你的身体已经停止长高，你的身体不会长更高，它已经是封闭系统，所有输入的必须要么被消耗，要么储存起来。

最终，你到达你想要的体重，最好是一直保持比“最小必要的数量”完全一样的碳脂蛋消耗。

好的！ 为了减肥更快，你必须进一步减少以上章节的数字。 进一步，但不是过度减少。

我应该承认，有一个快速减肥的方式：“禁食疗法”。 意思大概是，你完全停止吃饭，而只能喝水。 很快（几天内），你的消化系统会熄火，你不会再感到饿。 （当然前提是你旁边也没有什么食物。）

你可以禁食一个星期，或者甚至更长一些，以及体重会迅速减轻。 我不确定多快，我反正不建议这么做。

虽然禁食不是比很多人认为危险那么危险（如果你禁食超过两个星期，危险的现象才会发生），但是实际上禁食不是很有用。 你会减肥很快，但是禁食之后你也会变胖很快。 而且你禁食之后必须按照“准确碳脂蛋”来继续节食，所以“禁食冲锋”不会给你很大的效果。

另一方面，比“最低数量”再减少你的“碳脂蛋输入”是可以的，这会导致你减肥更快，即便不那么快。

你会减少什么？

事实上你只能减少碳水。 我已经说过脂肪和蛋白不是都一样的。 某些在你的身体被储存的脂肪不能用于制造/修复，你必须吃一点额外的脂肪。 而且身体完全不能储存任何蛋白，所以你必须每天吃它。

但是减少碳水是可以的。 比如说，你每天吃120克碳水。 没问题，你的身体会（为了补偿能量的缺口）消化更多脂肪。 这就是你想要的，因为大部分多余的体重就是脂肪。

减少脂肪… 不是那么被推荐的。 我猜测（不要相信我的话），你可以把60克减少到30克，但是我不确定，咨询一下你的医生。

我再一次说，不要过度。

这里是第二个打破幻想的常谈： 你会感到饿肚子 。

不幸的， 饿肚子 是成人的常态。 这可能是大开眼界，成人和小孩有区别的事实。 孩子可以吃饱，对他们这个是正常的，大部分他们吃的食物会用于长高或增长肌肉。

成人不可以。 几万年进化使成人的饥饿感超过导致你吃“准确消耗”（保持正常体重）的标准。

在这个阶段，“semaglutide”(司美格鲁肽)会干涉、调整成人的饥饿。 但是这个说明书不涉及“semaglutide”，因为小编不用它还是可以成功的减肥。

那如何规划准确、适合你生活方式的营养成分？

“碳脂蛋”信息你可以在食品包装背面看到，也可以在网上找到。

吃很多你一般吃的菜，比如卷心菜或者鸡胸肉。 如果1公斤不满足你，就吃两公斤。

有激烈饥饿感的最大的问题是：它能导致你吃不健康的菜，而不是你吃太多。 以上的列表还是给你比较多自由。

一方面来说，保持稳定的体重是比减肥更难的。

成功之后你不仅需要强迫你的KPI下降，你也需要防御它过度下降。

所以你不只是需要考虑是否吃太多了也需要考虑是否吃太少了。

这时候大概的目标“240/60/60”已经是不够准确的，你应该找到很准确的消耗标志。

但是你至少可以恭喜自己减肥成功了。

此后，你应该可以每天吃两顿。 这也应该导致你的体重测量会成为不那么准确，你也应该需要把它每天测量两次。 比一次又难又不准，但是我不知道更好的方法。 你应该需要买好智能秤，因为手写很麻烦。

总而言之，240克碳水比60克允许很多，但是我一般更喜欢增加数量，并不是多样化。

This claim is hard to prove, but many people would admit that in the times of stress they eat more and grow fat easier.

Whether this is a purely psychological phenomenon, or a physiological as well, I do not know. It is possible that the brain associated stress with a possibility of starvation, and wants to store more resources for future contingencies, so we feel more hungry, and it is also possible that stress produces some involuntary gene expression, which helps storing more fat.

In any case, limiting stress is certainly beneficial to a decent weight.

Of course, “limiting stress” is easier said than done, but there are certain measures that can be undertaken even when some external pressures will not abate by themselves.

1. Sleep more. Yes, this is very banal, but sleeping is equivalent to eating for the brain, to a degree. Everybody’s sleeping requirements are different, but about 9 hours is usually a good guess. It is believed that sleeping \(3*x\) hours is better than other numbers. That is, sleeping for 6 hours is better than for 7 hours. Most of us dedicate 8 hours, so the advice of this HOWTO is to try sleeping for 9 hours instead.
2. Go to bed early. After all, it does not really matter when you sleep, so I recommend going to bed right after sunset, as this is the most natural to the body clock. This also means that you will have some quiet time in the morning, when no cars are running, and the city is quiet. Also, going to bed right after you get off work has a nice benefit of resting right after getting tired, so your best relaxed hours (in the morning) are spent on your tasks, not your boss’ tasks.
3. Go for a walk. Walking is a good thing, it pumps lymph in your body, as lymph does not have any other pump. Lymphatic system does not have a heart. Finding time for a walk is usually a bit hard, but I just managed to have a walk while going to work. Just get off one metro station earlier or later, and walk to your office building.
4. Plan some time to relax. This seem banal, but the point of planning rest is that you can actually plan high quality rest rather than follow your bodily instincts in just grabbing whatever entertainment is available at hand. Use high-quality entertainment planned in advance instead! Yest, entertainment can also be high and low quality! Reading books is usually better than binge-watching Douyin. Going for a day trip on a weekend is usually better than playing computer games for 40 hours in a row.

The above measures will not lead to a measurable weight loss, but they will make keeping a diet easier.

In order to lose weight, it is necessary to eat less. Everyone sort of knows this maxim, but, judging by my conversations with people who tried to lose weight like I did, many people do not understand what they are doing.

Firstly, many people seem to believe that it is enough to count the amount of calories you eat.

While this might work for some people, I found that it does not work for me.

You see, people do not eat energy, people eat food, and food is measured in grams (or kilograms). While it is possible to find an amount of energy equivalent to consuming a certain amount of food, most of these calculations are highly dependent on the bodily efficiency of each particular individual, so I am finding most of them dubious.

What is, on the other hand, possible to calculate is the amount of carbohydrates, fats, and proteins consumed. Of course, this calculation is also approximate, as not all carbs, fats and proteins are equal, but at least they bear some resemblance to reality and do not rely of dubious formulas.

Moreover, the law of conservation of mass is much easier to grasp for a human than the law of conservation of energy, and unless nuclear reactions are involved (and they are not), it is good enough: if you consume less materials than your body uses and discards, after a consumption cycle (about a day or a few), you will have to weigh less than before, that’s just it.

(CFP is “carbs, fats, proteins”.)

This section is very subjective, but there are “overall numbers”, which seem to be good enough for most people.

Firstly, you need to find your normal body weight.

This is not easy, as everyone is different, men and women are different, and among even men and women of equal height, bone density and diameter differ, but the good thing is that this only needs to be done once for your body.

Okay, theoretically not once. The number is going to change if you regularly go to the gym and purposefully work on increasing your muscular mass. If you do, refer to your trainer and medical doctor tracking your progress, this HOWTO is not for you.

For the people who want to develop the program once, follow it forever, and forget the formulas, finding a proper weight should work. I cannot tell you a correct formula, your have to find it yourself, but there are plenty.

After you have found your desired weight, you need to find out the daily amount of CFP needed to maintain the normal weight, per kilogram of desired body weight:

1. Carbohydrates :: 4 grams
2. Fats :: 1 gram
3. Proteins :: 1+ gram

Let us have a look at a person who weighs 60 kg (something between a man and a woman):

240g of carbs, 60g of fat, 60g of protein.

These numbers are intended to contribute to total consumption of the body, which primarily consists of two channels:

1. Repairs
2. Energy

Now, in principle, all three resources are used used for both channels, the body needs all CFP to build from, and it can also “burn” all three CFP to produce energy. This is not too important, but carbs are transformed into ATP, fats are turned into ketone bodies, and proteins … I don’t know, but in any case, burning proteins is a very bad idea and a sign of your body malfunctioning to a large degree.

In practice, though, the body will most naturally use the resources for repairs with higher priority than for energy. After all, human body can be remarkably efficient at turning stuff into energy, but building … not so much, otherwise we would have all been bodybuilders.

Even if you eat just the amount of resources above, you will be already losing weight, because this is a bare minimum needed to maintain the desired weight, so there will be a delta between what the current body weight would need and the target body. You will be losing weight very slowly, but for many people that is enough.

Now, the first disillusionment maxim: Losing weight is a process which cannot be finished. Indeed, after you have found a correct amount of food, you have to use it until the end of your life, because, guess what, you will not grow up any more, your body is a now a closed system and whatever comes in has to come out or stay.

Eventually, after you have reached your desired body weight, it is best to keep the amount of food just equivalent to the recommended intake of CFP.

Okay, fine, if you want to lose weight faster, you can reduce the numbers mentioned in the previous section further. Further, but not that further.

Okay, there is a way to lose weight very fast, it is called “fasting”. Basically this means that you stop eating completely and only drink water. Very soon (in a few days), your digestive system will deactivate and your will stop feeling hungry, unless you see food.

You can fast for a week, or even longer, not eating any food, and losing weight very quickly. I am not sure how quickly, but in any case I do not recommend doing that at all.

While it is not that unhealthy as many people believe (in fact, dangerous effects from starvation would only appear after about … well, more than a week), it is also practically useless. You will lose weight fast, but you will also gain weight very fast after resuming eating, and you would have to follow the “correct CFP” diet after fasting anyway, so the “fasting streak” will not give you much.

However, you can reduce the amount of CFP below the bare minimum, and it will make you lose weight faster, even though not that faster.

What can you cut?

Practically speaking, the only resource you can really cut from the above numbers is carbs. Unfortunately, as I said, not all fats and proteins are equal. Some fats that are stored into your body cannot be used for construction, you have to eat some external fats, and the body cannot really store proteins at all, you need to eat them every day.

But cutting carbs is much possible. If you eat, say, 120g of carbs per day, no problem, the body will break down more fats to compensate for energy loss, which is more or less what you need, because most of the “bad weight” consists of fat.

Cutting fat… well, it not that recommended. I guess (do not trust my word), that it is fairly safe to go from 60g to 30g of fat, but I cannot prove it, talk to your doctor.

Again, do not overdo it.

The second disillusioning sad story: you will feel hungry.

Unfortunately, feeling hungry is natural for adults. This is one of the eye-opening realisations of how adults differ from children. Children can eat and feel full, and it is normal for them, most of the things they eat will turn into growth.

Adults cannot. Moreover, millennia of evolution produced in human adults a feeling of hunger which exceeds the amount needed to just consume the CFP needed to maintain normal weight.

This is where “semaglutide” is trying to intervene, by reducing this sense of hunger in adults. But this HOWTO is not about semaglutide, because I managed to successfully lose weight without it.

How do you find a correct composition of your diet?

CFP is often written on the back of the packs of food you are buying, or it is possible to search for it online.

By all means, go and read through your favourite nutritional information website and look and find you best diet. You do not have to follow my advice below.

But my desire is to feel as little hunger as possible given my (male, fairly tall) size, so I want to eat large volumes, and I am blessed by the nature of not being a gourmand, so I can eat what other people consider to be boring.

Which means that the food I am allowing myself to eat actually boils down to the following, very meagre list:

1. Carbs
   1. Cabbage :: 2/0/1 (all sorts of it, including cauliflower and broccoli)
   2. Cucumber :: 3/0/1
   3. Tomato :: 4/0/1
   4. Courgette :: 4/0/1
2. Proteins
   1. Chicken breast :: 1/4/31
   2. Prawns/Shrimp :: 1/1/20
   3. Horse meat :: 1/5/40
3. Desserts
   1. Egg :: 1/10/13
   2. Onion :: 10/0/1
   3. Garlic :: 3/0/1
4. Milk :: 5/3/3
5. Tea, Coffee :: 0
6. Sparkling soda :: 0
7. Cooking oil :: 0/0/99
8. Tomato sauce, Mayonnaise, Mustard :: 15/50/2

Even if you cut your carbs intake to 60g per day, you can still eat 3000 grams of cabbage, which is more than enough to not feel hungry any more. (3000 grams is still not infinity.)

Protein is not that detrimental to weight loss. The body is not very likely to turn it into fat (although overdoing proteins can have other negative effects), the limiting factor for chicken breast is fat. 60/4 = 15, which makes it about 1500 grams of chicken breast per day. Again, defeating hunger with 1500 grams of chicken breast and 3000 grams of cabbage is quite possible.

The biggest contributors to fat are, surprisingly, sauce and eggs. Eggs have a lot other, non CFP elements, such as vitamins, so eating an egg per day is probably worth it, but and egg becomes a dessert. Eating cabbage and chicken breast without sauce is tiresome, so I allow myself a spoon of it, 15g, which is already 7 grams of fat.

So, to sum up, I am usually eating once a day, a meal of about the following:

1. Roasted or boiled cabbage, 1kg :: 40/0/10
2. Roasted chicken breast, 500g :: 5/20/150
3. Garlic 50g :: 5/0/1
4. Cooking oil, 15g :: 0/15/0
5. Sauce :: 0/7/0
6. Coffee with milk 500ml :: 10/6/6

55/50/170

How do I measure those weights so precisely? Well, I bought kitchen scales which are ridiculously easy to use and check my components before cooking.

I am also drinking a lot of decaf coffee and tea, and pure water, and sometimes eating a cucumber or a tomato.

In total: 60/60/170

I would also like to say a few words about desserts and candy.

As can be seen from the “menu”, onion, garlic, and eggs end up being desserts in this program, rather than being normal food. This is not as bad as it sounds, after a couple of weeks of a cabbage diet, garlic and onion start tasting really sweet, but in any case, they are spicy, and it is hard to eat a lot of them (I only eat them raw and juicy).

Tea, coffee, and soda water are listed in the “menu”, even though their CFP value is close to zero. They are a huge help in combating hunger during the day. Warm coffee with milk is just excellent at reducing hunger, and tens of different soda flavours, seemingly identical, really start tasting really different after some time, and you can drink as many of them as you want.

It is very helpful to eat regularly, mostly because it makes monitoring progress very more consistent, because you body has the same weight in the same hours of the day, therefore measurements are “of the same entity”, comparing apples to apples.

Measurements are very important to understand whether you are doing into a correct direction, and to increase the feeling of success. (A popular maxim says “if you want to achieve a goal, first you need to measure it”.)

As I mentioned above, I eat solid food once a day, but I drink a lot of decaf coffee during the day, which is very helpful to not feel hungry.

Contrary to popular belief, it does not really matter if you eat in the evening or during the night, it seems to take about 3 days to incorporate the materials consumed into your body.

So I eat before going to sleep, because when feeling full, falling asleep is much easier than when hungry.

I am measuring my weight once a day, in the morning, when the water drunk and anything eaten not according to the schedule cannot affect the measurement.

(Be aware that any material going through the digestive tract cannot take any of your body weight away. The actual molecules which are leaving your body and are leaving through one of the three channels:

1. urinary tract
2. sweating
3. breathing out CO₂)

I am writing down the results on paper, but in the past I used to just write them on the bathroom mirror with a marker, since I have my scales there and I can measure my weight without clothing without disturbing anyone with an unsightly show.

With such a schedule, it is possible to achieve weight loss speed of about 1.5 kg per week. Do not try to exceed this speed, it is not healthy. 1.5 kg per week is a solid observable number, which brings a lot of joy, helping to maintain the schedule.

Suppose you want to lower your weight from 90 kilograms to 60 kilograms (a short, but very overweight man). It would not be healthy to do this in less that 20 weeks, but in about half a year this should be doable.

On the other hand, I cannot vouch for the fact that the above observations will be applicable to women mutatis mutandis, because body fat plays a larger role in female metabolism than in male metabolism. Fat is actually producing some female hormones, so its percentage in a female body is naturally higher, so, firstly, probably females need to consume more fat than men, and are probably not expected to reach 1.5 kg per week weight loss.

Some of this weight loss will be muscular weight, unfortunately, this cannot be avoided. Losing fat is inevitably accompanied by losing muscles too. But for me this has never been much of a concern, I am not doing a lot of physical work.

Exercising can possibly make muscular loss less prominent.

But in any case, it is very nice to have a table with the number “how my body weight goes down”. On some days it will actually go up, do not expect your weight to decrease by 200 grams every day, but on average it should be possible.

Eat a lot of food from the list above.

Instead of eating a 1 kilogram of cabbage, eat 3. If 500 grams of chicken do not satisfy you – add up 500 more grams of shrimp. If even that fails, add 500 more grams of horse meat.

Most of the time when we are DESPERATELY HUNGRY, the desire can be squelched by variety better than by a sheer amount. The list above, no matter how meagre, still gives you some slack in choosing food.

Well, maintaining a constant weight is, in some sense even harder than losing weight.

Now you have to care not just about the KPI going up, but also about it going down unduly.

So, if while “losing weight” you only have to care about “not exceeding” the value plus tolerance, now you also have to care about not cutting too much.

This is also the time when the rough target of 240/60/60 stops being valid, because you really need to be “within a reasonable margin” of your bare minimum.

But at least you can congratulate yourself with an achievement.

You would probably want to switch to eating twice a day, not once. Which will also make measuring your weight harder, but it is probably possible to measure the weight twice a day, before each meal. This is actually much worse than measuring once, in the morning, but better than nothing. You can probably use “smart scales”, as writing down twice-a-day measurements is tiresome.

Overall, 240 grams of carbs allows a variety much larger than 60, but I generally like increasing amount, not increasing variety. As I said, I am not a gourmand.

We will build grub-2.12 first, because it is easier, and because later we will be using its build artefacts. The touch command is needed, because GRUB developers just do not care if their software fails to build.

1. cd ~/grub/ ; gtar xvf grub-2.12.tar.gz ; cd grub-2.12
2. touch grub-core/extra_deps.lst

3. MAKE=gmake ./configure --disable-werror --disable-nls && gmake -j1

   Check results:

root:~/grub/grub-2.12>ls -l ./grub-core/{diskboot.img,boot.img}
-rwxr-xr-x  1 root  wheel   512 Sep 24 16:00 ./grub-core/boot.img
-rwxr-xr-x  1 root  wheel   512 Sep 24 16:00 ./grub-core/diskboot.img

Both must be exactly 512 bytes.

1. cd ~/grub ; gtar xvf grub-2.06.tar.gz ; cd grub-2.06
2. sed -i ’s/-falign-jumps=1//g’ configure
3. MAKE=gmake ./configure –disable-werror –disable-nls && gmake -j1
4. gmake install

We can see that the two images are, indeed, broken:

root:~/grub/grub-2.06>ls -l ./grub-core/{diskboot.img,boot.img}
-rwxr-xr-x  1 root  wheel  531 Sep 24 16:07 ./grub-core/boot.img
-rwxr-xr-x  1 root  wheel  531 Sep 24 16:07 ./grub-core/diskboot.img

But do not worry, we will replace them:

1. cd ~/grub/grub-2.12
2. cp ./grub-core/{diskboot.img,boot.img} usr/local/lib/grub/i386-pc

Now we can use grub-related commands.

As root

 echo test | nail -s  "Test $(( I = I + 1 ))" user

This should create a message in the user's "mailbox", which is usually /var/spool/mail/user

Well, not really. It is, indeed, not terribly useful by itself, but email is a thing which is inadvertently implementing a very widespread data structure: a series of entries with headers.

This data structure is so ubiquitous that we often do not even recognise it, but it everywhere.

It allows a lot of operations necessary for meaningful data organisation of a human.

In theory you could put everything into a single file, which is trivial to implement, because:

1. a flat file does not have boundaries between messages
2. is append(overwrite)-only
3. filtering is by pattern only

A messagebox is strictly stronger than a flat file:

1. you can delete messages
2. you can sort the box
3. you can filter messages by context
4. (optional) messages might be linked into a tree

A messagebox is weaker than a directory, because a messagebox does not support tree-like organisation. However, the maildir format, combined with IMAP allows a tree-like directory for messages. In theory this makes mail as expressive as a file system.

There is the GMail system of mail organisation using labels. Labels are a totally different beast, and I do not yet have a clear opinion on them. They are also ubiquitous, but I need to think more about them.

But "mail" is readily available, and is "at least" the most standard way to report actionable errors to the user on UNIX. If something breaks, is one-off, deserves user's attention, the best bet is to send an error message to the user in question.

Sometimes it is okay to just append an error message to the log file, but, again, the point is that messages are mutable, so a user can clear them off, whereas log files are append-only and are dedicated to faithfully representing history.

You can, and possibly even should, if (a) you trust your server enough to put an ssh login key onto this server, (b) your target machine accepts ssh/scp. This is often not the case, and badmail.test does not accept ssh at all.

Moreover, email is, so far, the only method of receiving messages anonymously, and which has a lot of provisions for reducing even anonymous spam.

Well, yes. You can do something like this:

 MX=$(dig MX badmail.test | grep -A1  'ANSWER SECTION' | tail -n 1 | awk  '{print $6;}')
 MX= "${MX%.}"
 echo test | nail -s  "Test $(( I = I + 1 ))"  \
                 -S  mta= "$MX"  \
                 user@badmail.test

This is likely going to fail even on badmail.test. Firstly, because even badmail.test is going to reject your mail with a thing called "Greylisting". That is they will give you a "soft error", like "we are busy, come later".

To avoid this loop, and to avoid querying the mail server yourself, you are expected to use this postfix.

 MX=$(dig MX badmail.test | grep -A1  'ANSWER SECTION' | tail -n 1 | awk  '{print $6;}')
 MX= "${MX%.}"
 echo test | nail -s  "Test $(( I = I + 1 ))"  \
                 user@badmail.test

This will also probably fail, and you won't even know this, because postfix will not even send a "bounce" (delivery error) message to you, so you have to look for a "bounce" in a log file.

The message will probably be that the sender address (that is effectively user@mymachine.local) is not valid. It is not valid because mymachine.local is not a globally recognised domain name.

Now did I mention registering a name on DNS in order to send mail from your own name? Maybe I didn't, but you have to, obviously.

But even giving this machine a real name will not solve your issue, because those domain names might expire, and you want your errors to be delivered with certainty.

But, yes it is the time to find out how to register a domain, and be ready to give it to your machine.

Well, let us first identify how to properly send mail "by hand".

 MX=$(dig MX badmail.test | grep -A1  'ANSWER SECTION' | tail -n 1 | awk  '{print $6;}')
 MX= "${MX%.}"
 l_pattern= 'bounced'
nohup tail -n0  --follow=name /var/log/maillog 2>/dev/null |   \
  grep --line-buffered -i -F  "$l_pattern" |  \
  tee |  \
   while  read -r l_inputline ;  do
     while ! su nobody -s/bin/sh -c  \
             "nail -s '/var/log/maillog:bounced' \
          -r 'account+mailer@badmail.test' \
          -S mta='smtp://${MX}:587 \
          -S smtp-use-starttls \
          -S smtp-auth=none \
          -S v15-compat=yes \
         account+mailer@badmail.test" <<<  "$l_inputline"
     do
       rc=$ ?
      logger  "sending bounce notification failed:inputline=$l_inputline"
      sleep 60
     done &
    disown
   done

I added a little more here to help you. This is effectively a log watcher, which can watch for all sorts of errors, and mail them to your receive-error-only mail account. This one watches for mail bounces, but it could watch for anything.

The positive side of this watcher is that it does not require the mail server to work on its own.

Now what is important in this log watcher script? Do you see the -r 'account+mailer@badmail.test' line there? It could have been -r '', which is called "null recipient". (Actually maybe this is even better.)

It lets you go around that error of the original server having a broken hostname. Again, we want to receive error messages even if our server is totally mis-configured (say, by invaders).

Okay, I already mentioned the config file called main.cf, where most of the postfix configuration go, now I will mention main.cf, which is its "service supervisor" config file.

master.cf defines the components that postfix is running. We will later use it to allow message submission on encrypted ports, but now we will define a separate delivery method (using msmtp) for our badmail.test.

Add the following line to your master.cf:

msmtp_for_badmail unix -    n    n    -    50   pipe    flags=R  user=nobody  argv=/etc/bin/postfix-to-msmtp-wrapper.bash --sender=${ sender} --user=${ user} --extension=${ extension} --recipient=${ recipient}

Where /etc/bin/postfix-to-msmtp-wrapper.bash is a script which reads the arguments and calls msmtp about like this:

 exec msmtp --host= "$MX" --port=587 --tls=on --from= "$recipient"  "$recipient" 2>&1

I won't give you full script, this is a trivial exercise.

You also need to specify that mail to this server should be sent in a special way.

main.cf

minimal_backoff_time = 60s
maximal_backoff_time = 600s
transport_maps = regexp:/etc/postfix/transport_maps

And in transport_maps:

/^.*@badmail.test/ msmtp_for_badmail:

Well, test it by sending mail like this:

 echo  "Test $(( I = I + 1 ))" | nail -s  "Test $I mail to badmail"  \
     root@my-domain.test

This should pass. Look into your /var/log/maillog for greylisting errors and such, and wait a few minutes for the message to be delivered.

This time I suggest not writing a mail, but rather making a cron entry like this:

* * * * * echo "Test $(date --iso=seconds)" 1>&2

Messages should be delivered to both user inbox, and your wonderful badmail account.

In general you want to have your mail encrypted. However, receiving mail over the classical unencrypted port 25 would still be nice if you have some old devices, and old does not mean that old, at the time of writing Android 6 already have expired TLS certificates and balk at random occasions.

By default Postfix allows unencrypted port 25 mail for local delivery, so let us try it from your laptop.

 echo  "Test $(( I = I + 1 ))" | nail -v -s  'test-public-port-25'  \
        -S  mta= 'smtp://subdomain.my-domain.test:25'  \
        -S smtp-auth=none  \
        -S v15-compat=yes  \
       user@my-domain.test

This should work and deliver mail to user, as well as duplicate it to your badmail.

 echo  "Test $(( I = I + 1 ))" | nail -v -s  'test-public-port-25-encryption-outbound'  \
        -S v15-compat=yes  \
       noreply@oligarch.com

And look at the postfix log. Delivery should fail, but Postfix log should have information about the TLS quality.

You can disable smtp_tls_loglevel afterwards.

On your laptop:

 echo  "Test $(( I = I + 1 ))" |
  nail -s  'test-submission-port-587'  \
       -v \
       -S  mta= 'smtp://mailer.domain.test:587'  \
       -S smtp-use-starttls  \
       -S smtp-auth=none  \
       -S v15-compat=yes  \
       root@mailer.domain.test

Connection should work, but sending mail should fail with the reason:

SMTP server: 554 5.7.1 : Recipient address rejected: Access denied

I haven't investigated it deeper, but it seems that this message is misleading. It is not the recipient address which is rejected, it is that the client is not authorised. The reason this happens is that postfix processes the client connection line by line, and the authentication is checked after the RCPT TO command is issued by the client.

SMTP does not have anything like "AUTH anon" command, which could fail with something like "unauthorized", the command may simply be missing, therefore the place where postfix fails is RCPT TO.

Why the error message is misleading? Well, I don't know.

On your laptop.

 echo  "Test $(( I = I + 1 ))" |
  nail -s  'test-submission-port-465'  \
       -v \
       -S  mta= 'smtp://mailer.domain.test:465'  \
       -S smtp-use-starttls  \
       -S smtp-auth=none  \
       -S v15-compat=yes  \
       root@mailer.domain.test

This test should connect, but timeout, because STARTTLS is a plain TCP connection, whereas port 465 expects a TCP+TLS conection.

Note that there is no smtp-use-starttls, but there is smtps://.

On your laptop.

 echo  "Test $(( I = I + 1 ))" |  \
  nail -v -s  'test-submission-port-465-tls'  \
       -S  mta= 'smtps://mailer.domain.test:465' \
       -S smtp-auth=none  \
       -S v15-compat=yes  \
       root@mailer.domain.test

This test should fail with the message:

SMTP server: 554 5.7.1 : Recipient address rejected: Access denied.

Again, this message is misleading, see Test 1.

On your laptop:

 echo  "Test $(( I = I + 1 ))" |  \
  nail -v -s  'test-submission-port-465'  \
       -S  mta= 'smtps://login:password@mailer.domain.test:465' \
       -S v15-compat=yes \
       root@mailer.domain.test

This should pass, and your root should receive this message, as well as your debugging account on an unrelated mail service.

On your laptop:

 echo  "Test $(( I = I + 1 ))" |  \
  nail -v -s  'test-submission-port-465'  \
       -S  mta= 'smtps://login:wrong_password@mailer.domain.test:465' \
       -S v15-compat=yes \
       root@mailer.domain.test

This test should, obviously, fail.

The error should be 535 5.7.8 Error: authentication failed: authentication failure.

 echo  "Test $(( I = I + 1 ))" |  \
  nail -v -s  'test-submission-port-465'  \
       -S  mta= 'smtps://login:pass@mailer.domain.test:465' \
       -S v15-compat=yes \
       -r  'user@mailer.domain.test'  \
       my-nick@oligarch.com

This may or may not work without DKIM. In general, SPF alone should be enough, if it is not, go to But this is not very sweet, because we do not want to send mail from the mailer machine, unless it is diagnostic mail. What we really want is "cloud mail", that is sending mail from an address like "my-nick@domain.test", not "user@mailer.domain.test".

Let us go to the test number 2.

On your laptop:

 echo  "Test $(( I = I + 1 ))" |  \
  nail -v -s  'test-submission-port-465'  \
       -S  mta= 'smtps://login:pass@mailer.my-domain.test:465' \
       -S v15-compat=yes \
       -r  'user@my-domain.test'  \
       my-nick@oligarch.com

This may or may not work without DKIM. Again, SPF is often enough.

 echo  "Test $(( I = I + 1 ))" |  \
  nail -v -s  'test-send-to-my-domain' \
       -S  mta= 'smtps://mailer.my-domain.test \
       -S smtp-use-startls \
       -S smtp-auth=none \
       -S v15-compat=yes \
       -r 'me@laptop ' \
       someone@my-domain.test

This should fail with 554 5.7.1 : Recipient address rejected: Access denied

This is because our domain is mailer.my-domain.test, not my-domain.test

Virtual maps is effectively what allows forwarding of mail that does not belong to the world of UNIX machines talking to each other, and instead connects people among themselves.

Postfix has two different kinds of "virtual" redirections, "virtual aliases" and "virtual mailboxes". The documentation for them is obscure and impenetrable that it's unlikely that a human being can understand it, but if you wish to try, here is the link:

1. https://www.postfix.org/ADDRESS_CLASS_README.html
2. https://www.postfix.org/VIRTUAL_README.html

 # main.cf
virtual_alias_domains = my-domain.test
virtual_alias_maps = regexp:/etc/postfix/virtual_maps_to_oligarch

 #  /etc/postfix/virtual_maps_to_oligarch
/^(.*)@my-domain.test/ user@oligarch.com

 cd /etc/postfix ; postmap virtual_maps_to_oligarch

We never intend to validate dkim, but we want to sign DKIM on at least the messages that we ourselves send from 'user@my-domain.test'. Well, actually, the actual thing is that we will sign all mail regardless. We do not intend to do any relaying of alien mail, all the mail is either originating from us (and thus deserves a signature), or is coming to us, so having it signed does not hurt.

OpenDKIM is very annoying to set up.

Set up signing with OpenDKIM

 # main.cf
smtpd_milters = inet:localhost:8891
non_smtpd_milters = inet:localhost:8891

LogWhy                 yes
Syslog                  yes
SyslogSuccess           yes
Canonicalization        relaxed/simple
Mode s
ExternalIgnoreList /etc/opendkim/refile_match_everything
InternalHosts /etc/opendkim/refile_match_everything
KeyTable refile:/etc/opendkim/keytable
SigningTable refile:/etc/opendkim/signingtable
Socket                  inet:8891@localhost
ReportAddress           postmaster@my-domain.test
SendReports             yes
UserID opendkim:opendkim

   # /etc/opendkim/refile_match_everything
127.0.0.1
::1
::/0
0.0.0.0/0

   # /etc/opendkim/keytable
default._domainkey.my-domain.test mydomain.test:default:/etc/opendkim/keys/default.private

  # /etc/opendkim/signingtable
* default._domainkey.my-domain.test

Test by trying to send mail to your friend, or your second OLigomail account and monitoring either OLigomail "Show original", or postfix log files for DKIM.

Try registering at cmake discourse, and look in postfix logs if the message goes through.

GMail has a decent GUI for editing filtering scripts. Unfortunately, they are not Sieve, and are strictly weaker. They can be exported as XML.

For the rest of us, we have to deal with writing Sieve code. Fortunately, it is easy, unfortunately, it is writing code.

There are at least three dedicated Sieve editors, which can manage scripts remotely:

1. Emacs/Sieve-mode
2. Thunderbird thsmi/sieve: Sieve Script Editor https://github.com/thsmi/sieve
3. KMail sieve editor

Thunderbird extension even has a decent GUI.

ManageSieve supports STARTTLS, so it is encrypted, but not TLS, which is bad. (This issue exists.)

Does not exist. Nuff said, unfortunately this is true.

If you have Termux on your android, you can edit the scripts in a semi-convenient manner:

1. Write a script in Emacs (on Android) Sieve-mode
2. Run a script to update your script.
3. sieveshell is in the "cyrus-imap" package, I had to compile it on termux. Termux for Android 8+ has everything you need, Android 6 might need a bit of manual work.

spawn sieveshell  --user=myuser --authname=myuser  myserver.test
expect "Please enter your password: "
send "\n"
expect "> "
send "deactivate\n"
expect "> "
send "delete first\n"
expect "> "
send "put first-filter.sieve first\n"
expect "> "
send "activate first\n"
expect "> "
send "quit\n"
expect eof

It does not let you do filter previews, unfortunately, but overall is a decent hack, fairly ergonomic.

IMAP IDLE is a mechanism for Push notifications for IMAP. There is a modification of it called "NOTIFY", but I did not study it. There is an even newer mechanism called "JMAP", but I did not study it too.

The issue with IDLE is that it only supports subscription to a single mailbox (or, rather, it is maintaining a TCP connection per mailbox), and you are probably expected to have at least a hundred of mailboxes, if not over a thousand.

In theory it is your own server, you can maintain a thousand connections, but this is inconvenient.

Let us do a better trick:

1. Make an "All-Mail" mailbox, where we will be copying all the messages, just in case. fileinto "All-Mail";
2. Copy messages you want to be notified of to a separate "NOTIFICATIONS" directory. It can also be "INBOX", really, as this is the most natural directory to subscribe to.
3. Update your mail when a notification arrives.

I call this method "IDLE-BRUTEFORCE". You can improve on it but making your IDLE connection wander about the IMAP directories and checking if they are updated, fetching them when necessary.

In a few words, there is an IMAP extension which allows one to subscribe to multiple directories, not just one.

It is just a single command, run before entering IDLE, which changes IDLE behaviour in that it starts sending notifications for all IMAP mailboxes (directories). After the mailboxes list is extracted, you can run synchronisation for those mailboxes.

GMail does not support NOTIFY, but it has

1. A mailbox called [Gmail]/All Mail, which has references to each message in the account.
2. A custom IMAP command X-GM-LABELS, which would return a list of directories which also reference this message.

So, make IDLE monitor All Mail, fetch its labels, and synchronise those "labels-mailboxes".

There is a hacky and broken script which does the sync:

TODO