How to install and start using LineageOS on your phone.

1. https://source.android.com/docs/setup/download/source-control-tools
2. https://www.akshaydeo.com/switching-branch-with-repo-for-android-source-code/
3. https://source.android.com/docs/setup/reference/repo

repo is a tool to fetch Android. It is essentially a different (to git submodule) approach to multi-repository source trees.

You initialise it by running

repo init -u https://mirrors.tuna.tsinghua.edu.cn/git/lineageOS/LineageOS/android.git -b lineage-23.2  --git-lfs

The same command should also work when switching versions. In theory it is enough to just replace the branch name.

There are tricks, however. repo is not error-resistant. When it fails, it does not take care to restore the original state of the source tree, it just fails, often leaving individual git repos in an inconsistent state. You might have to repair those repos manually, usually by aborting merges, committing uncommitted files, or something like that.

repo is basically just helps maintaining a list of git repositories, revisions, and remotes, and calls git commands for individual repos.

The files to study to understand it are: ./repo/manifests/default.xml and ./repo/.repo/local_manifests/local_manifest.xml.

The first one is the list of repos for the main tree of Android, and the second one is for you to add references to repos specific to your device.

Remote server definitions are quite self-explanatory, except, perhaps, a special path called "..". This is a parent directory with respect to the default repo.

There is an XML tag for “default settings”, called default:

<default revision="refs/heads/lineage-23.2"
         remote="github"
         sync-c="true"
         sync-j="4" />

Official build system documentation is so terrible that I am not even linking it here, as it’s completely pointless.

It is convoluted and consists of several mutually duplicating parts.

You might hear the words “Soong”, “CKati”, “Jack”, and “Bazel”. There are also shell functions (pay attention, they are not files!, they are bash functions), which are expected to control the build, and they also seem to have bit-rotted a bit.

First of all, go to build/make, and add a pair of commands to enable ccache when building Android.

diff --git a/envsetup.sh b/envsetup.sh
index 12a69fc77b..d20f01aac7 100644
--- a/envsetup.sh
+++ b/envsetup.sh
@@ -15,6 +15,10 @@
 # gettop is duplicated here and in shell_utils.mk, because it's difficult
 # to find shell_utils.make without it for all the novel ways this file can be
 # sourced.  Other common functions should only be in one place or the other.
+
+export CC_WRAPPER=/usr/bin/ccache
+export USE_CCACHE=1
+
 function _gettop_once
 {
     local TOPFILE=build/make/core/envsetup.mk

Or you can add the same lines to your own bash build script, which is expected to source the aforementioned source build/envsetup.sh. This is documented somewhere, but the page mentioned does not seem to have this any more.

The bash functions you might find are the following:

1. m
2. mm
3. make
4. mka
5. breakfast
6. brunch
7. lunch

In theory they were expected to have their own meaning, in particular, breakfast is expected to select the device configuration for building. However, in my experience you usually only have a single device in the source tree anyway, so you do not need it.

There should be a way to switch between debug, userdebug, and release configurations with these:

lunch lineage_<codename>-<build number>-<build-flavour>

Where codename is your device code name (you have to find it out!), and it is not the same code name as board code name. For example for my device the code name is xigua (Chinese for Watermelon), and the board name is kalama (a city name).

But in my case just typing brunch xigua was enough.

In any case, in order to just run the build with a single command, I wrote a bash script:

export CC_WRAPPER=/usr/bin/ccache
export USE_CCACHE=1
( reset
rm out/target/product/xigua/lineage-23.0-*-UNOFFICIAL-xigua.zip*
source  build/envsetup.sh
( cd  ~/Lineage-Camera/cam-oplus-15/
 ./mine_repack-cam.bash || exit 1
 exit 0 ; )
( cd ./vendor/oplus/camera/
  ./extract-files.py --no-cleanup  ~/Lineage-Camera/009_15.0.0.860 || exit 1
  exit 0 ;  ) || exit 1
printf "mine:camera files extracted\n"
( cd device/oneplus/xigua
  ./extract-files.py --only-target --no-cleanup  ~/Lineage-Camera/009_15.0.0.860 || exit 1
  exit 0 ; ) || exit 1
printf "mine:device files extracted\n"
export CC_WRAPPER=/usr/bin/ccache
export USE_CCACHE=1
brunch xigua  || exit 1
exit 0 ; )
RETVAL=$?
if (( RETVAL == 0 )) ; then
  espeak 'compilation finished with success'
else
  espeak 'compilation finished with failure'
fi

This build script is more complicated that I mentioned, but there are reasons for that.

1. There is a need to extract binary drivers from the original firmware.
2. There is a need to modify the camera app so that it would run on LineageOS.

It might happen that you do not actually need (2), but you will probably need (1).

This is documented here quite decently: https://wiki.lineageos.org/extracting_blobs_from_zips_manually.html

1. Android Studio :: https://developer.android.com/studio
2. Android Studio for Platform :: https://developer.android.com/studio/platform
3. Gradle :: https://gradle.org/
4. Emacs + jdtls

I need to mention these, because they exists. I did not use them much, except for writing a small polyfill library, so I cannot produce a meaningful critique. In general, Android Studio is quite a decent tool, it is a version of JetBrains IDE, so, basically, the best IDE available in the ecosystem. It is a safe choice, unless you are good enough of a developer to assemble your own tool-belt, based on Emacs.

If you want to setup Android Studio for Platform, see the howto file in the LineageOS repo: ./lineage/wiki/pages/how-tos/import_to_android_studio.md

I use Emacs for development, but even though Emacs supports Eclipse JDTLS LSP server, it does not support Android java, so I ended up doing find+grep (fd+rg), like in the old times.

1. https://source.android.com/docs/setup/build/building-kernels

Okay, I know nothing about building the kernel for LineageOS, except that it is built with bazel, and is shipped in three source tree modules:

1. kernel/<manufacturer>/sm8550
2. kernel/<manufacturer>/sm8550-devicetrees
3. kernel/<manufacturer>/sm8550-modules

Kernel is one of the most interesting things in any Linux system, but I did not need to deal with it yet.

The only thing to note is that Android is not using udev to determine hardware configuration in runtime, it requires that all system devices be declared in advance in a data structure called “device tree”, which I know nothing about except that it is compiled and that it is also used in Linux on Raspberry Pi.

About modules I also know nothing except that, as far as I understand, the kernel has to be Open Source, as it is under GPL, but modules do not, and a lot of phone manufacturers are providing drivers for their devices only as pre-built modules.

The entry to the kernel build system is vendor/lineage/build/tasks/kernel.mk.

This is another incredibly badly documented part of Android, even though it is central to its architecture. (https://source.android.com/docs/core/architecture/vintf)

So, Android is an operating system, and it needs to talk to hardware. But it does not want to talk to hardware directly, querying the drivers, or udev or /proc or /sys, it wants to talk to userspace daemons over Binder, and those daemons can implement the userspace<->kernelspace interaction in any way they want. So it collects a list of requirements , for some obscure reason called compatibility matrix into an XML file, and expects to compare that XML with a device manifest, which is also an XML file, in a similar format, which collects all capabilities of a particular device, which is called a manifest. The same thing also happens in another direction, so we have framework manifest, framework compatibility matrix, device manifest, and device compatibility matrix.

The most confusing thing is that both of those things are happening in the same source tree!

All four XML files are compiled by the build system, and compared to each other to find if some services are not provided.

I did not have to write them myself, but you probably will, if you want to port Lineage to your own device. I can only recommend looking at how this VINTF is declared and requested in device/oneplus/xigua.

Device manifest files are usually placed into “vendor/etc/vintf”, or “odm/etc/vintf”, I don’t know which one goes where, and I don’t know how to tell the build system that “these XML files are for the VINTF, and these are not”.

This part is also documented poorly and I did not understand it.

As mentioned above, Binder services have to be declared in the “VINTF” XML files, and there should be services, implemented as Linux binaries, started by init from /{vendor,odm,system,system_ext}/etc/init/servicename.rc, which provide their services via Binder calls.

Binder has to do with memory mapping, but it’s not just a place in memory into which you can write some garbage, it’s more like Sun-RPC, there are two versions of the library, and there is a serialising-deserialising server between them. (https://en.wikipedia.org/wiki/Sun_RPC) In Sun-RPC, there is an RPC compiler, rpcgen, which creates .c-files from .x files, which can be compiled and linked into the client and server programs, so that they would call functions and do not deal with serialisation and network transfers themselves.

Android Binder does something similar, but instead of rpcgen, there are hidl and aidl.

But I cannot actually give you an example of a three-stage compilation: AIDL, Server, Client, because I have not found it, and I didn’t actually need it for anything.

But AIDL services can be called both from C++, and from Java, and somehow even from apps. I wonder, how AIDL interfaces are getting to apk writers?

In any case, services not just have to be declared in the VINTF, but also registered when they start, with “service manager” or “hw service manager”. I don’t know how this is done, but I have seen a mention of this thing.

But there is more to it. There is such a thing as SP-HAL. SP stands for Same Process.

This means that you can just link to a library, or dlopen a library, and use its functions. They do not have to actually do anything with hardware… I think.

In any case, I have no idea whether all libraries supplied by Android are a part of SP-HAL, or not. Again, all of this is very confusing and poorly documented. If you can explain this to me in a consistent way, feel free to email me.

So, as mentioned in the previous section, Android cares a lot about abstraction barriers between different parties involved into the creation of a device. In most cases there are at least three parties:

1. Google (framework)
2. BBK/OnePlus (ODM)
3. Qualcomm (vendor)

Google thinks that the interactions between them three should be if not tightly controlled, then at least well declared, and hence introduces VINTF.

Now believe me or not, but each of those parties has a partition allocated specifically for itself, and since Project Treble even two (suffixed _a and _b).

Wait, I didn’t say anything about partitioning on Android before. This should be a whole section dedicated to it, but so far I can tell you that “vendor”, “odm”, and “system_ext” are separate partitions. Moreover, at least on my phone, both vendor and odm are read-only images of an ext2 file system, and cannot be remounted read-write, only rebuilt.

Why is that? Well, because the three parties do not communicate to ensure common library interface, and end up with API and ABI incompatibility issues. So, the decision ended up being: give each party its own set of basic Android libraries, and make the dynamic linker ensure in runtime that programs associated with vendor, say, the camera provider services (a process run by init, which is allowed to open /dev/video0 and which services captured frames via a Binder service registered with the servicemanager) can only link dynamically and dlopen .so files from /vendor/lib64, even if it is given an LD_LIBRARY_PATH or LD_PRELOAD.

Linker also enforces restrictions on which libraries can be accessed by .so libraries shipped with apk files.

Now the important question: Which libraries can be used where? Does anybody have a list?

1. https://source.android.com/docs/core/architecture/configuration/add-system-properties

So, Android has a unified configuration system called “properties”. Properties are essentially a global key-value database, accessible through the /proc filesystem, where they are represented as files, which allows them to have SELinux labels. Properties are accessible to console programs with getprop, setprop, and __android_property_get, and through Java API.

They are stored in

1. /default.prop
2. /system/build.prop
3. /system/default.prop
4. /vendor/build.prop
5. some other *.prop files

This part, surprisingly, actually has a decent HOWTO about it written by Google, so I will just link it here:

https://source.android.com/docs/core/architecture/configuration/add-system-properties

I will note that properties are divided into “system” properties and “vendor” properties, distinguished by specific prefixes.

ro. properties are writable only by init, or by the build system. persist properties survive a reboot.

In the build system, properties are defined either with PRODUCT_{PARTITION}_PROPERTIES (in the Makefile), or in TARGET_{PARTITION}_PROP (in separate files).

The biggest question is “why are properties different by partition?”. I don’t know, but if your program or app cannot read or write a property, probably it needs to be moved to a different partition.

Properties have SELinux labels, and those are defined in the files called property_contexts, for example ro.vendor.oplus.camera.frontCamSize u:object_r:vendor_camera_prop:s0

The full list of property contexts is probably somewhere under system/sepolicy or in Qualcomm directories.

Now the good idea would be to make an exhaustive list of all properties with explanation what those properties do. The list can be obtained by running just getprop, but what the meanings… Maybe at least make a wiki page?

I know nothing about this, except that there is a data structure called AHardwareBuffer, and you can manipulate it a lot.

SurfaceFlinger is the Android GUI system, I guess, the only one which can read/write /dev/drm or /dev/dri.

I don’t know anything about them.

1. https://lineageos.org/engineering/HowTo-SELinux/
2. https://source.android.com/docs/core/architecture/aidl/aidl-hals#sepolicy
3. https://stackoverflow.com/questions/36790794/what-is-c512-c768-of-selinux-process
4. https://rtx.meta.security/reference/2024/07/03/Android-system-apps.html
5. https://github.com/SELinuxProject/selinux-notebook/

SELinux is decently documented by “SELinux Notebook”, and the basic principle is not too hard, but the implementation is very convoluted.

SEAndroid (a modification of SELinux for Android) is decently described here: https://lineageos.org/engineering/HowTo-SELinux/

Basically, SELinux works like this: source x target x operation => allow, and everything else is denied.

Firstly, the language is very confusing. Essentially, there is not difference between labels on source and target, so I am calling them “labels”. This is the most visible in the following example: suppose a project A wants to kill project B, then a check would be scontext=label_a tcontext=label_b class=process acton=kill. When a process B wants to kill process A, the action would be the same, but scontext and tcontext would be reversed.

However, you may often see the words “domain” and “context”, which are basically the same thing – labels.

In Android, many things have labels:

1. processes/apps
2. properties
3. files
4. pseudo-files

Apps are labelled in files called seapp_context, properties are labelled in property_context, and files in file_context. Permissions are given in /.te files.

Of course, writing all permissions and re-flashing the system is a pain, so there is a tool called sepolicy-inject, which allows injecting policies into the running kernel. File contexts can be changed with chcon, but process/app contexts have to be rebuilt.

SELinux has essentially seven types of statements:

1. allow :: permit an operation and print its denial in log
2. neverallow :: prohibit an operation
3. auditallow :: allow an operation but print log
4. dontaudit :: prohibit an operation and do not print its denial in log
5. define :: define a macro with arguments which will expand to something
6. type :: define a new label
7. typeargument :: add property to a label

define is tricky, because it usually allows a lot of permissions, while prohibiting (with neverallow) some actions which can only be allowed via this macro This make a naive policy of “Enable SELinux, look for denials in the log (dmesg), allow as needed” impractical, because while you know that you need to allow something, you do not necessarily know the correct macros for allowing that operation, and allowing the actions directly will be prohibited via neverallow.

For the list of macros, see:

1. system/sepolicy
2. find hardware -name '*.te'

A convenient pattern for debugging SELinux is to find an object you are interested in (say, a camera app), give it a separate label (say mycamera_app) in seapp_contexts, and look at dmesg for the actions denied to it.

As a last topic in this chapter, I want to touch partitions and Android boot process.

In fact, in most operating system manuals this is the first chapter, combined with “how to install an OS”, but Android is a little different in that:

1. most people do not to install Android themselves,
2. installing Android is difficult, the whole chapter 1.4 is dedicated to this,
3. boot process and partition details vary significantly between different manufacturers

In principle we know how a system should boot:

1. BIOS finds a bootloader
2. bootloader finds an OS kernel and root file system
3. bootloader boots the OS kernel, passing the root file system location as a command-line
4. kernel mounts the root filesystem, sometimes additional file systems, and starts the init, which does all the initialisation work

But Android is different. Indeed, there are partitions, and in theory they are on the MMC storage, there is a bootloader and a kernel, but how exactly they are implemented, seems to depend on the manufacturer a lot.

It should be possible to see the full list of partitions on a device … But the method how see them differs from a manufacturer to a manufacturer.

And what is also bad, their amount is significantly greater than on a normal OS.

/dev/block/by-name # ls
ALIGN_TO_128K_1    fsc             oplusdycnvbk    storsec
ALIGN_TO_128K_2    fsg             oplusreserve1   super
DRIVER             hyp_a           oplusreserve2   toolsfv
abl_a              hyp_b           oplusreserve3   tz_a
abl_b              imagefv_a       oplusreserve4   tz_b
aop_a              imagefv_b       oplusreserve5   tzsc
aop_b              init_boot_a     oplusstanvbk_a  uefi_a
aop_config_a       init_boot_b     oplusstanvbk_b  uefi_b
aop_config_b       keymaster_a     param           uefisecapp_a
apdp               keymaster_b     persist         uefisecapp_b
apdp_full          keystore        qmcs            uefivarstore
apdpb              last_parti      qupfw_a         userdata
bluetooth_a        limits          qupfw_b         vbmeta_a
bluetooth_b        limits-cdsp     qweslicstore_a  vbmeta_b
boot_a             logdump         qweslicstore_b  vbmeta_system_a
boot_b             logfs           rawdump         vbmeta_system_b
cdt                mdcompress      recovery_a      vbmeta_vendor_a
connsec            mdm1oemnvbktmp  recovery_b      vbmeta_vendor_b
cpucp_a            mdtp_a          rtice           vendor_boot_a
cpucp_b            mdtp_b          rticmpdata_a    vendor_boot_b
ddr                mdtpsecapp_a    rticmpdata_b    vm-bootsys_a
devcfg_a           mdtpsecapp_b    sda             vm-bootsys_b
devcfg_b           metadata        sdb             vm-data
devinfo            misc            sdc             vm-persist
dinfo              modem_a         sdd             xbl_a
dip                modem_b         sde             xbl_b
dsp_a              modemst1        sdf             xbl_config_a
dsp_b              modemst2        secdata         xbl_config_b
dtbo_a             multiimgoem_a   shrm_a          xbl_ramdump_a
dtbo_b             multiimgoem_b   shrm_b          xbl_ramdump_b
engineering_cdt_a  multiimgqti_a   splash_a        xbl_sc_logs
engineering_cdt_b  multiimgqti_b   splash_b        xbl_sc_test_mode
featenabler_a      ocdt            splash_odm
featenabler_b      oplus_sec_a     spunvm
frp                oplus_sec_b     ssd

The _a/_b partitions are an implementation of a system called “Project Treble”, which was expected to help phone manufacturers deliver updates to their phones faster. The idea was that system, odm, and vendor partitions could be updated independently, and if an update fails, the phone reboots using a dual partition _b if _a failed, and vice-versa.

In theory this would allow booting separate operating system images while keeping the driver partitions odm and vendor intact. I am not sure this actually works anywhere except Google’s reference devices (Pixel).

So, in theory, a Treble-enabled phone has at least two system “root” partitions to boot, but there are actually more. Many devices are able to boot into

1. bootloader
2. fastboot
3. fastbootd
4. edl
5. recovery
6. sideload
7. system

The bootloader, in theory, should be independent from the MMC storage and be bootable regardless of a system installed. Fastboot is a “debugging interface”, usable through a command-line tool “fastboot”, which should allow re-flashing the system or individual partitions. It is often combined with Fastboot into the same interface. Usually it is also not kept in the main MMC.

Recovery is a stripped-down version of Android, using the same kernel and loadable into ram filesystem. It usually has some kind of interface, but its main purpose is to be able to repair Android in case it is broken. Together with recovery, there is also “fastbootD” (note the D). It is a system implementing the fastboot protocol, but running from a booted system, such as recovery. It is used to flash partitions which are not real partitions, but are read-only files (file system images) mounted loopback.

Note that the list of partitions above does not have a system partition. This is because the system partition is not real.

For some reason, Android really likes having different components of the boot process as separate partitions.

For example, on many phones you install Android like this:

fastboot flash boot boot.img
fastboot flash init_boot init_boot.img
fastboot flash vendor_boot vendor_boot.img
fastboot flash dtbo dtbo.img
fastboot flash recovery recovery.img
fastboot reboot sideload
adb sideload lineage*.zip

fastboot reboot sideload might be replaced with fastboot reboot recovery and tapping “install update”.

1. boot is essentially the kernel
2. init_boot is the ramfs with initial userspace for the booted kernel
3. vendor_boot is a similar ramfs with vendor files
4. dtbo is the “device tree” describing the system you are running on
5. recovery is also a ramfs with a the few recovery options and an interface to install the main system

Android is not Windows, and ARM is not x86.

In general, ARM is a much worse of a system, restrictive, repressive, and limiting. While on x86, it is possible to query the system for a lot of things about itself, ARM systems have not ACPI, or BIOS, or EFI. Well, something like an EFI does exist, but is not widespread.

As a result, firmware (like, real firmware, not the Android system) is often only accessible through secret tools sold by the manufacturer for a lot of money.

Sometimes such tools are available to download though, and they are not user-friendly.

These tools are often used by repair shops to restore completely broken devices. The boot mode used by such tools is sometimes called EDL. Some bootloaders allow booting into EDL manually.

1. /system :: system files
2. /system_ext :: emm… more system files?
3. /odm :: brand-related (say OnePlus) things
4. /vendor :: chipset manufacturer (say, Qualcomm) related things
5. /data :: data
   1. /data/data/ :: app data. Usually only one sub-directory there is available for an app to write files, say /data/data/com.termux
   2. /data/data/adb :: directory to place adb-related files
   3. /data/tombstones :: crash minidumps
   4. /data/vendor/camera :: camera intermediate directory and debug dumping directory
6. /apex :: sub-tree to mount specific sets of versions of system libraries, for compatibility reasons
7. various *etc* directories :: configuration files
8. various *lib64* directories :: shared libraries
9. various *bin* directories :: binary executable files

1. Background reading: https://www.kernel.org/doc/Documentation/filesystems/sharedsubtree.txt
2. https://android.stackexchange.com/questions/203951/how-can-i-make-a-symlink-or-equivalent-inside-storage-emulated-0/
3. sdcardfs : https://www.xda-developers.com/diving-into-sdcardfs-how-googles-fuse-replacement-will-reduce-io-overhead/
4. https://android.stackexchange.com/questions/197959/why-partition-gets-unmounted-automatically-after-some-time/200449#200449
5. https://android.stackexchange.com/questions/214288/how-to-stop-apps-writing-to-android-folder-on-the-sd-card/
6. https://android.stackexchange.com/questions/217741/how-to-bind-mount-a-folder-inside-sdcard-with-correct-permissions/217936#217936
7. https://developer.android.com/training/data-storage#scoped-storage

Android has suffered a series of transformations from one build system to another, and the current one is sort of a mess. I have not personally met Bazel, even though it is believed to be somewhere inside there too, but there are Android.mk and Android.pb to be aware of.

Both are a variation of classical UNIX Make, you define targets, connect them with dependencies, and the system traverses the dependency tree and builds the resulting object. Android.mk is essentially a Makefile, but it is parsed by ckati into a Ninja script rather than executing the statements directly. Android.bp is also parsed into Ninja, but by a different tool.

You probably will not have to tweak Ninja scripts at all, but Make and Android.bp files are interesting, and you will probably also not call ckati or soong (the proper name of the build system) directly.

Good for us, Android.bp kinds of targets are well documented:

1. Syntax :: https://source.android.com/docs/setup/reference/androidbp
2. Modules :: https://ci.android.com/builds/submitted/13288697/linux/latest/view/soong_build.html

Android.mk is not so well documented, especially since it is expected to be phased out eventually.

Nevertheless, Make files are still very important, as they are used for device definitions, and you will have to write our own one (or adapt someone else’s) in order to successfully port Lineage.

1. device :: mostly things related to a particular device, say, Samsung Galaxy S10
   1. device/oneplus/sm8550-common :: source files related to all devices by OnePlus built on Qualcomm sm8550
   2. device/oneplus/xigua :: source files related to a particular device. Will most likely inherit from a common-phone.mk profile
2. vendor :: mostly related to things extracted from original firmware
   1. /vendor/oneplus/sm8550-common :: binaries from original firmware, which are chipset-related
   2. /vendor/oneplus/xigua :: ditto, concrete device related
   3. /vendor/lineage/... :: despite being in vendor, not really binaries, actually I don’t know what it is, it makes little sense
3. kernel :: kernel-related directories
4. frameworks :: system libraries available to Java apps
5. packages :: pre-built apps
6. hardware :: em… some more drivers, but not chipset-related? I am confused
7. external :: things used in Android transparently, for example, Linux libraries, such as libxml2
8. system :: system libraries and services unavailable to Java apps

Let us have a look at the files in a typical device module: device/oneplus/xigua/.

# ls
Android.bp
AndroidProducts.mk
BoardConfig.mk
board-info.txt
device.mk
extract-files.py
lineage.dependencies
lineage_xigua.mk
odm.prop
overlay
overlay-lineage
proprietary-files.txt
proprietary-firmware.txt
setup-makefiles.py
system_ext.prop
vendor.prop

As far as I understand, their meaning is the following:

1. AndroiProducts.mk :: defines a “device” for the purposes of the Android build system.
2. lineage_xigua.mk :: defines a module, which actually defines the device
3. BoardConfig.mk :: defines … well, I don’t know what it defines and what is the need for it to be separate from lineage_xigua, but it includes Makefile from vendor/oneplus/xigua, which is the module for pre-built drivers, so it is obviously very important.
4. device.mk :: seemingly also important, because defines a lot of device properties on part with lineage_xigua.mk and BoardConfig.mk. I have no idea why there are three files.
5. lineage.dependencies :: tells the system that your device is actually designed on the basis of a System-on-Chip which is used for other devices as well and some drivers can be shared
6. board-info.txt :: I have no idea what it is, but it includes the code name of my SoC
7. extract-files.py :: extracts binaries from an unpacked original firmware and post-processes them.
8. setup-makefiles.py :: just calls extract-files.py
9. proprietary-files.txt :: a file from which extract-files.py takes information on which files to extract
10. proprietary-firmware.txt :: I have no idea what it is, but it is probably somehow connected to the partitions on the device which are not Open Source, such as modem firmware
11. {overlay,overlay-lineage} :: no idea what they are for
12. Android.bp :: empty

If you want to port Android to a device which is to an extent similar to some other device already supported, chances are that you only need to find the true board name to inherit in BoardConfig.mk, and adjust the parameters in the files to match your device.

Extracting the correct binary files is very important for making the device run.

In order to do that, it is important to understand which files serve which purpose and correspond to which packages. As has already been said, there is no way to identify that using normal Android means, although I suspect that hacking something on the basis of xattrs should be possible.

Nevertheless, if you look at proprietary-files.txt, and extract-files.py, you will see that those files do more than just copy files into a temporary directory.

There are a lot of examples on how to rewrite files documented in tools/extract-utils/templates/single-device/extract-files.py, but I will note the few most important here.

1. .regex_replace(from, to) replaces a string
2. .binary_regex_replace(bytes.fromhex('cafebabe'),bytes.fromhex('deadbeef')) replaces a binary sequence
3. .replace_needed('libfoo.so', 'libbar.so') replaces a dependency
4. .fix_soname() fixes an SONAME if the .so metadata

The last two are very important for the case when you need to link an app to an original version of a library, and copy it into /odm/lib64/libfoo_vendor.so.

1. https://docs.qualcomm.com/doc/80-70017-3/topic/features.html
2. https://wiki.postmarketos.org/wiki/Device_Tree_(dtb)

I don’t know what to write here, I am ignorant, but I never had to write a device tree (dtb), or overlay it with some customisations with a device tree overlay (dtbo).

Look at the examples of real device trees in kernel/<manufacturer>/<chip>-devicetrees

You can try decompiling your own phone device trees:

#!/bin/bash
mkdtboimg dump dtbo.img -b mydtb
for i in mydtb.* ; do
  dtc -I dtb -O dts "$i" -o "${i/mydtb/mydts}"
done

But the result will be megabytes in size.

1. https://github.com/Heydarchi/AIDL-HAL-Service

Missing place.

Presumably you need to write an “*.aidl” file, add it to Soong, then write the server and the client for the service and include the headers generated by Soong. But that is as much as I know.

But there seems to be at least one working example: https://github.com/Heydarchi/AIDL-HAL-Service

To begin is always the hardest thing for a programmer. You change one line in a huge project, and the crystal edifice shatters into myriads of tiny shards. At best itn starts to fail building. At worse, it builds, but begins to misbehave in a subtle way. At worst, it works, seemingly, as intended, but either leaks memory or becomes horribly insecure.

This is why having an introductory manual is so important.

1. https://lineageos.org/engineering/HowTo-Debugging/

Of course, when you are writing code, everything can go wrong. You need to debug it.

Basically, in software there are just two ways of debugging: logging and debuggers.

The following are the main log sources in Android:

1. dmesg
2. logcat
3. /data/tombstones
4. /tmp/recovery.log (in recovery)

Quoting Lineage’s guide:

For apps crashing, logcat is your best choice. For native code, look at the “tombstones”, at least they are going to have stack traces.

1. https://github.com/NationalSecurityAgency/ghidra/issues/6386#issuecomment-2455257933
2. Debugging Android aarch64 with Ghidra :: https://blog.nviso.eu/2024/01/15/deobfuscating-android-arm64-strings-with-ghidra-emulating-patching-and-automating/

Okay, I do not really understand this, but there are some compatibility issues between different versions of Android, SDK and NDK. This is so convoluted that I do not know how to understand it, so you have to search.

When you want to port an app for LineageOS, you usually only have two options: (1) modify the app, (2) modify the Operating System.

In both of those cases you want to make your development iterations as short as possible.

I already mentioned this, but it is never hurts to repeat:

1. apps can be signed with the platform key and copied
2. native libraries can be copied to /data/local/tmp and mounted over
3. selinux permissions can be injected using sepolicy-inject

First of all, you need to make sure that working with code is convenient, and stupid useless security theatre is not getting in the way.

So, the first thing you need to do is turn off ADB authentication to avoid tapping stupid permission windows on each re-flash.

go to vendor/lineage and apply the patch.

diff --git a/config/common.mk b/config/common.mk
index abc2b9ec..69aff098 100644
--- a/config/common.mk
+++ b/config/common.mk
@@ -33,7 +33,7 @@ ifdef WITH_ADB_INSECURE
 PRODUCT_SYSTEM_DEFAULT_PROPERTIES += ro.adb.secure=0
 else
 # Enable ADB authentication
-PRODUCT_SYSTEM_DEFAULT_PROPERTIES += ro.adb.secure=1
+PRODUCT_SYSTEM_DEFAULT_PROPERTIES += ro.adb.secure=0
 endif

 # Disable extra StrictMode features on all non-engineering builds

Also, I added the following lines to my device/oneplus/xigua/device.mk:

PRODUCT_PROPERTY_OVERRIDES += \
    ro.debuggable=1 \
    persist.service.adb.enable=1 \
    persist.service.debuggable=1 \
    persist.sys.usb.config=adb \
    ro.adb.secure=0

Again, this is making a lot of fiddling with the settings after each system reboot unnecessary.

Some debugging does not require complicated techniques.

Very often you can see what an app needs right into the adb logcat output.

In order to make your life simpler, you can edit the Android log-printing functions to produce more information.

For example, when I was debugging “Vendor Tags”, I added the following line to the code:

 status_t VendorTagDescriptor::lookupTag(const String8& name, const String8& section,
                                         /*out*/ uint32_t* tag) const {
+  ALOGE("my-own:%s: Section: '%s', name: '%s'.", __FUNCTION__, section.c_str(), name.c_str());

This line appears in “ADB logcat” and is easy to filter. ALOGE is essentially Java’s Log.e(TAG, message);

Install your app, run it, look at logcat, dmesg and /data/tombstones.

Often it will tell you what is missing.

Another “naive”, but useful and practical tools are grep (or ripgrep) and strings. Those let you search binary files for the presence of content.

Look for something in the log, grep for this string, find out which binary contains it.

A lot of problems with an app running on Lineage are coming from Lineage just missing some libraries that the manufacturer is shipping in his firmware.

Firstly, you can unpack the apk with apktool and examine its shipped JNI and native libraries:

look at the lib directory and see which libraries the apk is carrying.

You can examine those libraries with objdump, readelf, and ldd. Unfortunately, Android’s ldd is unreliable, and it is listing all possible dependencies, transitively.

But readelf -d libfoo.so | grep NEEDED will give you a list of dependencies.

I have written a small script in Scheme to plot those dependencies as a tree:

#!/usr/bin/chibi-scheme -q
(import (chibi shell))
(import (chibi filesystem))
(import (chibi show))
(import (srfi 69))
(import (srfi 113))
(import (only (scheme base) when))
(import (chibi pathname))
(import (only (srfi 1) zip filter))
(import (srfi 27))
(import (srfi 95))
(define to-string
  (lambda (num)
    (substring (show #f (written num)) 0 4)))
(define graph-file "dep-graph.dot")
(define stopfiles (set equal?
                  "/apex/com.android.runtime/lib64/bionic/libdl.so"
                  "/apex/com.android.runtime/lib64/bionic/libc.so"
                  "/apex/com.android.runtime/lib64/bionic/libm.so"
                  "/system/lib64/libc++.so"
                  "/system/lib64/liblog.so"
                  "/system/lib64/libz.so"))
(define processed (set equal?))
(define processed-vertices (set equal?))
(define (get-dep-names file)
  (map (lambda (x) (substring x 1 (- (string-length x) 1)))
   (shell->string-list ("adb" "-s" a740e4ea shell readelf -d ,file) (grep "Shared") (awk "{print $5;}") )))

(define (get-dep-paths filepath)
  (let* ((extractor (lambda (x) ;;:/apex/com.android.i18n/lib64  /vendor/lib64/hw:/vendor/lib64
                      (shell->string-list ("adb" "-s" a740e4ea shell ,(string-append "LD_LIBRARY_PATH='/apex/com.android.i18n/lib64" "'" " ldd  " filepath)) (grep "-v" "vdso") (awk ,(string-append "{print $" x "}")))))
         (deps (extractor "1"))
         (paths (extractor "3")))
    (zip deps paths)))

(define systemp (lambda (target)
             (equal? (substring target 0 (string-length "/system"))
                     "/system")))

(define max-depth 7)

(define walk-tree
  (lambda (path depth)
    (show #t "Processing " path " at depth " depth "\n")
    (set-adjoin! processed path)
    (if (< depth max-depth)
       (let* ((deps (get-dep-names path))
              #;(_ (show #t "deps:" (displayed deps) "\n"))
              (paths (get-dep-paths path))
              #;(_ (show #t "paths:" (displayed paths) "\n"))
              (pairs (filter (lambda (t) (not (equal? #f t)))
                             (map (lambda (dep) (assoc dep paths)) deps)))
              #;(_ (show #t "pairs:" (displayed pairs) "\n"))
              (targets (filter
                        (lambda (t)
                          (not (set-contains? stopfiles t)))
                        (map cadr pairs)))
              #;(_ (show #t "targets:" (displayed targets) "\n"))
              )
         #;(map (lambda (x) (display x) (newline)) pairs)
         (map (lambda (target)
                (let ()
                  (if (not (set-contains? processed-vertices target))
                     (begin
                       (shell (echo ,(string-append "\"" target "\" "
                                                    (if (systemp target)
                                                       "[color=red]"
                                                       "[color=green]"))) (>> ,graph-file))
                       (set-adjoin! processed-vertices target)))
                  (shell (echo ,(string-append "\"" path "\" -> \"" target "\"
                               [color=\""
                               (to-string (random-real))
                               "+"
                               (to-string (random-real))
                               "+"
                               "0.5\"]")) (>> ,graph-file))))
              targets)
         (map (lambda (target)
                (if (and (not (set-contains? processed target))
                      (not (systemp target)))
                   (walk-tree target (+ 1 depth))
                   'already-processed))
              targets))
       (show #t "depth of " depth " reached\n"))))
;; (define my-deps (get-dep-names "/vendor/lib64/libcamerapostproc.so"))
;; (define my-paths (get-dep-paths "/vendor/lib64/libcamerapostproc.so"))
;; (map (lambda (x) (display x) (newline)) my-deps)
;; (map (lambda (x) (display x) (newline)) my-paths)
(define main1
  (lambda ()
    (shell (echo "strict digraph {
ranksep=\"1.2 equally\"" ) (> ,graph-file))
    #;(walk-tree "/vendor/lib64/libcamerapostproc.so" 0)
    #;(walk-tree "/vendor/lib64/vendor.qti.hardware.camera.postproc@1.0-service-impl.so" 0)
    #;(walk-tree "/vendor/bin/hw/vendor.qti.camera.provider-service_64" 0)
    (walk-tree "/vendor/lib64/hw/com.qti.chi.override.so" 0)
    (shell (echo "}" ) (>> ,graph-file))
    ))
(main1)
(shell (dot -Tpdf "dep-graph.dot") (> "dep-graph.pdf"))
(define main2
  (lambda ()
    (let* ((get-rel (lambda (p) (list->set equal? (filter systemp (map cadr (get-dep-paths p))))))
           (deps-camera (set-union (get-rel "/vendor/lib64/libcamerapostproc.so")
                                   (get-rel "/vendor/lib64/vendor.qti.hardware.camera.postproc@1.0-service-impl.so")
                                   (get-rel "/vendor/bin/hw/vendor.qti.camera.provider-service_64")
                                   (get-rel "/vendor/lib64/hw/com.qti.chi.override.so")))
           (deps-libAlgoProcess    (set-union
                                    (get-rel "/odm/lib64/libAlgoProcess.so")
                                    (get-rel "/odm/lib64/libAlgoInterface.so")))
           (difference (set-difference deps-camera deps-libAlgoProcess))
           (shower (lambda (e) (show #t (written e) "\n")))
           (mysort (lambda (s) (sort (set->list s) string<?))))
      (show #t "deps-camera:\n")
      (map shower (mysort deps-camera))
      (show #t "\ndeps-libAlgoProcess:\n")
      (map shower (mysort deps-libAlgoProcess))
      (show #t "\ndifference:\n")
      (map shower (mysort difference)))))
(main2)

Just look at how beautiful it is. It is generating a graphviz tree, and you can compile it into a searchable pdf.

When you find missing dependencies, just add them to proprietary-files.txt.

When simple dependency and log-based methods stop working, it is time to do dig a little bit deeper.

The previous chapter already mentioned decompiling an apk with apktool. An extremely nice property of apktool compared to traditional decompilers is that it is possible not just to decompile an apk, but to also re-compile it, and the resulting apk will be as good as the old one, except, maybe, losing the platform key, but since we are building our own Lineage, we have the key.

Some adaptations might require you to just adjust the file AndroidManifest.xml and recompile the apk, but sometimes you have to actually edit the code in order to make the app compatible with your platform.

Here comes Smali, the Java assembly.

This is a decent howto on Smali: https://sallam.gitbook.io/sec-88/android-appsec/smali/smali-cheat-sheet See the chapter 1.6.7 to see how actual smali patches work.

apktool will decompile the code successfully, but trawling through megabytes of Smali is tedious, so we have JADX at our disposal.

JADX is a decent decompiler. You can rename variables, add comments, and search nicely.

It is not almighty, and it cannot compile the apps back into apk, so when you find a place you want to modify, you have to edit the original smali code produced by apktool.

Luckily for us, Dalvik does limit the amount of registers per function, so we can add as many as we want, and call Log.e() whenever we need.

const-string v4, "mine-parseMenuSetting"
const-string v5, "17"
invoke-static {v4, v5}, Landroid/util/Log;->e(Ljava/lang/String;Ljava/lang/String;)I
invoke-interface {v0, v2}, Ljava/util/List;->add(Ljava/lang/Object;)Z

As I mentioned, JADX can theoretically connect to a running app and become a debugger, but I failed to make work.

1. ELF Format specification :: https://refspecs.linuxfoundation.org/elf/elf.pdf
2. Learning Linux Binary Analysis by Ryan “Elfmaster” O’Neill :: https://github.com/PacktPublishing/Learning-Linux-Binary-Analysis
3. The Ghidra Book The Definitive Guide by Chris Eagle and Kara Nance :: https://nostarch.com/GhidraBook
4. https://blog.k3170makan.com/2018/09/introduction-to-elf-format-elf-header.html
5. https://en.wikipedia.org/wiki/Executable_and_Linkable_Format
6. https://thinkingeek.com/categories/aarch64/

If Java decompiling does not work, and you need to use heavier artillery, have a look at Ghidra.

1. https://www.youtube.com/watch?v=RoNppaG1re4
2. https://github.com/iddoeldor/frida-snippets?tab=readme-ov-file
3. https://cilynx.com/vulnerabilities/exploring-native-functions-on-android-and-runtime-analyses-using-jadx-ghidra-and-frida/1565/
4. https://github.com/Hamz-a/frida-android-libbinder

Ghidra is programmable in Java and Python, in theory you can run Ghidra scripts in Ghidra debugger, but it is a bit inconvenient, because Ghidra works by pulling memory to your machine via lldb, which sometimes happens to be slow.

Frida goes into the opposite direction. It embeds a JavaScript interpreter into the target process and allows you to run arbitrary JS code at trace points. You can use this, for example, to trace important functions calls.

Frida is instantaneously fast.

I did not use Frida a lot (just traced a few function calls), but generally a programmable debugger-tracer is a very mighty tool.

For example, you can monitor Binder transactions. https://github.com/iddoeldor/frida-snippets?tab=readme-ov-file#binder-transactions

1. https://www.jemarch.net/poke-elf.html

This section is not developed, because I did not do it.

So, the idea is that one might want to write a C function, compile it into an .o file and all-link it to an .so binary. The goal is to be able to call that function from within some other functions, for example inserting a call with ddisasm.

I suspect that GNU Poke-ELF (https://www.jemarch.net/poke-elf.html) should be able to do that, but I am not sure.

1. Working features
   1. Normal photos
   2. 50 MPx photos
   3. “Livephoto”
   4. 1080@60Hz video
   5. 4k@30Hz video
   6. “Pro” photos
   7. Panorama
   8. SELinux
2. Partially working features
   1. 4k@60Hz video
   2. slomo videos
   3. RAW images
   4. RAW+ images
3. Broken features
   1. Portrait/Bokeh
   2. AI ID scanning
   3. 10bit+HEIC images

I started, of course, standing on the shoulders of giants, which in my case are

1. https://gitlab.com/pjgowtham/proprietary_vendor_oplus_camera
2. https://gitlab.com/ThankYouMario/proprietary_vendor_oplus_camera

I did not really care whether I am getting an “OnePlus Camera” (the international build), or the “OPlus Camera” (the Chinese build), as I only needed the offline features, and the Chinese version turned out to be easier to port.

A few points deserve attention.

1. Binary extraction
2. Init script
3. Mounting the files to override selinux context
4. Main Camera project name

Extracting blobs is documented here: https://wiki.lineageos.org/extracting_blobs_from_zips_manually

The important thing is that at some point LineageOS has moved from extract-blobs.sh to extract-blobs.py, which seems to be more functional, but is a little weird to use.

I was extracting blobs from an unpacking of the most recent version of the OnePlus original operating system, done as usually done with most Lineage builds, with a single caveat.

My system used a single “APEX” package to provide alternative versions of the binary libraries to (at least some of) camera .so libraries as dependencies.

So in addition to unpacking the first layer of firmware into separate directories (system, system_ext, vendor, odm), I had to unpack that APEX file and add an additional directory to the unpack.

It is very important that some binaries are “pinned”, that is that they are not expected to be extracted and are provided with the camera module itself. In my project those are the files of the camera and gallery themselves, because they are actually re-compiled from Smali sources.

Init script, mount, and camera project name come from my predecessors, and I did not try removing them, and I do not know if they still serve any purpose.

It is surprising how far one can go without any specific debugging tools, just by reading logs and following the intuition.

Essentially I kept reading adb -s <devid> logcat and adb -s <devid> dmesg for errors, and tried fixing those which were appearing.

Most of the problems were of these kinds:

1. Missing Java classes :: those I implemented in the oplus-fwk
2. Missing start-time .so libraries :: those I copied from the original firmware.
3. Missing dlopen .so libraries :: those were also usually reported in the log, except when they were not.

At some point I used a script written by myself in Scheme in order to plot the dependency tree of the camera .so files.

This is just a fragment, the full graph is a PDF, and it kinda helped me to track which dependencies are already installed and which are not related to the camera.

At some point I found that the requirements for the apk files changed with between Android 13, 14, 15 and 16.

Those changes, luckily, were primarily those which can be accommodated with one of the two methods:

1. Adding some statements into the AndroidManifest.xml to declare using new permissions.
2. Adding some statements to the /system_ext/etc/permissions/*xml and /system/etc/sysconfig/*xml /system_ext/etc/default-permissions/*xml to grant those permissions to the app without asking. See the files in the camera project repo.

But if changing the xml files in on the system is easy, changing the xml file in the apk requires recompiling the apk.

Here is my script for it:

cd ~/Lineage-Camera
(
  set -x
  cd OplusCamera-15 && \
  rm -rf ./dist/OplusCamera-15.apk  && \
  apktool b && \
  java -jar ~/Lineage-Build/out/host/linux-x86/framework/apksigner.jar sign --in ./dist/OplusCamera-15.apk -out ./dist/OplusCamera-15.apk.signed --cert ~/Lineage-Build/build/make/target/product/security/platform.x509.pem --key ~/Lineage-Build/build/make/target/product/security/platform.pk8 --v3-signing-enabled false && \
  mv ./dist/OplusCamera-15.apk.signed ./dist/OplusCamera-15.apk && \
  mysha=$(sha1sum ./dist/OplusCamera-15.apk | cut -d ' ' -f 1) &&
  printf 'Opluscam sha1sum=%s\n' "$mysha" &&
  sed -i "/my_product\/app\/OplusCamera\/OplusCamera\.apk/s/|.*$/|$mysha/"   ~/Lineage-Build/vendor/oplus/camera/proprietary-files.txt &&
  cp ./dist/OplusCamera-15.apk ~/Lineage-Build/vendor/oplus/camera/proprietary/system_ext/priv-app/OplusCamera/OplusCamera.apk || exit 1
) && \
(
  cd OppoGallery2 && \
  rm -rf ./dist/OppoGallery2.apk && \
  apktool b && \
  jarsigner -storepass "android" -verbose -sigalg SHA1withRSA -digestalg SHA1 -keystore ~/.android/debug.keystore ./dist/OppoGallery2.apk androiddebugkey && \
  mysha=$(sha1sum ./dist/OppoGallery2.apk | cut -d ' ' -f 1) &&
  printf 'Oplusgallery sha1sum=%s\n' "$mysha" &&
  sed -i "/my_stock\/priv-app\/OppoGallery2\/OppoGallery2\.apk/s/|.*$/|$mysha/"   ~/Lineage-Build/vendor/oplus/camera/proprietary-files.txt &&
  cp ./dist/OppoGallery2.apk ~/Lineage-Build/vendor/oplus/camera/proprietary/system_ext/priv-app/OppoGallery2/OppoGallery2.apk
) || exit 2

if ! diff -u OplusCamera-15/dist/OplusCamera-15.apk ~/Lineage-Build/vendor/oplus/camera/proprietary/system_ext/priv-app/OplusCamera/OplusCamera.apk  ; then
  printf 'Cameras do not match!\n'
  exit 1
fi

if ! diff -u OppoGallery2/dist/OppoGallery2.apk ~/Lineage-Build/vendor/oplus/camera/proprietary/system_ext/priv-app/OppoGallery2/OppoGallery2.apk ; then
  printf 'Galleries do not match!\n'
  exit 1
fi

Note that I am actually signing the camera apk with the platform key. This is superfluous when building Android once, but if you want to avoid re-flashing the image each time you rebuild the apk, you can sign the apk with the “platform key” and just copy it to the target directory.

1. https://xdaforums.com/t/edge30ultra-moto30xpro-camera-research.4492159/page-3

There was a third, “bonus” file to edit, /etc/camera/camxoverridesettings.txt, but it only required two edits.

raiserecoverysigabrt=0
disableFDHWProcessing=1

The first one does not crash the process when ASAN (https://clang.llvm.org/docs/AddressSanitizer.html) failures occur, and they will occur because, come on, the project is not built from one source.

The second disables hardware face detection (FD is Face Detection), because there were some crashes connected to it? (I guess, I actually forgot).

See the link to xda-forums for more details about using camxoverridesettings.txt. In particular, by setting enableFeature2Dump=1 it is possible to get not just DNG RAW, but full-sized Bayer matrix dumps into /data/vendor/camera.

The full list of parameters is in the project repo, but I don’t know what they mean.

I had to stub a lot of oplus-framework.jar functions, mostly blindly making them return false, null, throw an exception, or do nothing.

Perhaps this is the part of my work most likely to be accepted to LineageOS’ upstream. Not all of the code is written by me, a lot of it is written by my predecessor, sx75 from crdroid.

In total, this is:

git log HEAD~2..HEAD --patch -- oplus-fwk/ | wc -l
16099

in

git log HEAD~2..HEAD --stat -- oplus-fwk | grep -F '|'  | wc -l
135

I had to add the following exceptions to the declared “allowed framework classes” into build/soong/scripts/check_boot_jars/package_allowed_list.txt

diff --git a/scripts/check_boot_jars/package_allowed_list.txt b/scripts/check_boot_jars/package_allowed_list.txt
index 983650039..7c5ed0ca2 100644
--- a/scripts/check_boot_jars/package_allowed_list.txt
+++ b/scripts/check_boot_jars/package_allowed_list.txt
@@ -280,6 +280,12 @@ com\.nvidia\..*
 # OPLUS adds
 com\.oplus\..*
+oplus\..*
+com.coloros\..*
+vendor.oplus\..*
+com.heytap\..*
+net.oneplus\..*
+com.color\..*
 # QC adds
 com.qualcomm.qti

Even though I have introduced Smali in 1.5.5, I actually did not need to write much Smali to solve concrete issues. Essentially, I added just two patches: one fixed a race condition deadlock, another one removed a hanging check.

The deadlock is fixed like this, and I am afraid to touch it, because it is just making the correct thread “win” the race and is not actually solving the bug.

diff --git a/smali/com/oplus/camera/common/config/CameraSettingsConfig.smali b/smali/com/oplus/camera/common/config/CameraSettingsConfig.smali
index e80eaccf..befe96b8 100644
--- a/smali/com/oplus/camera/common/config/CameraSettingsConfig.smali
+++ b/smali/com/oplus/camera/common/config/CameraSettingsConfig.smali
@@ -1383,7 +1383,7 @@
 .end method
 .method public static declared-synchronized initialize(Landroid/content/Context;)V
-    .locals 3
+    .locals 6
     .line 1
     const-class p0, Lcom/oplus/camera/common/config/CameraSettingsConfig;
@@ -1472,6 +1472,9 @@
     .line 45
     sput-boolean v0, Lcom/oplus/camera/common/config/CameraSettingsConfig;->sbInit:Z
+    const-string v4, "mine-initialize"
+    const-string v5, "after sbInit=true;"
+    invoke-static {v4, v5}, Landroid/util/Log;->e(Ljava/lang/String;Ljava/lang/String;)I
     .line 46
     .line 47
@@ -1487,23 +1490,23 @@
     .line 50
     :cond_0
     :goto_0
-    const-string v0, "CameraSettingsConfig"
+#    const-string v0, "CameraSettingsConfig"
     .line 51
     .line 52
-    new-instance v1, Lcom/oplus/camera/common/config/a;
+#    new-instance v1, Lcom/oplus/camera/common/config/a;
     .line 53
     .line 54
-    const/4 v2, 0x0
+#    const/4 v2, 0x0
     .line 55
-    invoke-direct {v1, v2}, Lcom/oplus/camera/common/config/a;-><init>(I)V
+#    invoke-direct {v1, v2}, Lcom/oplus/camera/common/config/a;-><init>(I)V
     .line 56
     .line 57
     .line 58
-    invoke-static {v0, v1}, Ll7/a;->r(Ljava/lang/String;Ljava/util/function/Supplier;)V
+#    invoke-static {v0, v1}, Ll7/a;->r(Ljava/lang/String;Ljava/util/function/Supplier;)V
     :try_end_0
     .catchall {:try_start_0 .. :try_end_0} :catchall_0
@@ -1513,6 +1516,9 @@
     monitor-exit p0
     .line 62
+    const-string v4, "mine-initialize"
+    const-string v5, "after before return-void;"
+    invoke-static {v4, v5}, Landroid/util/Log;->e(Ljava/lang/String;Ljava/lang/String;)I
     return-void
     .line 63

The second is just as simple:

--- a/smali/com/oplus/camera/common/config/CameraSettingsConfig.smali
+++ b/smali/com/oplus/camera/common/config/CameraSettingsConfig.smali
@@ -3683,32 +3683,37 @@
     .line 157
     .line 158
     .line 159
-    invoke-static {}, Lv6/k2;->c()Z
+#    invoke-static {}, Lv6/k2;->c()Z
     .line 160
     .line 161
     .line 162
-    move-result v0
+#    move-result v0
     .line 163
-    if-eqz v0, :cond_5
+#    if-eqz v0, :cond_5
     .line 164
     .line 165
-    iget-object v0, p0, Lcom/oplus/camera/common/config/CameraSettingsConfig;->mMenuSettingList:Ljava/util/List;
+#    iget-object v0, p0, Lcom/oplus/camera/common/config/CameraSettingsConfig;->mMenuSettingList:Ljava/util/List;
     .line 166
     .line 167
-    const-string v2, "pref_spruce_function_key"
+#    const-string v2, "pref_spruce_function_key"
     .line 168
     .line 169
-    invoke-interface {v0, v2}, Ljava/util/List;->add(Ljava/lang/Object;)Z
     .line 170
     .line 171
     .line 172
-    :cond_5
+#    :cond_5
     invoke-static {}, Lcom/oplus/camera/configure/CameraConfig;->C()Z
     .line 173

That is just it. I just removed some hanging function calls.

This issues is not entirely solved, but at least is somehow progressed. When I was trying to find out why the “240Hz slomo” mode is not opening in the camera, I encountered a message about not enough buffers being allocated for a stream.

This is related to an Android feature called “HAL3 buffer management API.” (https://source.android.com/docs/core/camera/buffer-management-api)

requestStreamBuffers is a function which is sometimes called by the “Camera HAL” client to receive … something. I guess, some buffer pointers, for stream data.

The log was complaining about the code requiring too many buffers, more than had been allocated in advance. I added a hack to allocate more buffers, but the issue is probably that BBK ships a patched version of their cameraserver and its dependencies, which somehow manage buffers differently. It was also complaining about wrong frame numbers… whatever that is.

diff --git a/services/camera/libcameraservice/device3/Camera3OutputUtils.cpp b/services/camera/libcameraservice/device3/Camera3OutputUtils.cpp
index 7a53847765..ddcf59c096 100644
--- a/services/camera/libcameraservice/device3/Camera3OutputUtils.cpp
+++ b/services/camera/libcameraservice/device3/Camera3OutputUtils.cpp
@@ -654,10 +654,15 @@ void processCaptureResult(CaptureOutputStates& states, const camera_capture_resu
	 std::lock_guard<std::mutex> l(states.inflightLock);
	 ssize_t idx = states.inflightMap.indexOfKey(frameNumber);
	 if (idx == NAME_NOT_FOUND) {
-            SET_ERR("Unknown frame number for capture result: %d",
-                    frameNumber);
+          if (states.inflightMap.size() != 0) {
+            idx = states.inflightMap.size() - 1;
+            goto mine_out;
+          }
+          SET_ERR("Unknown frame number for capture result: %d",
+                  frameNumber);
	     return;
	 }
+    mine_out:
	 InFlightRequest &request = states.inflightMap.editValueAt(idx);
	 ALOGVV("%s: got InFlightRequest requestId = %" PRId32
		 ", frameNumber = %" PRId64 ", burstId = %" PRId32
diff --git a/services/camera/libcameraservice/device3/Camera3Stream.cpp b/services/camera/libcameraservice/device3/Camera3Stream.cpp
index ae76e603bd..0a0360e306 100644
--- a/services/camera/libcameraservice/device3/Camera3Stream.cpp
+++ b/services/camera/libcameraservice/device3/Camera3Stream.cpp
@@ -382,7 +382,9 @@ status_t Camera3Stream::finishConfiguration(/*out*/bool* streamReconfigured) {
	 std::string name = std::string("Stream ") + std::to_string(mId);
	 mStatusId = statusTracker->addComponent(name);
     }
-
+    if (camera_stream::max_buffers >=6 || camera_stream::max_buffers <= 24 ) {
+      camera_stream::max_buffers = 25;
+    }
     // Check if the stream configuration is unchanged, and skip reallocation if
     // so.
     if (mState == STATE_IN_RECONFIG &&

I suspect it should increase memory usage significantly.

Somehow this made the “slomo” work, even though in some kind of crippled way. It is “slow” (whatever that means for a slo-mo), and crashes in the dark. I guess this issues is still awaiting its champion.

Is suspect it is somehow related to “stream metadata”, the same issue as in 1.6.11.

When the app began starting successfully, I discovered that it crashes in the native code when making photos.

Native code means that disassembling and binary debugging now became mandatory.

The first steps in debugging the camera were a detour, but I still found some useful stuff.

That is, I found some system properties which enable debugging logging for the Oplus Camera, and some for the Qualcomm backend:

ro.control_privapp_permissions=log \
persist.sys.state.is.otest.running=1 \
vendor.oplus.apsSN.algo.enable=0 \
ro.control_privapp_permissions=log \
sys.gcsupression.optimize.enable=0 \
persist.camera.aps.core.warningMask=0xffffffffffffffffffffffffffffffff \
persist.camera.aps.core.infoMask=0xffffffffffffffffffffffffffffffff \
persist.camera.aps.core.debugMask=0xffffffffffffffffffffffffffffffff \
oplus.autotest.camera.debug.forcelog=true \
persist.sys.camera.lao.enable=true \
persist.oplus.aps.trace=true \
com.oplus.camera.pw.debug.capture.flow=true \
ro.build.release_type=false \
ro.build.version.ota=PRE \
ro.version.confidential=false \
persist.camera.rotate.dump=true \
persist.camera.cfr.debug=1 \
vendor.aps.logic.snapshot.metadata.debug=1\
persist.sys.assert.panic=true  \
persist.camera.assert.panic=true \
persist.vendor.camera.oplus.enableLogging=true \
persist.sys.dump.aps=1 \
persist.vendor.camera.jpeghwencode=1

Not all of them enable debugging, but the names are quite self-explanatory.

The “dumps” usually go to /data/vendor/camera/, the logs go to logcat. There are some logging parameters in camxoverridesettings.txt, but I failed at using them, too difficult.

As I said, some parameters are actually misleading and led to a detour. In particular, persist.vendor.camera.jpeghwencode=1 does make the post-processing flow go into a different direction, but it turned out to be harder to debug than the “hardware” one.

At the end of the day, the camera supports hardware post-processing. (The name of the parameter is misleading, it is not just activating hardware JPEG.)

1. https://developer.android.com/ndk/reference/group/a-hardware-buffer

One of the crashes that I had to debug for the camera to work was a crash related to a wrong version of AHardwareBuffer_lock (https://developer.android.com/ndk/reference/group/a-hardware-buffer#group___a_hardware_buffer_1gaf10c050f1ddfe2a074ab0f80660b2925)

I found that a piece of code was crashing the program by accessing a non-allocated memory, as so-called “segmentation failure”.

Of course, I had had many segmentation faults before, but never studied in depth where they may be coming from. It turns out that program memory space is fixed for a program when it is loaded into memory and how in-process addressing works in modern operating systems.

To keep the long story short, the code was crashing in an completely innocuous function, which was just manipulating some bits, i.e. no complicated system calls, stack unwinding, or any other magic.

I started investigating why it crashes at accessing some memory, and found out that neither my debugger (lldb), nor Ghidra debugger can access that memory and it is shown as “unmapped”. (Note that in Linux, “unmapped” my mean both “not mapped to a different process”, and “not allocated even in the current process space”.)

Neither the Ghidra Book, nor “Learning Linux Binary Analysis” helped me with solving this mystery, but at some point I encountered a StackOverflow question asked by a programmer who could not read memory-mapped region with his debugger, and I guessed that this might be my case.

To this day I do not know whether this memory is from Android Binder or from just a simple mmap, but it did not matter for my case.

I started to look around that code, and found that the mapped memory region was produced by a call to a function in the “AHardwareBuffer” family.

I looked at the code for extracting binaries from the vendor’s firmware, and found the following lines (https://github.com/xigua-dev/device_oneplus_xigua/blob/183e8f24df43b390f8e26a8d7c6f40bcc99f65e5/extract-files.py#L55):

('odm/lib64/libHIS.so', 'odm/lib64/libOGLManager.so'): blob_fixup()
      .clear_symbol_version('AHardwareBuffer_allocate')
      .clear_symbol_version('AHardwareBuffer_describe')
      .clear_symbol_version('AHardwareBuffer_lock')
      .clear_symbol_version('AHardwareBuffer_release')
      .clear_symbol_version('AHardwareBuffer_unlock'),

“What the hell?” – I thought. “These are the same functions which are used near the place where my code crashes. Why are symbols there even versioned?”

Also, symbol versions for those functions were weird. They were not what we usually have in Linux, as in @GLIBC_2.24, but just @LIBNATIVEWINDOW.

It is worth noting that in Linux linking, symbols with symbol versions can be linked with libraries requiring functions without symbol versions, but not the other way round. I.e. if you library requires some symbols, versioned will do, but if it requires versioned, un-versioned will not work.

Then I added the copy of libnativewindow.so to the list of extracted binaries from the vendor’s firmware, and also a set of its dependencies transitively, and had to rewrite some “sonames” and replace dependencies, and boom, the camera started to start up successfully and not crash when trying to make photos. The photos still did not appear in the DCIM directory, but at least the camera did not crash.

I assume that the different versions of the AHardwareBuffer_* functions are creating different mappings of the P010 raw pixels arrays in the memory, and the code in the camera was expecting some particular mapping.

After I made the camera successfully try to make photos, I found that they do not appear in the DCIM directory. Luckily, at the same time some crashdumps, “tombstones” started to appear in the /data/tombstones/ directory.

Those crashdumps indicated that the crash was happening in the process called vendor.qti.camera.provider-service_64.

This seems to be a process written by Qualcomm, which is responsible for managing the camera devices in the Linux kernel (/dev/video*) and wrapping their usage into the Android HAL API.

I am still not exactly sure whether the camera app is interacting with the HAL process through the Binder calls directly, or through a different process running under the same cameraserver user, inventively called cameraserver.

In any case, the crash was happening because the function called PostProc was querying the “stream metadata” for the Logical Camera ID and the Physical Camera ID, and since both were missing, it was using some inappropriate heuristic to get the physical ID, and was accessing unmapped memory.

I started to investigate why exactly they were missing, but failed and never found the cause. However, grepping and searching for various relevant symbols create a strong impression in me that BBK has somehow noticeably patched the process called cameraserver, and the library called libcsextimpl.so, so that they would be writing more metadata into the streams, which the Qualcomm process, in turn, would be relying upon.

As far as I understand, this is not a polite way of writing code. The code of the “framework” should be unchanged, and only “ODM” code should be tweaked by the phone manufacturer, but hey, who am I to say what should and what should not be done outside of my area of expertise.

Initially I tried to do the same thing with libcsextimpl.so, but found that in the current Android version Google has turned it into a static library instead of a dynamic one, and even though making it dynamic again and trying to replace the open source version with a binary one, but it turned out to require way more dependency rewriting than I had expected, and I gave up.

However, it turned out that by using a small binary patch I can set that “Logical Camera ID” to always be zero, and it was enough to placate the PostProcess function to do its function correctly. I generated the patched binary in Ghidra (by using the “Patch instruction” feature), and then used diff -u and xxd to generate a two-line binary diff, which I added to the extract-files.py, and voila, it works.

.binary_regex_replace( # metadata error in in ChiOfflineIPEUpscale::PostProcess(PostProcSessionParams*)
    bytes.fromhex('6299009400060035844F40F962034039940040B982051836460100B0C6F444F9'),
    bytes.fromhex('0000805200060035844F40F9620340391400805282051836460100B0C6F444F9'))

1. https://source.android.com/docs/core/camera/camerax-vendor-extensions
2. https://source.android.com/docs/core/camera/camera3

So, in addition to capturing CCD pixel intensity, the camera also measures a lot of parameters, such as aperture, shutter delay, flash, timestamp of each frame, et cetera.

Those are saved into a data structure called “stream metadata”, which is basically a map/dictionary from “vendor tags” to metadata values. That is, metadata keys are binary numbers, not text strings, and each participant in this trio: Google, Qualcomm, BBK, is having its own system for converting strings such as "com.oplus.LogicalCameraID" into a code for querying metadata.

There is an issue about it https://issuetracker.google.com/issues/261760717

I suspect that in the current setup a lot of metadata is not getting into the pipeline.

Tags can be dumped using the property vendor.aps.logic.snapshot.metadata.debug=1, I think. The file called /data/vendor/camera/metadata/metadatatable.txt.

Making the camera work with SELinux set to “Enforcing” was the last part of my debugging process.

Setting SELinux to “Enforcing” produced quite a few denial errors in the dmesg, but fixing all of them with sepolicy-inject and chcon did not really make the camera work, or even start.

02-15 20:16:53.495  8722  8722 I PreviewReceived: type=1400 audit(0.0:287): avc:  denied  { open } for  path="/dev/__properties__/u:object_r:vendor_default_prop:s0" dev="tmpfs" ino=464 scontext=u:r:platform_app:s0:c512,c768 tcontext=u:object_r:vendor_default_prop:s0 tclass=file permissive=1 app=com.oplus.camera
02-15 20:16:53.495  8722  8722 I PreviewReceived: type=1400 audit(0.0:288): avc:  denied  { getattr } for  path="/dev/__properties__/u:object_r:vendor_default_prop:s0" dev="tmpfs" ino=464 scontext=u:r:platform_app:s0:c512,c768 tcontext=u:object_r:vendor_default_prop:s0 tclass=file permissive=1 app=com.oplus.camera
02-15 20:16:53.495  8722  8722 I PreviewReceived: type=1400 audit(0.0:289): avc:  denied  { map } for  path="/dev/__properties__/u:object_r:vendor_default_prop:s0" dev="tmpfs" ino=464 scontext=u:r:platform_app:s0:c512,c768 tcontext=u:object_r:vendor_default_prop:s0 tclass=file permissive=1 app=com.oplus.camera

and

06-10 00:05:59.224  1026  1026 E SELinux : avc:  denied  { find } for interface=vendor.qti.hardware.camera.postproc::IPostProcService sid=u:r:platform_app:s0:c512,c768 pid=8558 scontext=u:r:platform_app:s0:c512,c768 tcontext=u:object_r:vendor_hal_camera_postproc_hwservice:s0 tclass=hwservice_manager permissive=0
06-10 00:05:59.264     0     0 E [T201019] SELinux: avc:  denied  { find } for pid=11375 uid=10184 name=oiface scontext=u:r:platform_app:s0:c512,c768 tcontext=u:object_r:default_android_service:s0 tclass=service_manager permissive=0
06-10 00:05:59.265     0     0 E [T201019] SELinux: avc:  denied  { find } for pid=11375 uid=10184 name=oiface scontext=u:r:platform_app:s0:c512,c768 tcontext=u:object_r:default_android_service:s0 tclass=service_manager permissive=0

audit2allow was always meeting “neverallow” failures, and all of this stuff is so terribly documented that I was almost in despair.

However, two ideas came to my mind. The first one is that it does make a lot of sense to give the app process its own SELinux context (domain, label). It took me a while to find out how to do that, as doing it in a naive way (just adding a type opluscamera_app and adding a line to seapp_contexts) was causing a boot-loop.

However, at the end of the day I did something, and this something satisfies me.

I added a type opluscamera_app, and literally copied the definition of this type from platform_app.te. This made sure that my app has all the same permissions as a normal “platform app”, but is running at a different label, and I can filter the logs by this label. It also meant that I can add permissions to this label, without interfering with the main SELinux policy.

The drawback is that I had to declare this type not in the camera module but in the main sepolicy module (so my could would not be taken upstream), and add exceptions to the neverallow rules, which I do not mind, as there is only a single app running as this label, mine, and I do not intend to upstream it anyway.

I had to add neverallow exceptions like this:

system/sepolicy:

diff --git a/private/app.te b/private/app.te
index e4ac05b9c..441fb1889 100644
--- a/private/app.te
+++ b/private/app.te
@@ -521,7 +521,7 @@ allow appdomain zygote_tmpfs:file { map read };

 # Superuser capabilities.
 # bluetooth requires net_admin and wake_alarm. network stack app requires net_admin.
-neverallow { appdomain -bluetooth -network_stack } self:capability_class_set *;
+neverallow { appdomain -bluetooth -network_stack -opluscamera_app } self:capability_class_set *;

 # Block device access.
 neverallow appdomain dev_type:blk_file { read write };
@@ -746,7 +746,7 @@ neverallow {
 }:file no_x_file_perms;

 # Don't allow apps access to any of the following character devices.
-neverallow appdomain {
+neverallow {appdomain -opluscamera_app} {
     audio_device
     camera_device
     dm_device
@@ -758,6 +758,7 @@ neverallow appdomain {
 # needs access to /dev/video* for interfacing with the host
 neverallow {
     appdomain
+    -opluscamera_app
     -device_as_webcam
 } video_device:chr_file { read write };

device/qcom/sepolicy_vndr/sm8550/:

diff --git a/generic/vendor/common/domain.te b/generic/vendor/common/domain.te
index 0f4f0539e..2e256d693 100644
--- a/generic/vendor/common/domain.te
+++ b/generic/vendor/common/domain.te
@@ -85,6 +85,7 @@ neverallow { domain
 - hal_contexthub_default
 - hal_sensors_default
 - hal_camera_default
+- opluscamera_app
 userdebug_or_eng(` -vendor_usta_app')
 userdebug_or_eng(` -vendor_ustaservice_app')
 userdebug_or_eng(` -vendor_sysmonapp_app_test')
@@ -111,6 +112,6 @@ neverallow { domain
 userdebug_or_eng(` -vendor_sysmonapp_app_test')
 } vendor_qdsp_device:chr_file *;
 neverallow { domain -init -vendor_init - ueventd } vendor_qdsp_device:chr_file ~{r_file_perms};
-neverallow { appdomain - shell userdebug_or_eng(`-vendor_sysmonapp_app_test') } vendor_qdsp_device:chr_file ~{ioctl read};
+neverallow { appdomain - shell userdebug_or_eng(`-vendor_sysmonapp_app_test') -opluscamera_app } vendor_qdsp_device:chr_file ~{ioctl read};
 neverallow mediacodec vendor_qdsp_device:chr_file ~{ioctl read};

This allowed me to study the behaviour of the camera app without interfering with the workings of the “platform_app” label.

Another revelation, and I cannot call it by anything else, came when I was painfully and hopelessly staring through these two documents:

1. https://source.android.com/docs/core/architecture/aidl/aidl-hals#sepolicy
2. https://android.googlesource.com/platform/system/sepolicy/+/refs/heads/master/public/te_macros

The first one is written so badly that having it written in Swahili and read by a non-Swahili speaker would not hurt much. The second is just “the source”, you cannot expect much from it.

In any case, some of my SELinux denials were about vendor_hal_camera_postproc_hwservice, and none of the statements like hal_client_domain(opluscamera_app, hal_camera_postproc_hwservice), or hal_client_domain(opluscamera_app, hal_camera_postproc), or hal_hwclient_domain(opluscamera_app, hal_camera_postproc) were working.

Finally after spending a long time studying hardware/oplus/sepolicy/qti/vendor/, something clicked in my head and I wrote hal_client_domain(opluscamera_app, hal_camera). This worked. After looking once more at https://source.android.com/docs/core/architecture/aidl/aidl-hals#sepolicy , I decided that with a lot of mental gymnastics and imagination stretching, it is possible to see how the official document is claiming that this is actually what should be written, but, honestly, this approach to documentation is just horrible.

In any case, even though I had to add a type and edit the main tree, I am satisfied with this solution, because the app is running confined.

As a side-note, the camera app is actually requiring selinux_check_access(opluscamera_app), that is, its behaviour is different depending on whether SELinux is Enforcing, Permissive, or off. Which is a bad coding practice.

I do not have time to work on the camera any more, but I hope that someone can take over this honourable task.

I also did not find a good way to identify thread pools and find out which places are placing the tasks on the queue, so the code for me looks like “functions starting at libc pthread_start”.

It would be nice if someone submits a fragment to this howto on how to track task entry into thread pool.

This section is speculative, I am by no means an expert on Android Cameras.

I saw three processes interaction over Binder:

1. com.oplus.camera :: app
2. cameraserver :: Android camera HAL service process
3. vendor.qti.camera.provider-service_64 :: Qualcomm device process

The camera app had basically two big image-processing parts: preview and shooting. Both of those were implemented in the libAlgoProcess.so, which was dlopen from some of the camera built-in .so files. libAlgoProcess.so interacted with the cameraserver (?) somehow to obtain the image (buffer array), or several images as AHardwareBuffer_Planes.

After a buffer was obtained, it did post-processing on this buffer, either inside itself (depending on a key persist.vendor.camera.jpeghwencode), or passing the buffer to the vendor.qti.camera.provider-service_64 to do post-processing in hardware.

All of this was followed by “CameraMetadata”, indexed by “VendorTags”, which contained some important information about streams and buffers.

The camera process had some kind of “collection of typical pipelines”, called “usecases”, and in fact, a lot of Qualcomm code is named “CHI::usecase” – Camera HAL Interface Usecase.

I cannot believe that vendor.qti.camera.provider-service_64 is used only for post-processing. A camera process is naturally expected to also do capturing. But I did not find where capturing happens.