This is my note on basic android kernel development, covering the following aspects:
- Building the kernel from source
- Flashing a custom kernel to Pixel 6
- Basic development settings for android linux kernel
- Adding custom syscalls
- Writing loadable kernel modules
- Rust module on Android [TBD]
- KGDB through UART [TBD]
The source code of this post could be found at here.
0x00 Environment Preparation
Make sure the Android SDK
and google’s repo
tool are installed and available in PATH
:
# For adb and fastboot tools
export PATH=$PATH:/home/user/Android/Sdk/platform-tools/
# For repo
export PATH=$PATH:/home/user/.bin/
# For some countries that cannot access google = =
export REPO_URL='https://mirrors.tuna.tsinghua.edu.cn/git/git-repo/'
0x01 Build the kernel from source
Fetch the kernel repository
- Find the
BRANCH
corresponding to the target device here. For example,android-gs-raviole-5.10-android13-qpr3
for Pixel 6. - Use the
repo
tool to fetch the source code and required tools/firmware for the target device.
repo init -u https://mirrors.tuna.tsinghua.edu.cn/git/AOSP/kernel/manifest -b BRANCH
repo sync
Build
Run the following command to build the kernel.
build/build.sh
If there is no error we will get the packed images under /out/BRANCH/dist
(the vmlinux
and other binaries are also available in this folder.). For example:
$ ls out/android13-gs-pixel-5.10/dist | grep .img
boot.img
dtb.img
dtbo.img
initramfs.img
vendor-bootconfig.img
vendor_boot.img
vendor_dlkm.img
Note: in my laptop (latest Arch linux in 2023-12-31), the source code of branch will not compile successfully when using
libc 2.38
. An older libc solved this issue.
0x02 Flashing the custom kernel to the device
We should get an OEM-unlocked device and get it rooted. There are many tutorials on the XDA forum so I omit it here.
Before you start, make sure you back up all the data on your phone.
- Connnect Pixel 6 to the computer with a USB cable.
- Run
adb reboot bootloader
to restart the phone and enter the bootloader mode. - If we only made modifactions to the kernel, run the following command to flash the boot.img:
$ cd ./out/android13-gs-pixel-5.10/dist
$ sudo fastboot flash boot boot.img
# output
Sending 'boot_b' (65536 KB) OKAY [ 1.763s]
Writing 'boot_b' OKAY [ 0.291s]
Finished. Total time: 2.055s
If we want to replace everything:
sudo fastboot flash boot boot.img
sudo fastboot flash dtbo dtbo.img
sudo fastboot flash vendor_boot vendor_boot.img
sudo fastboot reboot fastboot
sudo fastboot flash vendor_dlkm vendor_dlkm.img
- Reboot into
Recovery mode
(for Pixel 6, pressing thevol +
button together with thepower
button). Clear all the data in the recovery mode. - Reboot and it’s done. We may have to re-root the phone since we replaced the patched
boot.img
with a new one, if we did not modify the kernel to give us root power.
0x03 Modify the kernel for fun and profit
Build the kernel for fuzzing
Just like Linux, enable the KCOV
option in android-kernel/private/gs-google/arch/arm64/configs/gki_defconfig
Defeat the anti-debug detection
A common method to detect whether the process is being debugged is to check the /proc/self/status
. We can just modify the source code of the proc fs
to bypass it, for example, masking the TracePid
:
diff --git a/fs/proc/array.c b/fs/proc/array.c
index 18a4588c3..119360232 100644
--- a/fs/proc/array.c
+++ b/fs/proc/array.c
@@ -184,6 +184,8 @@ static inline void task_state(struct seq_file *m, struct pid_namespace *ns,
seq_puts(m, "State:\t");
seq_puts(m, get_task_state(p));
+ tpid = 0;
+
seq_put_decimal_ull(m, "\nTgid:\t", tgid);
seq_put_decimal_ull(m, "\nNgid:\t", ngid);
seq_put_decimal_ull(m, "\nPid:\t", pid_nr_ns(pid, ns));
0x04 Android kernel source auditing and development
Env setting
For me the kernel source is too complex to read in a pure text editor (yeah I’m
stupidlazy).
Follow my gist on turning vscode to an IDE for Linux kernel, but replace the command to specify the out
path explicitly:
python ./scripts/clang-tools/gen_compile_commands.py -d /home/qsp/Android/android-kernel/out/android13-gs-pixel-5.10/private/gs-google
and then clangd
will give us a comfortable development experience(auto-completion, macro parsing, etc).
Adding custom syscall
This is exactly same as the linux kernel. Some references:
– https://www.kernel.org/doc/html/v4.10/process/adding-syscalls.html
– https://redirect.cs.umbc.edu/courses/undergraduate/421/spring21/docs/project0.html
– https://android.blogs.rice.edu/2017/05/16/adding-a-system-call-to-aarch64-linux/
For convenience I omit the CONFIG
adding step and just integrat the new syscall directly:
- Adding the
sys_??
prototype ininclude/linux/syscall.h
:
asmlinkage long sys_justatest(int a, int b);
- Adding a
syscall table entry
ininclude/uapi/asm-generic/unistd.h
. Modify the upper bound of syscall number__NR_syscalls
if necessary.
#define __NR_justatest 600
__SYSCALL(__NR_justatest, sys_justatest)
#undef __NR_syscalls
// #define __NR_syscalls 449
#define __NR_syscalls 666
- Add the source code for the syscall, for example wrting a
mysyscall.c
underkernel
folder. Note that the linux kernel useC99
standard:
#include <linux/syscalls.h>
#include <linux/printk.h>
#include <linux/kernel.h>
// asmlinkage long sys_justatest(int a, int b);
SYSCALL_DEFINE2(justatest, int, a, int, b)
{
int c;
printk(KERN_ALERT "Params: %d %d\n", a, b);
c = a * b;
return c;
}
- Add the object
mysyscall.o
underobj-y
inkernel/Makefile
. - Rebuild the kernel.
Full patch:
diff --git a/include/linux/syscalls.h b/include/linux/syscalls.h
index 1c170be3f..55a89b03b 100644
--- a/include/linux/syscalls.h
+++ b/include/linux/syscalls.h
@@ -1254,6 +1254,9 @@ asmlinkage long sys_old_mmap(struct mmap_arg_struct __user *arg);
*/
asmlinkage long sys_ni_syscall(void);
+/* Custom syscalls */
+asmlinkage long sys_justatest(int a, int b);
+
#endif /* CONFIG_ARCH_HAS_SYSCALL_WRAPPER */
diff --git a/include/uapi/asm-generic/unistd.h b/include/uapi/asm-generic/unistd.h
index f7b735dab..f7af79274 100644
--- a/include/uapi/asm-generic/unistd.h
+++ b/include/uapi/asm-generic/unistd.h
@@ -862,8 +862,12 @@ __SYSCALL(__NR_process_madvise, sys_process_madvise)
#define __NR_process_mrelease 448
__SYSCALL(__NR_process_mrelease, sys_process_mrelease)
+#define __NR_justatest 600
+__SYSCALL(__NR_justatest, sys_justatest)
+
#undef __NR_syscalls
-#define __NR_syscalls 449
+// #define __NR_syscalls 449
+#define __NR_syscalls 666
/*
* 32 bit systems traditionally used different
diff --git a/kernel/Makefile b/kernel/Makefile
index ed1aa304b..899f7f06b 100644
--- a/kernel/Makefile
+++ b/kernel/Makefile
@@ -10,7 +10,7 @@ obj-y = fork.o exec_domain.o panic.o \
extable.o params.o \
kthread.o sys_ni.o nsproxy.o \
notifier.o ksysfs.o cred.o reboot.o \
- async.o range.o smpboot.o ucount.o regset.o
+ async.o range.o smpboot.o ucount.o regset.o mysyscall.o
obj-$(CONFIG_USERMODE_DRIVER) += usermode_driver.o
obj-$(CONFIG_MODULES) += kmod.o
diff --git a/kernel/mysyscall.c b/kernel/mysyscall.c
new file mode 100644
index 000000000..afa0a836c
--- /dev/null
+++ b/kernel/mysyscall.c
@@ -0,0 +1,12 @@
+#include <linux/syscalls.h>
+#include <linux/printk.h>
+#include <linux/kernel.h>
+
+// asmlinkage long sys_justatest(int a, int b);
+SYSCALL_DEFINE2(justatest, int, a, int, b)
+{
+ int c;
+ printk(KERN_ALERT "Params: %d %d\n", a, b);
+ c = a * b;
+ return c;
+}
Mofidied files:
(base) ➜ gs-google git:(6e771b230) ✗ git status
HEAD detached at 6e771b230
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: include/linux/syscalls.h
modified: include/uapi/asm-generic/unistd.h
modified: kernel/Makefile
Untracked files:
(use "git add <file>..." to include in what will be committed)
kernel/mysyscall.c
no changes added to commit (use "git add" and/or "git commit -a")
Test the syscall with a binary program
We could test the syscall by a user mode ELF using NDK, which can be easily invoked from the adb as a regular linux binary.
Source:
#include <unistd.h>
#include <sys/syscall.h>
#include <stdio.h>
#define __NR_justatest 600
int main(){
int a,b,c;
a=1337, b=42;
c=syscall(__NR_justatest, a, b);
printf("Result from the kernel: %d * %d = %d\n", a, b, c);
return 0;
}
Android.mk:
LOCAL_PATH := $(call my-dir)
include $(CLEAR_VARS)
LOCAL_MODULE := user
LOCAL_SRC_FILES := user.c
include $(BUILD_EXECUTABLE)
# LOCAL_CFLAGS += -pie -fPIE
# LOCAL_LDFLAGS += -pie -fPIE
Application.mk:
APP_ABI := arm64-v8a
Build:
– Add NDK folder to your $PATH
– Run ndk-build
in this directory
– The output locates in libs/
Run:
sudo adb push libs/arm64-v8a/user /data/local/tmp/user
sudo adb shell
oriole:/ $ su
oriole:/ # cd /data/local/tmp
oriole:/data/local/tmp # file user
user: ELF shared object, 64-bit LSB arm64, dynamic (/system/bin/linker64), for Android 21, built by NDK r25c (9519653), BuildID=e4bc7b1a3610019b57e4afa69a9096d7212639a3, stripped
oriole:/data/local/tmp # chmod +x user
oriole:/data/local/tmp # ./user
Result from the kernel: 1337 * 42 = 56154
Writing loadable kernel modules (LKM)
Required files
Create a folder for the custom LKM, containing the LKM’s source code .c
, a Makefile
and a Kconfig
, for example:
lkm_test
├── Kconfig
├── lkm_test.c
└── Makefile
Kconfig:
config LKM_MOD
tristate "Linux Kernel Module Test"
default m
depends on MODULES
help
Linux Kernel Module Test
m
means compiled as a loadable kernel module.
Makefile:
obj-$(CONFIG_LKM_MOD) += lkm_test.o
Test code: lkm_test.c
#include <linux/init.h>
#include <linux/module.h>
#include <linux/kernel.h>
static int __init lkm_test_init(void) {
printk(KERN_INFO "lkm-test: init\n");
return 0;
}
static void __exit lkm_test_exit(void) {
printk(KERN_INFO "lkm-test: exit\n");
}
module_init(lkm_test_init);
module_exit(lkm_test_exit);
MODULE_AUTHOR("itewqq");
MODULE_DESCRIPTION("lkm_test");
MODULE_LICENSE("GPL");
MODULE_VERSION("0.1");
Add the LKM to the building system
- Add the source path to
drivers/Kconfig
. - Add the folder path to
drivers/Makefile
.
diff --git a/drivers/Kconfig b/drivers/Kconfig
index dfc46a7b7..081c51ec8 100644
--- a/drivers/Kconfig
+++ b/drivers/Kconfig
@@ -240,4 +240,6 @@ source "drivers/most/Kconfig"
source "drivers/bts/Kconfig"
+source "drivers/lkm_test/Kconfig"
+
endmenu
diff --git a/drivers/Makefile b/drivers/Makefile
index d22c56ce4..0ce049467 100644
--- a/drivers/Makefile
+++ b/drivers/Makefile
@@ -191,3 +191,5 @@ obj-$(CONFIG_INTERCONNECT) += interconnect/
obj-$(CONFIG_COUNTER) += counter/
obj-$(CONFIG_MOST) += most/
obj-$(CONFIG_EXYNOS_BTS) += bts/
+# custom modules
+obj-$(CONFIG_LKM_MOD) += lkm_test/
Build the LKM
Rebuild the kernel image, and lkm_test.ko
could be found at the same folder of boot.img
(see the above kernel buiding section).
Sign
According the docs in AOSP, LKM should be signed. However, IDK why the LKMs compiled together with the kernel by build/build.sh
could be installed successfully without siging.
I would appreciate it if someone could add clarification.
Install the LKM and test
$ sudo adb push lkm_test.ko /data/local/tmp/
$ sudo adb shell
oriole:/ $ cd data/local/tmp/
oriole:/data/local/tmp $ su
oriole:/data/local/tmp # insmod lkm_test.ko
oriole:/data/local/tmp # rmmod lkm_test
oriole:/data/local/tmp # dmesg | grep lkm-test
[10150.911992] lkm-test: init
[10292.042018] lkm-test: exit