Building a real-time kernel for the Nvidia Jetson TK1

NVIDIA Jetson TK1 development kit

NVIDIA Jetson TK1 development kit

Patching the kernel L4T kernel

Get the kernel source

[chris@thinkpad Tegra]$ wget http://developer.download.nvidia.com/embedded/L4T/r21_Release_v3.0/sources/kernel_src.tbz2
[chris@thinkpad Tegra]$ tar -xf kernel_src.tbz2
[chris@thinkpad Tegra]$ head kernel/Makefile
VERSION = 3
PATCHLEVEL = 10
SUBLEVEL = 40
EXTRAVERSION =
NAME = TOSSUG Baby Fish

# *DOCUMENTATION*
# To see a list of typical targets execute "make help"
# More info can be located in ./README
# Comments in this file are targeted only to the developer, do not
[chris@thinkpad Tegra]$

Get the PREEMPT_RT patch

[chris@thinkpad Tegra]$ wget https://www.kernel.org/pub/linux/kernel/projects/rt/3.10/older/patch-3.10.39-rt40.patch.gz
[chris@thinkpad Tegra]$ wget https://www.kernel.org/pub/linux/kernel/projects/rt/3.10/older/patch-3.10.44-rt46.patch.gz
[chris@thinkpad Tegra]$

Commit everything to git

[chris@thinkpad Tegra]$ gunzip patch-3.10.*
[chris@thinkpad Tegra]$ cd kernel
[chris@thinkpad kernel]$ git init
[chris@thinkpad kernel]$ git add .
[chris@thinkpad kernel]$ git commit -m "Added 3.10.40 L4T kernel source"
[chris@thinkpad kernel]$ cp ../patch-3.10.* .
[chris@thinkpad kernel]$ git add -f patch-3.10.*
[chris@thinkpad kernel]$ git commit -m"Added RT patch files"
[chris@thinkpad kernel]$

Apply the patch

With the stock kernel from kernel.org it’s quite easy to build a real time kernel, the patch will apply cleanly and normally you won’t encounter any build errors or stability issues. But the L4T kernel source has quite some modifications:

[chris@thinkpad TK1]$ wget https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.10.40.tar.xz
[chris@thinkpad TK1]$ tar -xf linux-3.10.40.tar.xz
[chris@thinkpad TK1]$ tar -xf kernel_src.tbz2
[chris@thinkpad TK1]$ git diff --dirstat linux-3.10.40 kernel
   3.2% kernel/arch/arm/boot/dts/tegra124-platforms/
  15.4% kernel/arch/arm/mach-tegra/
   3.1% kernel/arch/
   3.5% kernel/drivers/gpu/nvgpu/gk20a/
   3.9% kernel/drivers/input/
   4.4% kernel/drivers/media/platform/tegra/
   3.1% kernel/drivers/misc/
   3.7% kernel/drivers/net/wireless/bcmdhd/include/
   6.4% kernel/drivers/net/wireless/bcmdhd/
   6.7% kernel/drivers/net/wireless/sd8897/mlan/
   5.2% kernel/drivers/net/wireless/sd8897/mlinux/
   3.2% kernel/drivers/video/tegra/dc/
   3.2% kernel/drivers/video/
  18.8% kernel/drivers/
   4.0% kernel/include/
   4.3% kernel/sound/soc/codecs/
   3.1% kernel/sound/soc/
   3.6% kernel/
[chris@thinkpad TK1]$

So it won’t be that easy and we will have some more work.

Let’s see where the patch fails to apply:

[chris@thinkpad kernel]$ patch -p1 --dry-run <patch-3.10.39-rt40.patch |grep FAIL
Hunk #1 FAILED at 279.
1 out of 1 hunk FAILED
Hunk #1 FAILED at 53.
1 out of 1 hunk FAILED
Hunk #1 FAILED at 1254.
1 out of 1 hunk FAILED
Hunk #1 FAILED at 75.
1 out of 2 hunks FAILED
Hunk #2 FAILED at 230.
1 out of 9 hunks FAILED
Hunk #1 FAILED at 47.
1 out of 31 hunks FAILED
Hunk #5 FAILED at 1126.
Hunk #46 FAILED at 7458.
2 out of 47 hunks FAILED
Hunk #18 FAILED at 5532.
1 out of 21 hunks FAILED
[chris@thinkpad kernel]$ patch -p1 --dry-run <patch-3.10.44-rt46.patch |grep FAIL
Hunk #1 FAILED at 279.
1 out of 1 hunk FAILED
Hunk #1 FAILED at 53.
1 out of 1 hunk FAILED
Hunk #1 FAILED at 373.
1 out of 1 hunk FAILED
Hunk #1 FAILED at 1254.
1 out of 1 hunk FAILED
Hunk #1 FAILED at 75.
1 out of 2 hunks FAILED
Hunk #2 FAILED at 230.
1 out of 9 hunks FAILED
Hunk #1 FAILED at 47.
1 out of 31 hunks FAILED
Hunk #3 FAILED at 164.
1 out of 14 hunks FAILED
Hunk #5 FAILED at 1126.
Hunk #46 FAILED at 7457.
2 out of 47 hunks FAILED
Hunk #18 FAILED at 5532.
1 out of 21 hunks FAILED
[chris@thinkpad kernel]$

There exists no 3.10.40 patch for the kernel on the Jetson TK1, only a 3.10.39 and a 3.10.44 patch. Maybe this is the problem.

But the patches for the 3.10.39 and the 3.10.44 kernel are identical:

[chris@thinkpad kernel]$ git diff patch-3.10.39-rt40.patch patch-3.10.44-rt46.patch
[chris@thinkpad kernel]$ diff patch-3.10.39-rt40.patch patch-3.10.44-rt46.patch

So nothing important should have changed in between and this shouldn’t be the problem.

So let’s just patch it and apply the failed hunks later.

[chris@thinkpad kernel]$ patch -p1 <patch-3.10.44-rt46.patch >patch.log
[chris@thinkpad kernel]$ cat patch.log |grep FAILED>patch_fail.log
[chris@thinkpad kernel]$ git-cola
[chris@thinkpad kernel]$ git add .
[chris@thinkpad kernel]$ git-cola
[chris@thinkpad kernel]$ git commit -m"Applied the RT patch"
[chris@thinkpad kernel]$ gitk --all

Apply the failed hunks of the patch manually

[chris@thinkpad kernel]$ cat patch_fail.log
Hunk #1 FAILED at 279.
1 out of 1 hunk FAILED -- saving rejects to file arch/arm/mm/fault.c.rej
Hunk #1 FAILED at 53.
1 out of 1 hunk FAILED -- saving rejects to file drivers/misc/Makefile.rej
Hunk #1 FAILED at 373.
1 out of 1 hunk FAILED -- saving rejects to file include/linux/list.h.rej
Hunk #1 FAILED at 1254.
1 out of 1 hunk FAILED -- saving rejects to file include/linux/mm.h.rej
Hunk #1 FAILED at 75.
1 out of 2 hunks FAILED -- saving rejects to file init/main.c.rej
Hunk #2 FAILED at 230.
1 out of 9 hunks FAILED -- saving rejects to file kernel/fork.c.rej
Hunk #1 FAILED at 47.
1 out of 31 hunks FAILED -- saving rejects to file kernel/hrtimer.c.rej
Hunk #3 FAILED at 164.
1 out of 14 hunks FAILED -- saving rejects to file kernel/irq/manage.c.rej
Hunk #5 FAILED at 1126.
Hunk #46 FAILED at 7457.
2 out of 47 hunks FAILED -- saving rejects to file kernel/sched/core.c.rej
Hunk #18 FAILED at 5532.
1 out of 21 hunks FAILED -- saving rejects to file mm/page_alloc.c.rej
[chris@thinkpad kernel]$

I will just open the source file and the file with the rejected parts of the patch side by side in the editor, have a close look and add apply the failed hunks.

[chris@thinkpad kernel]$ kate arch/arm/mm/fault.c arch/arm/mm/fault.c.rej
[chris@thinkpad kernel]$ kate drivers/misc/Makefile drivers/misc/Makefile.rej
[chris@thinkpad kernel]$ kate include/linux/list.h include/linux/list.h.rej
[chris@thinkpad kernel]$ kate include/linux/mm.h include/linux/mm.h.rej
[chris@thinkpad kernel]$ kate init/main.c init/main.c.rej
[chris@thinkpad kernel]$ kate kernel/fork.c kernel/fork.c.rej
[chris@thinkpad kernel]$ kate kernel/hrtimer.c kernel/hrtimer.c.rej
[chris@thinkpad kernel]$ kate kernel/irq/manage.c kernel/irq/manage.c.rej
[chris@thinkpad kernel]$ kate kernel/sched/core.c kernel/sched/core.c.rej
[chris@thinkpad kernel]$ kate mm/page_alloc.c mm/page_alloc.c.rej
[chris@thinkpad kernel]$ git add .
[chris@thinkpad kernel]$ git commit -m "Applied the failed parts of the patch manually"

Compile the kernel

[chris@thinkpad kernel]$ ls arch/arm/configs/tegra*
arch/arm/configs/tegra12_android_defconfig        arch/arm/configs/tegra_bonaire_mods_defconfig         arch/arm/configs/tegra_loki_mods_defconfig
arch/arm/configs/tegra12_defconfig                arch/arm/configs/tegra_bonaire_sim_android_defconfig  arch/arm/configs/tegra_p2360_gnu_linux_defconfig
arch/arm/configs/tegra_ardbeg_mods_defconfig      arch/arm/configs/tegra_bonaire_sim_defconfig          arch/arm/configs/tegra_vcm30t124_android_defconfig
arch/arm/configs/tegra_bonaire_android_defconfig  arch/arm/configs/tegra_bonaire_sim_mods_defconfig     arch/arm/configs/tegra_vcm30t124_gnu_linux_defconfig
arch/arm/configs/tegra_bonaire_defconfig          arch/arm/configs/tegra_defconfig
[chris@thinkpad kernel]$
[chris@thinkpad kernel]$ export ARCH=arm
[chris@thinkpad kernel]$ export CROSS_COMPILE=arm-none-eabi-
[chris@thinkpad kernel]$ make tegra12_defconfig
  HOSTCC  scripts/basic/fixdep
  HOSTCC  scripts/kconfig/conf.o
  HOSTCC  scripts/kconfig/zconf.tab.o
  HOSTLD  scripts/kconfig/conf
arch/arm/configs/tegra12_defconfig:571:warning: override: reassigning to symbol TEGRA_WATCHDOG
#
# configuration written to .config
#
[chris@thinkpad kernel]$ make menuconfig
  HOSTCC  scripts/kconfig/lxdialog/checklist.o
  HOSTCC  scripts/kconfig/lxdialog/inputbox.o
  HOSTCC  scripts/kconfig/lxdialog/menubox.o
  HOSTCC  scripts/kconfig/lxdialog/textbox.o
  HOSTCC  scripts/kconfig/lxdialog/util.o
  HOSTCC  scripts/kconfig/lxdialog/yesno.o
  HOSTCC  scripts/kconfig/mconf.o
  HOSTLD  scripts/kconfig/mconf
scripts/kconfig/mconf Kconfig


*** End of the configuration.
*** Execute 'make' to start the build or try 'make help'.

[chris@thinkpad kernel]$ make

And it compiles fine.

Enable the PREEMPT_RT option and compile the kernel again

Now enable PREEMPT_RT:

[chris@thinkpad kernel]$ make clean
[chris@thinkpad kernel]$ make tegra12_defconfig
[chris@thinkpad kernel]$ make menuconfig
Kernel Features  --->
  Preemption Model (No Forced Preemption (Server))  --->
    (X) Fully Preemptible Kernel (RT)
  Preemption Model (Fully Preemptible Kernel (RT))  --->
[chris@thinkpad kernel]$ make

Fix the build errors

  CC      fs/timerfd.o
fs/timerfd.c: In function 'do_timerfd_settime':
fs/timerfd.c:384:30: error: 'struct timerfd_ctx' has no member named 'tmr'
make[1]: *** [fs/timerfd.o] Error 1
make: *** [fs] Error 2
[chris@thinkpad kernel]$

See git commit.

  CC      drivers/input/misc/mpu/inv_gyro.o
drivers/input/misc/mpu/inv_gyro.c: In function 'nvi_aux_dbg':
drivers/input/misc/mpu/inv_gyro.c:1369:2: error: 'struct mutex' has no member named 'count'
make[4]: *** [drivers/input/misc/mpu/inv_gyro.o] Error 1
make[3]: *** [drivers/input/misc/mpu] Error 2
make[2]: *** [drivers/input/misc] Error 2
make[1]: *** [drivers/input] Error 2
make: *** [drivers] Error 2
[chris@thinkpad kernel]$

See git commit, just commented it out.

In file included from drivers/usb/host/xhci.c:4776:0:
drivers/usb/host/xhci-tegra.c: In function 'must_have_sync_lock':
drivers/usb/host/xhci-tegra.c:454:2: error: 'struct mutex' has no member named 'owner'
drivers/usb/host/xhci-tegra.c: In function 'hsic_power_show':
drivers/usb/host/xhci-tegra.c:4203:36: warning: comparison of distinct pointer types lacks a cast [enabled by default]
drivers/usb/host/xhci-tegra.c: In function 'hsic_power_store':
drivers/usb/host/xhci-tegra.c:4231:36: warning: comparison of distinct pointer types lacks a cast [enabled by default]
drivers/usb/host/xhci-tegra.c: In function 'hsic_power_create_file':
drivers/usb/host/xhci-tegra.c:4266:4: warning: passing argument 1 of 'snprintf' discards 'const' qualifier from pointer target type [enabled by default]
In file included from include/asm-generic/bug.h:13:0,
                 from /mnt/data/data/Projects/Tegra/kernel/arch/arm/include/asm/bug.h:59,
                 from include/linux/bug.h:4,
                 from include/linux/thread_info.h:13,
                 from include/asm-generic/preempt.h:4,
                 from arch/arm/include/generated/asm/preempt.h:1,
                 from include/linux/preempt.h:18,
                 from include/linux/spinlock.h:50,
                 from include/linux/kobject.h:23,
                 from include/linux/pci.h:28,
                 from drivers/usb/host/xhci.c:23:
include/linux/kernel.h:361:5: note: expected 'char *' but argument is of type 'const char *'
In file included from drivers/usb/host/xhci.c:4776:0:
drivers/usb/host/xhci-tegra.c:4267:34: warning: assignment from incompatible pointer type [enabled by default]
drivers/usb/host/xhci-tegra.c:4268:35: warning: assignment from incompatible pointer type [enabled by default]
drivers/usb/host/xhci-tegra.c: In function 'tegra_xhci_probe':
drivers/usb/host/xhci-tegra.c:4346:28: warning: comparison between pointer and integer [enabled by default]
drivers/usb/host/xhci-tegra.c:4291:6: warning: unused variable 'irq' [-Wunused-variable]
drivers/usb/host/xhci.c: In function 'tegra_xhci_resume':
drivers/usb/host/xhci-tegra.c:3752:18: warning: ignoring return value of 'regulator_enable', declared with attribute warn_unused_result [-Wunused-result]
drivers/usb/host/xhci-tegra.c:3753:18: warning: ignoring return value of 'regulator_enable', declared with attribute warn_unused_result [-Wunused-result]
make[3]: *** [drivers/usb/host/xhci.o] Error 1
make[2]: *** [drivers/usb/host] Error 2
make[1]: *** [drivers/usb] Error 2
make: *** [drivers] Error 2
[chris@thinkpad kernel]$

See git commit.

  CC      drivers/video/tegra/host/nvhost_syncpt.o
drivers/video/tegra/host/nvhost_syncpt.c: In function 'nvhost_mutex_unlock':
drivers/video/tegra/host/nvhost_syncpt.c:569:34: error: macro "mutex_unlock" passed 2 arguments, but takes just 1
drivers/video/tegra/host/nvhost_syncpt.c:569:13: error: value computed is not used [-Werror=unused-value]
cc1: all warnings being treated as errors
make[4]: *** [drivers/video/tegra/host/nvhost_syncpt.o] Error 1
make[3]: *** [drivers/video/tegra/host] Error 2
make[2]: *** [drivers/video/tegra] Error 2
make[1]: *** [drivers/video] Error 2
make: *** [drivers] Error 2
[chris@thinkpad kernel]$

See git commit.

  CC      drivers/video/tegra/nvmap/nvmap_pp.o
In file included from drivers/video/tegra/nvmap/nvmap_pp.c:28:0:
include/linux/shrinker.h:37:2: error: unknown type name 'atomic_long_t'
make[4]: *** [drivers/video/tegra/nvmap/nvmap_pp.o] Error 1
make[3]: *** [drivers/video/tegra/nvmap] Error 2
make[2]: *** [drivers/video/tegra] Error 2
make[1]: *** [drivers/video] Error 2
make: *** [drivers] Error 2
[chris@thinkpad kernel]$

There were some other problems, you find all the modification in the git commit log at https://github.com/christianjann/L4T_PREEMPT_RT/commits/master.

Everything is patched, kernel compiles

[chris@thinkpad kernel]$ git shortlog
Christian Jann (11):
      Added 3.10.40 L4T kernel source
      Added RT patch files
      Applied the RT patch
      Applied the failed parts of the patch manually
      error: 'struct timerfd_ctx' has no member named 'tmr'
      error: 'struct mutex' has no member named 'count'
      error: 'struct mutex' has no member named 'owner'
      error: macro "mutex_unlock" passed 2 arguments, but takes just 1
      error: unknown type name 'atomic_long_t'
      Fix: Unable to handle kernel NULL pointer dereference at virtual address 00000000
      error: invalid 'asm': operand number out of range

[chris@thinkpad kernel]$
[chris@thinkpad kernel]$ export ARCH=arm
[chris@thinkpad kernel]$ export CROSS_COMPILE=arm-none-eabi-
[chris@thinkpad kernel]$ make clean
[chris@thinkpad kernel]$ make tegra12_defconfig
[chris@thinkpad kernel]$ make menuconfig
Kernel Features  --->
  Preemption Model (No Forced Preemption (Server))  --->
    (X) Fully Preemptible Kernel (RT)
  Preemption Model (Fully Preemptible Kernel (RT))  --->
[chris@thinkpad kernel]$ grep -i PREEMPT_RT .config
CONFIG_PREEMPT_RT_BASE=y
# CONFIG_PREEMPT_RTB is not set
CONFIG_PREEMPT_RT_FULL=y
[chris@thinkpad kernel]$ make -j8

...

  LD [M]  net/mac80211/mac80211.ko
  LD [M]  net/rfkill/rfkill-gpio.ko
  LD [M]  net/rfkill/rfkill.ko
  LD [M]  net/wireless/cfg80211.ko
  LD      arch/arm/boot/compressed/vmlinux
  OBJCOPY arch/arm/boot/zImage
  Kernel: arch/arm/boot/zImage is ready
[chris@thinkpad kernel]$

You will find the patched kernel source at https://github.com/christianjann/L4T_PREEMPT_RT.

Installing the kernel on the board

You will find the compiled kernel image and the modules at:

I assume you have a clean installation of Linux For Tegra R21.3 on the board:

You will find some additional information at Debugging the Linux kernel via JTAG on the NVIDIA Jetson TK1 / Jetson Pro DevKit.

Building the L4T_PREEMPT_RT kernel from source

[chris@thinkpad TK1]$ git clone https://github.com/christianjann/L4T_PREEMPT_RT.git
[chris@thinkpad TK1]$ cd L4T_PREEMPT_RT
[chris@thinkpad L4T_PREEMPT_RT]$ export ARCH=arm
[chris@thinkpad L4T_PREEMPT_RT]$ export CROSS_COMPILE=arm-none-eabi-
[chris@thinkpad L4T_PREEMPT_RT]$ make clean
[chris@thinkpad L4T_PREEMPT_RT]$ make tegra12_defconfig
  HOSTCC  scripts/basic/fixdep
  HOSTCC  scripts/kconfig/conf.o
  HOSTCC  scripts/kconfig/zconf.tab.o
  HOSTLD  scripts/kconfig/conf
arch/arm/configs/tegra12_defconfig:571:warning: override: reassigning to symbol TEGRA_WATCHDOG
#
# configuration written to .config
#
[chris@thinkpad L4T_PREEMPT_RT]$ make menuconfig
Kernel Features  --->
  Preemption Model (No Forced Preemption (Server))  --->
    (X) Fully Preemptible Kernel (RT)
  Preemption Model (Fully Preemptible Kernel (RT))  --->
[chris@thinkpad L4T_PREEMPT_RT]$ make -j5
...
  LD      vmlinux
  SORTEX  vmlinux
  SYSMAP  System.map
  OBJCOPY arch/arm/boot/Image
  Building modules, stage 2.
  Kernel: arch/arm/boot/Image is ready
  MODPOST 54 modules
  AS      arch/arm/boot/compressed/head.o
  GZIP    arch/arm/boot/compressed/piggy.gzip
  CC      arch/arm/boot/compressed/misc.o
  CC      arch/arm/boot/compressed/decompress.o
  CC      arch/arm/boot/compressed/string.o
  SHIPPED arch/arm/boot/compressed/hyp-stub.S
  SHIPPED arch/arm/boot/compressed/lib1funcs.S
  SHIPPED arch/arm/boot/compressed/ashldi3.S
  AS      arch/arm/boot/compressed/hyp-stub.o
  AS      arch/arm/boot/compressed/lib1funcs.o
  AS      arch/arm/boot/compressed/ashldi3.o
  AS      arch/arm/boot/compressed/piggy.gzip.o
  LD      arch/arm/boot/compressed/vmlinux
  OBJCOPY arch/arm/boot/zImage
  Kernel: arch/arm/boot/zImage is ready
  CC      crypto/md4.mod.o
 ...
  LD [M]  net/rfkill/rfkill.ko
  LD [M]  net/wireless/cfg80211.ko
[chris@thinkpad L4T_PREEMPT_RT]$ cp arch/arm/boot/zImage .
[chris@thinkpad L4T_PREEMPT_RT]$ mkdir modules_install
[chris@thinkpad L4T_PREEMPT_RT]$ make INSTALL_MOD_PATH=./modules_install modules_install
[chris@thinkpad L4T_PREEMPT_RT]$

Using the pre-compiled L4T_PREEMPT_RT kernel

[chris@thinkpad ~]$ mkdir L4T_PREEMPT_RT
[chris@thinkpad ~]$ cd L4T_PREEMPT_RT
[chris@thinkpad L4T_PREEMPT_RT]$ curl -L http://sourceforge.net/projects/janncc/files/jetson-tk1/3.10.40-rt46-g7477c2b/zImage.tar.gz/download -o zImage.tar.gz
[chris@thinkpad L4T_PREEMPT_RT]$ curl -L http://sourceforge.net/projects/janncc/files/jetson-tk1/3.10.40-rt46-g7477c2b/modules_install.tar.gz/download -o modules_install.tar.gz
[chris@thinkpad L4T_PREEMPT_RT]$ ls
modules_install.tar.gz  zImage.tar.gz
[chris@thinkpad L4T_PREEMPT_RT]$ tar -xf modules_install.tar.gz
[chris@thinkpad L4T_PREEMPT_RT]$ tar -xf zImage.tar.gz
[chris@thinkpad L4T_PREEMPT_RT]$ ls
modules_install  modules_install.tar.gz  zImage  zImage.tar.gz
[chris@thinkpad L4T_PREEMPT_RT]$

Installation on the board

On the board, allow root logins and restart the ssh server:

ubuntu@tegra-ubuntu:~$ sudo vim /etc/ssh/sshd_config
  PermitRootLogin yes

ubuntu@tegra-ubuntu:~$ sudo service ssh restart
ssh stop/waiting
ssh start/running, process 1348
ubuntu@tegra-ubuntu:~$

ubuntu@tegra-ubuntu:~$ sudo passwd root
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
ubuntu@tegra-ubuntu:~$ sudo ifconfig eth0 up
ubuntu@tegra-ubuntu:~$ sudo dhclient eth0
ubuntu@tegra-ubuntu:~$ ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:04:4b:25:cb:f0
          inet addr:10.42.0.26  Bcast:10.42.0.255  Mask:255.255.255.0
          inet6 addr: fe80::204:4bff:fe25:cbf0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:97 errors:0 dropped:0 overruns:0 frame:0
          TX packets:173 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:12598 (12.5 KB)  TX bytes:19498 (19.4 KB)

ubuntu@tegra-ubuntu:~$ ping google.de
PING google.de (173.194.112.95) 56(84) bytes of data.
64 bytes from fra07s29-in-f31.1e100.net (173.194.112.95): icmp_seq=1 ttl=49 time=22.0 ms
^C
--- google.de ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 22.072/22.072/22.072/0.000 ms
ubuntu@tegra-ubuntu:~$

Try to connect from the host:

[chris@thinkpad L4T_PREEMPT_RT]$ ssh root@10.42.0.26
The authenticity of host '10.42.0.26 (10.42.0.26)' can't be established.
ECDSA key fingerprint is 70:37:dd:30:e0:3c:3b:51:fd:01:cf:3e:84:5a:a2:f0.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.42.0.26' (ECDSA) to the list of known hosts.
root@10.42.0.26's password:
Welcome to Ubuntu 14.04.1 LTS (GNU/Linux 3.10.40-gc017b03 armv7l)

 * Documentation:  https://help.ubuntu.com/

0 packages can be updated.
0 updates are security updates.


The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

root@tegra-ubuntu:~#

Copy the kernel image to the board:

[chris@thinkpad L4T_PREEMPT_RT]$ scp zImage root@10.42.0.26:/boot/zImage_RT
root@10.42.0.26's password:
zImage                                                                                                                                                       100% 5919KB   5.8MB/s   00:00
[chris@thinkpad L4T_PREEMPT_RT]$

Install the kernel modules:

[chris@thinkpad L4T_PREEMPT_RT]$ rsync -av ./modules_install/ root@10.42.0.26:/

Edit the bootloader configuration and add a second menu entry for our new kernel:

[chris@thinkpad L4T_PREEMPT_RT]$ ssh root@10.42.0.26
root@10.42.0.26's password:
Welcome to Ubuntu 14.04.1 LTS (GNU/Linux 3.10.40-gc017b03 armv7l)

 * Documentation:  https://help.ubuntu.com/

343 packages can be updated.
136 updates are security updates.

Last login: Sat Jan  1 00:54:14 2000 from 10.42.0.1
root@tegra-ubuntu:~# vim /boot/extlinux/extlinux.conf
root@tegra-ubuntu:~# cat /boot/extlinux/extlinux.conf
TIMEOUT 30
DEFAULT primary

MENU TITLE Jetson-TK1 eMMC boot options

LABEL primary
      MENU LABEL primary kernel
      LINUX /boot/zImage
      FDT /boot/tegra124-jetson_tk1-pm375-000-c00-00.dtb
      APPEND console=ttyS0,115200n8 console=tty1 no_console_suspend=1 lp0_vec=2064@0xf46ff000 mem=2015M@2048M memtype=255 ddr_die=2048M@2048M section=256M pmuboard=0x0177:0x0000:0x02:0x43:0x00 tsec=32M@3913M otf_key=c75e5bb91eb3bd947560357b64422f85 usbcore.old_scheme_first=1 core_edp_mv=1150 core_edp_ma=4000 tegraid=40.1.1.0.0 debug_uartport=lsport,3 power_supply=Adapter audio_codec=rt5640 modem_id=0 android.kerneltype=normal fbcon=map:1 commchip_id=0 usb_port_owner_info=0 lane_owner_info=6 emc_max_dvfs=0 touch_id=0@0 board_info=0x0177:0x0000:0x02:0x43:0x00 root=/dev/mmcblk0p1 rw rootwait tegraboot=sdmmc gpt

LABEL secondary (PREEMPT_RT)
      MENU LABEL secondary (PREEMPT_RT)
      LINUX /boot/zImage_RT
      FDT /boot/tegra124-jetson_tk1-pm375-000-c00-00.dtb
      APPEND console=ttyS0,115200n8 console=tty1 no_console_suspend=1 lp0_vec=2064@0xf46ff000 mem=2015M@2048M memtype=255 ddr_die=2048M@2048M section=256M pmuboard=0x0177:0x0000:0x02:0x43:0x00 tsec=32M@3913M otf_key=c75e5bb91eb3bd947560357b64422f85 usbcore.old_scheme_first=1 core_edp_mv=1150 core_edp_ma=4000 tegraid=40.1.1.0.0 debug_uartport=lsport,3 power_supply=Adapter audio_codec=rt5640 modem_id=0 android.kerneltype=normal fbcon=map:1 commchip_id=0 usb_port_owner_info=0 lane_owner_info=6 emc_max_dvfs=0 touch_id=0@0 board_info=0x0177:0x0000:0x02:0x43:0x00 root=/dev/mmcblk0p1 rw rootwait tegraboot=sdmmc gpt


root@tegra-ubuntu:~$ sudo reboot

Testing the real-time kernel

Testing on the TK1

ubuntu@tegra-ubuntu:~$ uname -a
Linux tegra-ubuntu 3.10.40-rt46-g7477c2b #2 SMP PREEMPT RT Sun May 17 15:26:34 CEST 2015 armv7l armv7l armv7l GNU/Linux
ubuntu@tegra-ubuntu:~$

Sometimes it boots... sometimes not... so there is still work...

[   12.941547] vgaarb: this pci device is not a vga device
[   13.300705] vgaarb: this pci device is not a vga device
[   13.843631] BUG: scheduling while atomic: swapper/2/0/0x00000002
[   13.843647] Preemption disabled at:[<  (null)>]   (null)
[   13.923618] BUG: scheduling while atomic: swapper/3/0/0x00000002
[   13.923622] BUG: scheduling while atomic: swapper/1/0/0x00000002
[   13.923646] Preemption disabled at:
[   13.923649] Preemption disabled at:[<  (null)>]   (null)
[   13.923813] bad: scheduling from the idle thread!
[   13.983612] BUG: scheduling while atomic: swapper/0/0/0x00000002
[   13.983617] BUG: scheduling while atomic: swapper/1/0/0x00000002
[   13.983619] Modules linked in:
[   13.983621] Modules linked in:

It seems with the network cable disconnected it boots more reliable up to the command prompt.

Running some rt-tests as described here: https://git.kernel.org/cgit/linux/kernel/git/clrkwllms/rt-tests.git/tree/README.markdown:

To run one test thread per CPU or per CPU core, each thread on a separate
processor, type

    sudo ./cyclictest -a -t -n -p99

On a non-realtime system, you may see something like

   T: 0 ( 3431) P:99 I:1000 C: 100000 Min: 5 Act:   10 Avg:   14 Max:   39242
   T: 1 ( 3432) P:98 I:1500 C:  66934 Min: 4 Act:   10 Avg:   17 Max:   39661

The rightmost column contains the most important result, i.e. the worst-case
latency of 39.242 milliseconds. On a realtime-enabled system, the result may
look like

   T: 0 ( 3407) P:99 I:1000 C: 100000 Min: 7 Act:   10 Avg:   10 Max:      18
   T: 1 ( 3408) P:98 I:1500 C:  67043 Min: 7 Act:    8 Avg:   10 Max:      22

and, thus, indicate an apparent short-term worst-case latency of 18
microseconds.

Compile the cyclictest program:

ubuntu@tegra-ubuntu:~$ sudo apt-get update
ubuntu@tegra-ubuntu:~$ sudo apt-get install build-essential
ubuntu@tegra-ubuntu:~$ sudo apt-get install git
ubuntu@tegra-ubuntu:~$ git clone https://git.kernel.org/pub/scm/linux/kernel/git/clrkwllms/rt-tests.git
ubuntu@tegra-ubuntu:~$ cd rt-tests/
ubuntu@tegra-ubuntu:~/rt-tests$ make

Run it on the standard kernel:

ubuntu@tegra-ubuntu:~/rt-tests$ uname -a
Linux tegra-ubuntu 3.10.40-gc017b03 #1 SMP PREEMPT Mon Feb 2 17:50:48 PST 2015 armv7l armv7l armv7l GNU/Linux
ubuntu@tegra-ubuntu:~/rt-tests$ sudo ./cyclictest -a -t -n -p99
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 2.09 1.67 1.23 1/153 3999

T: 0 ( 3980) P:99 I:1000 C: 378948 Min:      9 Act:  566 Avg:  412 Max:   18046
ubuntu@tegra-ubuntu:~/rt-tests$

Run it on the RT kernel:

ubuntu@tegra-ubuntu:~$ uname -a
Linux tegra-ubuntu 3.10.40-rt46-g7477c2b #2 SMP PREEMPT RT Sun May 17 15:26:34 CEST 2015 armv7l armv7l armv7l GNU/Linux
ubuntu@tegra-ubuntu:~$ cd rt-tests/
ubuntu@tegra-ubuntu:~/rt-tests$ sudo ./cyclictest -a -t -n -p99
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 1.49 0.93 0.4 1/216 1081

T: 0 ( 1081) P:99 I:1000 C: 172793 Min:     91 Act:  134 Avg:  152 Max:    1813
ubuntu@tegra-ubuntu:~/rt-tests$

The worst-case latency of the RT kernel is far too high, this needs to be improved.

Comparison with the Jetson Pro

On the Jetson Pro, even with the stock 3.10.61 kernel the worst-case latency is much less than on the TK1, even under load... I don’t know why yet...

With the stock kernel:

root@nvidia rt-tests$ uname -a
Linux nvidia 3.10.61-g456b5a6 #1 SMP PREEMPT Thu Feb 12 10:39:49 PST 2015
armv7l armv7l armv7l GNU/Linux
root@nvidia ~$ apt-get install stress
root@nvidia ~$ stress --cpu 4 --timeout 60
stress: info: [1031] dispatching hogs: 4 cpu, 0 io, 0 vm, 0 hdd
stress: info: [1031] successful run completed in 60s
root@nvidia ~$
root@nvidia ~$ top
top - 11:10:35 up 13 min,  3 users,  load average: 2.39, 0.74, 0.31
Tasks: 123 total,   5 running, 118 sleeping,   0 stopped,   0 zombie
%Cpu(s): 99.3 us, 0.3 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.4 si, 0.0 st
KiB Mem:   1898492 total,   221608 used,  1676884 free,     7660 buffers
KiB Swap:        0 total,        0 used,        0 free.   163416 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 1033 root      20   0    1652     72      0 R  99.7  0.0   0:49.52 stress
 1034 root      20   0    1652     72      0 R  99.7  0.0   0:49.48 stress
 1032 root      20   0    1652     72      0 R  98.7  0.0   0:49.28 stress
 1035 root      20   0    1652     72      0 R  98.7  0.0   0:48.68 stress
  990 root      20   0   34308    540    444 S   2.0  0.0   0:05.19 cyclictest
   31 root      20   0       0      0      0 S   0.3  0.0   0:00.55 kworker/2:1
  919 root      20   0    9716   3048   2308 S   0.3  0.2   0:10.24 sshd
    1 root      20   0    3400   2024   1112 S   0.0  0.1   0:09.11 init
...
root@nvidia rt-tests$ ./cyclictest -a -t -n -p99
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 2.64 0.85 0.35 1/124 1070

T: 0 (  991) P:99 I:1000 C: 177195 Min:      4 Act:    9 Avg:    7 Max:     142
T: 1 (  992) P:99 I:1500 C: 118130 Min:      4 Act:    9 Avg:    7 Max:      39
T: 2 (  993) P:99 I:2000 C:  88597 Min:      4 Act:    8 Avg:    6 Max:      20
T: 3 (  994) P:99 I:2500 C:  70878 Min:      5 Act:    8 Avg:    6 Max:     134
root@nvidia rt-tests$

With the RT kernel:

root@nvidia rt-tests$ uname -a
Linux nvidia 3.10.61-rt65 #3 SMP PREEMPT RT Thu Apr 23 14:40:49 CEST 2015 armv7l armv7l armv7l GNU/Linux
root@nvidia ~$ stress --cpu 4 --timeout 60
stress: info: [1430] dispatching hogs: 4 cpu, 0 io, 0 vm, 0 hdd
stress: info: [1430] successful run completed in 60s
root@nvidia ~$
root@nvidia ~$ top
top - 08:31:18 up 9 min,  3 users,  load average: 2.31, 1.21, 0.54
Tasks: 206 total,   5 running, 201 sleeping,   0 stopped,   0 zombie
%Cpu(s): 99.8 us,  0.2 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   1895876 total,    99320 used,  1796556 free,     8776 buffers
KiB Swap:        0 total,        0 used,        0 free.    38052 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 1441 root      20   0    1652     72      0 R  98.7  0.0   0:20.89 stress
 1440 root      20   0    1652     72      0 R  97.4  0.0   0:20.78 stress
 1438 root      20   0    1652     72      0 R  97.1  0.0   0:20.71 stress
 1439 root      20   0    1652     72      0 R  97.1  0.0   0:20.75 stress
 1443 root      20   0   34308    540    444 S   3.0  0.0   0:00.22 cyclictest
 1019 root      20   0    9716   3044   2308 S   2.6  0.2   0:01.14 sshd
  161 root     -51   0       0      0      0 S   1.7  0.0   0:01.29 irq/53-ehci_hcd
    3 root      -2   0       0      0      0 S   0.7  0.0   0:00.93 ksoftirqd/0
...
root@nvidia rt-tests$ ./cyclictest -a -t -n -p99
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 2.30 1.09 0.48 3/208 1435

T: 0 ( 1426) P:99 I:1000 C:  74425 Min:      5 Act:    9 Avg:    6 Max:      17
T: 1 ( 1427) P:99 I:1500 C:  49617 Min:      5 Act:   10 Avg:    6 Max:      13
T: 2 ( 1428) P:99 I:2000 C:  37212 Min:      5 Act:    8 Avg:    6 Max:      14
T: 3 ( 1429) P:99 I:2500 C:  29770 Min:      5 Act:    8 Avg:    5 Max:      14
root@nvidia rt-tests$

TODO

  • The kernel panics need to be fixed.
  • The worst-case latency of the RT kernel on the TK1 is far too high, this needs to be improved.
  • There are more realistic tests, this needs to be tested too.

To be continued...