August 24, 2013

Debugging the iMX233-OLinuXino via SJTAG with OpenOCD

OLinuXino Micro, SEGGER J-Link JTAG/SWD Debugger and iMX233-SJTAG

Overview

What you will need

Hardware

OlinuXino Micro, iMX233-SJTAG, SEGGER J-Link, ARM-USB-TINY

OLinuXino, iMX233-SJTAG, SEGGER J-Link or ARM-USB-TINY

A iMX233 based OLinuXino board: https://www.olimex.com/Products/OLinuXino/iMX233/

I’m using a iMX233-OLinuXino-MICRO and since it comes without a plastic header for the SJTAG interface I have added one.

OlinuXino Micro with SJTAG header

A standard 20-pin JTAG debugger, I have tested the following two:
- SEGGER J-Link JTAG/SWD Debugger (Version 6)
  Update: I was not successful by using the J-Link to debug the kernel, maybe the iMX233-SJTAG is somehow incompatible with the J-Link.
- Olimex’s ARM-USB-TINY
  - https://www.olimex.com/Products/ARM/JTAG/ARM-USB-TINY
  - This is the old slow version, there exists also a newer version: https://www.olimex.com/Products/ARM/JTAG/ARM-USB-TINY-H/
Olimex’s iMX233-SJTAG adapter
- https://www.olimex.com/Products/ARM/JTAG/iMX233-SJTAG
- It should be possible to power the adapter from USB or SJTAG.
- The red wire on the adapter is power, the pin is missing on the connector of my adapter so that I need to power it over USB.
  
  Update: I’ve repaired the broken pin, It works better if the debugger powers up at the same time as the OLinXino.
- The green LED on the iMX233-SJTAG should be off before starting OpenOCD. If not: press the yellow button or power off/on your OLinuXino until it goes off. Else it is likely that you get errors in OpenOCD.

Software

OpenOCD

I’m building OpenOCD (the Open On-Chip Debugger) from source:

[chris@thinkpad ARM]$ git clone git://git.code.sf.net/p/openocd/code openocd.git
[chris@thinkpad ARM]$ cd openocd.git
[chris@thinkpad openocd.git]$ ./bootstrap
[chris@thinkpad openocd.git]$ ./configure --prefix=/usr --enable-maintainer-mode --enable-stlink --enable-ti-icdi --enable-jlink --enable-ft2232_libftdi
[chris@thinkpad openocd.git]$ make
[chris@thinkpad openocd.git]$ sudo make install
[chris@thinkpad openocd.git]$ openocd --version
Open On-Chip Debugger 0.8.0-dev-00121-ga4d3446 (2013-08-24-15:55)
Licensed under GNU GPL v2
For bug reports, read
        http://openocd.sourceforge.net/doc/doxygen/bugs.html
[chris@thinkpad openocd.git]$

An ARM bare metal cross compiler

You can get the latest arm-none-eabi-gcc compiler from Launchpad:

https://launchpad.net/gcc-arm-embedded/+download

Extract the tarball somewhere and add the compiler to the $PATH, e.g. edit your ~/.bashrc file and add a line like this:

export PATH=$PATH:$HOME/tools/gcc-arm-none-eabi-4_7-2013q1/bin

[chris@thinkpad ~]$ arm-none-eabi-gcc --version
arm-none-eabi-gcc (GNU Tools for ARM Embedded Processors) 4.7.3 20130312 (release) [ARM/embedded-4_7-branch revision 196615]
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[chris@thinkpad ~]$

Setup OpenOCD

Luckily Olimex provides a sample configuration (mirror) for the iMX233.

OpenOCD schips also with some configuration files for other iMX processors if you are using a similar one:

[chris@thinkpad ~]$ ls /usr/share/openocd/scripts/board/imx*
/usr/share/openocd/scripts/board/imx27ads.cfg
/usr/share/openocd/scripts/board/imx31pdk.cfg
/usr/share/openocd/scripts/board/imx53-m53evk.cfg
/usr/share/openocd/scripts/board/imx27lnst.cfg
/usr/share/openocd/scripts/board/imx35pdk.cfg
/usr/share/openocd/scripts/board/imx28evk.cfg
/usr/share/openocd/scripts/board/imx53loco.cfg
[chris@thinkpad ~]$

[chris@thinkpad Downloads]$ mkdir SJTAG-Example
[chris@thinkpad Downloads]$ cd SJTAG-Example/
[chris@thinkpad SJTAG-Example]$ wget https://www.olimex.com/Products/OLinuXino/iMX233/_resources/SJTAG_TINY-H_imx233_openOCD.zip
[chris@thinkpad SJTAG-Example]$ unzip SJTAG_TINY-H_imx233_openOCD.zip
Archive:  SJTAG_TINY-H_imx233_openOCD.zip
  inflating: imx233.cfg
  inflating: README.txt
  inflating: Example 2.png
  inflating: Example 1.png
  inflating: olimex-arm-usb-tiny-h.cfg
[chris@thinkpad SJTAG-Example]$ cat imx233.cfg
# page 3-34 of "MCIMC27 Multimedia Applications Processor Reference Manual, Rev 0.3"
# SRST pulls TRST
#
# Without setting these options correctly you'll see all sorts
# of weird errors, e.g. MOE=0xe, invalid cpsr values, reset
# failing, etc.
reset_config trst_and_srst srst_pulls_trst

set _CHIPNAME imx23

set _ENDIAN little

# Note above there are 2 taps

# The CPU TAP.
jtag newtap $_CHIPNAME cpu -irlen 4 -expected-id 0x079264f3


adapter_khz 800

# Create the GDB Target.
set _TARGETNAME $_CHIPNAME.cpu
target create $_TARGETNAME arm926ejs -endian $_ENDIAN -chain-position $_TARGETNAME -variant arm926ejs

# REVISIT what operating environment sets up this virtual address mapping?
$_TARGETNAME configure -work-area-virt 0x0 -work-area-phys 0x0 \
  -work-area-size  0x4000 -work-area-backup 1
# Internal to the chip, there is 16K of usable SRAM.
#

arm7_9 dcc_downloads enable
arm7_9 fast_memory_access  enable


[chris@thinkpad SJTAG-Example]$ cat README.txt
Tested with openocd-0.6.0-dev-120229143915 and ARM-USB-TINY-H.
If you have a different debugger use different than the provided
.cfg for debugger.

Remember that you need libUSB drivers for our programmers to work
with openOCD, not FTDI ones.

Check the following links for guides:

https://www.olimex.com/dev/soft/arm/JTAG/Manual_TELNET.pdf

https://www.olimex.com/dev/soft/arm/JTAG/Manual_PROGRAMMER.pdf

https://www.olimex.com/dev/arm-usb-tiny-h.html

July 2012
Lub/OLIMEX[chris@thinkpad SJTAG-Example]$ cat olimex-arm-usb-tiny-h.cfg
#
# Olimex ARM-USB-TINY-H
#
# http://www.olimex.com/dev/arm-usb-tiny-h.html
#

interface ft2232
ft2232_device_desc "Olimex OpenOCD JTAG ARM-USB-TINY-H"
ft2232_layout olimex-jtag
ft2232_vid_pid 0x15ba 0x002a

[chris@thinkpad SJTAG-Example]$

I’ve also added a new udev rule, so that I can use both JTAG adapters in OpenOCD without root privileges:

[chris@thinkpad ~]$ lsusb
Bus 001 Device 007: ID 15ba:0004 Olimex Ltd. OpenOCD JTAG TINY
Bus 001 Device 008: ID 1366:0101 SEGGER J-Link ARM
[chris@thinkpad ~]$ sudo vim /etc/udev/rules.d/10-local.rules
[chris@thinkpad ~]$ cat /etc/udev/rules.d/10-local.rules
ATTR{idVendor}=="15ba", ATTR{idProduct}=="0004", GROUP="plugdev", MODE="0660" # Olimex Ltd. OpenOCD JTAG TINY
ATTR{idVendor}=="1366", ATTR{idProduct}=="0101", GROUP="plugdev", MODE="0660" # SEGGER J-Link ARM
[chris@thinkpad ~]$ sudo usermod -aG plugdev chris # replace chris with your username
[chris@thinkpad ~]$ sudo udevadm control --reload-rules
[chris@thinkpad ~]$ # maybe log out and log in again or reboot

You can also use the udev rules file that OpenOCD ships, like pointed out in the comments:

[chris@thinkpad ~]$ sudo cp /usr/share/openocd/contrib/openocd.udev /etc/udev/rules.d/60-openocd.rules

Now test if we can communicate with our debugger (hardware is not connected):

[chris@thinkpad SJTAG-Example]$ openocd -f interface/jlink.cfg -f imx233.cfg
Open On-Chip Debugger 0.8.0-dev-00121-ga4d3446 (2013-08-24-17:09)
Licensed under GNU GPL v2
For bug reports, read
        http://openocd.sourceforge.net/doc/doxygen/bugs.html
Info : only one transport option; autoselect 'jtag'
trst_and_srst srst_pulls_trst srst_gates_jtag trst_push_pull srst_open_drain connect_deassert_srst
adapter speed: 800 kHz
dcc downloads are enabled
fast memory access is enabled
Info : J-Link initialization started / target CPU reset initiated
Info : J-Link ARM V6 compiled Feb  1 2011 14:28:14
Info : J-Link caps 0x99ff7bbf
Info : J-Link hw version 60000
Info : J-Link hw type J-Link
Info : J-Link max mem block 8864
Info : J-Link configuration
Info : USB-Address: 0xff
Info : Kickstart power on JTAG-pin 19: 0xffffff01
Info : Vref = 0.0 TCK = 1 TDI = 1 TDO = 1 TMS = 1 SRST = 1 TRST = 1
Error: Vref too low. Check Target Power
Info : J-Link JTAG Interface ready
Error: jlink_usb_message failed with result=5)
Error: jlink_tap_execute, wrong result -107 (expected 1)
Info : clock speed 800 kHz
Error: jlink_usb_message failed with result=5)
Error: jlink_tap_execute, wrong result -107 (expected 1)
Error: jlink_usb_message failed with result=5)
Error: jlink_tap_execute, wrong result -107 (expected 1)
in procedure 'transport'
in procedure 'init'
[chris@thinkpad SJTAG-Example]$

[chris@thinkpad SJTAG-Example]$ openocd -f /usr/share/openocd/scripts/interface/olimex-jtag-tiny.cfg -f imx233.cfg
Open On-Chip Debugger 0.8.0-dev-00121-ga4d3446 (2013-08-24-17:09)
Licensed under GNU GPL v2
For bug reports, read
        http://openocd.sourceforge.net/doc/doxygen/bugs.html
Info : only one transport option; autoselect 'jtag'
trst_and_srst srst_pulls_trst srst_gates_jtag trst_push_pull srst_open_drain connect_deassert_srst
adapter speed: 800 kHz
dcc downloads are enabled
fast memory access is enabled
Info : clock speed 750 kHz
Error: JTAG scan chain interrogation failed: all ones
Error: Check JTAG interface, timings, target power, etc.
Error: Trying to use configured scan chain anyway...
Error: imx23.cpu: IR capture error; saw 0x0f not 0x01
Warn : Bypassing JTAG setup events due to errors
Info : Embedded ICE version 15
Error: unknown EmbeddedICE version (comms ctrl: 0xffffffff)
Info : imx23.cpu: hardware has 2 breakpoint/watchpoint units
Warn : WARNING: unknown debug reason: 0xf
Warn : ThumbEE -- incomplete support

^C
[chris@thinkpad SJTAG-Example]$

Then I removed the SD card from my OLinuXino, connected the power adapter and a USB serial cable so that it prints 0x8020a014 at 115200 8N1 on the serial line (which means ERROR_DDI_SD_MMC_DEVICE_NOT_SUPPORTED, http://sasamy.narod.ru/IMX23_ROM_Error_Codes.pdf) because no SD card is found. Then I’ve connected the J-Link, the iMX233-SJTAG and the OLinuXino like you can see on the first image of this blog post.

Lets try to connect OpenOCD again:

[chris@thinkpad SJTAG-Example]$ openocd -f interface/jlink.cfg -f imx233.cfg
Open On-Chip Debugger 0.8.0-dev-00121-ga4d3446 (2013-08-24-17:09)
Licensed under GNU GPL v2
For bug reports, read
        http://openocd.sourceforge.net/doc/doxygen/bugs.html
Info : only one transport option; autoselect 'jtag'
trst_and_srst srst_pulls_trst srst_gates_jtag trst_push_pull srst_open_drain connect_deassert_srst
adapter speed: 800 kHz
dcc downloads are enabled
fast memory access is enabled
Info : J-Link initialization started / target CPU reset initiated
Info : J-Link ARM V6 compiled Feb  1 2011 14:28:14
Info : J-Link caps 0x99ff7bbf
Info : J-Link hw version 60000
Info : J-Link hw type J-Link
Info : J-Link max mem block 8864
Info : J-Link configuration
Info : USB-Address: 0xff
Info : Kickstart power on JTAG-pin 19: 0xffffff01
Info : Vref = 3.313 TCK = 1 TDI = 0 TDO = 0 TMS = 0 SRST = 0 TRST = 0
Info : J-Link JTAG Interface ready
Info : clock speed 800 kHz
Error: JTAG scan chain interrogation failed: all zeroes
Error: Check JTAG interface, timings, target power, etc.
Error: Trying to use configured scan chain anyway...
Error: imx23.cpu: IR capture error; saw 0x00 not 0x01
Warn : Bypassing JTAG setup events due to errors
Info : Embedded ICE version 0
Error: unknown EmbeddedICE version (comms ctrl: 0x00000000)
Info : imx23.cpu: hardware has 2 breakpoint/watchpoint units

[chris@thinkpad SJTAG-Example]$ openocd -f /usr/share/openocd/scripts/interface/olimex-jtag-tiny.cfg -f imx233.cfg
Open On-Chip Debugger 0.8.0-dev-00121-ga4d3446 (2013-08-24-17:09)
Licensed under GNU GPL v2
For bug reports, read
        http://openocd.sourceforge.net/doc/doxygen/bugs.html
Info : only one transport option; autoselect 'jtag'
trst_and_srst srst_pulls_trst srst_gates_jtag trst_push_pull srst_open_drain connect_deassert_srst
adapter speed: 800 kHz
dcc downloads are enabled
fast memory access is enabled
Info : clock speed 750 kHz
Error: JTAG scan chain interrogation failed: all zeroes
Error: Check JTAG interface, timings, target power, etc.
Error: Trying to use configured scan chain anyway...
Error: imx23.cpu: IR capture error; saw 0x00 not 0x01
Warn : Bypassing JTAG setup events due to errors
Info : Embedded ICE version 0
Error: unknown EmbeddedICE version (comms ctrl: 0x00000000)
Info : imx23.cpu: hardware has 2 breakpoint/watchpoint units

So there are still some ugly errors...

If I connect the debugger to another iMX233 based device then the output looks like this:

[chris@thinkpad SJTAG-Example]$ openocd -f /usr/share/openocd/scripts/interface/olimex-jtag-tiny.cfg -f imx233.cfg
Open On-Chip Debugger 0.8.0-dev-00121-ga4d3446 (2013-08-24-17:09)
Licensed under GNU GPL v2
For bug reports, read
        http://openocd.sourceforge.net/doc/doxygen/bugs.html
Info : only one transport option; autoselect 'jtag'
trst_and_srst srst_pulls_trst srst_gates_jtag trst_push_pull srst_open_drain connect_deassert_srst
adapter speed: 800 kHz
dcc downloads are enabled
fast memory access is enabled
Info : clock speed 750 kHz
Info : JTAG tap: imx23.cpu tap/device found: 0x079264f3 (mfg: 0x279, part: 0x7926, ver: 0x0)
Info : Embedded ICE version 6
Info : imx23.cpu: hardware has 2 breakpoint/watchpoint units

So there is a problem with the OLinuXino.

It works if I power the iMX233-SJTAG via USB and first boot a kernel with imx-bootlets from the SD card, then press the yellow button on the iMX233-SJTAG (until the green LED1 starts blinking) and then start OpenOCD:

Debugging the iMX233-OLinuXino via SJTAG

[chris@thinkpad SJTAG-Example]$ openocd -f /usr/share/openocd/scripts/interface/olimex-jtag-tiny.cfg -f imx233.cfg
Open On-Chip Debugger 0.8.0-dev-00121-ga4d3446 (2013-08-24-17:09)
Licensed under GNU GPL v2
For bug reports, read
        http://openocd.sourceforge.net/doc/doxygen/bugs.html
Info : only one transport option; autoselect 'jtag'
trst_and_srst srst_pulls_trst srst_gates_jtag trst_push_pull srst_open_drain connect_deassert_srst
adapter speed: 800 kHz
dcc downloads are enabled
fast memory access is enabled
Info : clock speed 750 kHz
Info : JTAG tap: imx23.cpu tap/device found: 0x079264f3 (mfg: 0x279, part: 0x7926, ver: 0x0)
Info : Embedded ICE version 6
Info : imx23.cpu: hardware has 2 breakpoint/watchpoint units

This is probably because the SJTAG gets first initialized by the bootlets. If everything is OK then the green LED on the iMX233-SJTAG should blink really fast.

Note

To test this I’ve used a older 3.7.1 kernel (http://sourceforge.net/projects/janncc/files/olinuxino/kernel/3.7.1/) with bootlets that I’ve written over a newer Arch Linux ARM image which uses U-Boot by default.

[chris@thinkpad 3.7.1]$ sudo dd if=sd_mmc_bootstream.raw of=/dev/mmcblk0p1

To use SJTAG with U-Boot you need an additional patch like Rouming has written under: http://www.jann.cc/2013/02/07/u_boot_for_the_imx233_olinuxino.html. I will test this later.

With the J-Link I get the same errors as before and the serial tty hangs after I start OpenOCD whereas with the ARM-USB-TINY I can still access the serial tty of my OLinuXino. Maybe the J-Link resets the processor somehow and then the SJTAG interface is again uninitialized (just a guess) but this can be examined later since it works with the ARM-USB-TINY.

Now we should be able to connect via telnet (telnet localhost 4444) or with arm-none-eabi-gdb (target remote localhost:3333) to OpenOCD and load a simple program into memory.

Minimal example (blinking LED without OS)

You can find the announcement here: http://tech.groups.yahoo.com/group/olinuxino/message/546 and the original code here: http://sourceforge.net/p/lyre/code/90/tree/propendous-imx233_board/firmware/blink_led/.

I’ve modified it a little bit and packed it into a zip file: blink_led.zip

[chris@thinkpad SJTAG-Example]$ wget http://www.jann.cc/_downloads/blink_led.zip
[chris@thinkpad SJTAG-Example]$ unzip blink_led.zip
...
[chris@thinkpad SJTAG-Example]$ cd blink_led/
[chris@thinkpad blink_led]$ cat src/main.c

#include "regspinctrl.h"
#include "regsdigctl.h"

/* LED pin (gpio65) = bank 2, pin 1
* 65/32 = 2 remainder 1 -> bank 2, pin 1
* HW_PINCTRL_DOUT**bank**_SET(LED);
* #define LED (1 << pin)
*/
#define LED (1 << 1)


void delay_us (int us)
{
  volatile int start = HW_DIGCTL_MICROSECONDS_RD();

  while (HW_DIGCTL_MICROSECONDS_RD() - start < us)
      ;
}

int main(void)
{
    /* Enable parallel JTAG, 12mA drive and the MUX pins for it */
    HW_PINCTRL_MUXSEL4_CLR(0x3FFF);
    HW_PINCTRL_MUXSEL4_SET(0x2AAE);

    //HW_PINCTRL_DRIVE8_CLR(0x3FFF);
    //HW_PINCTRL_DRIVE8_SET(0x2AAE);

    /* Do not enable parallel JTAG, we are using serial JTAG (SJTAG) */
    //HW_DIGCTL_CTRL_CLR((1 << 1) | (1 << 3) | (1 << 6));
    /**************************************/

    /* Configue PIO for LED */
    HW_PINCTRL_DOUT2_SET(LED);
    HW_PINCTRL_DOE2_SET(LED);

    while(1)
    {
        HW_PINCTRL_DOUT2_SET(LED);
        delay_us(500000);
        HW_PINCTRL_DOUT2_CLR(LED);
        delay_us(500000);
    }

    return 0;
}

Building the blink example

First you can start with a clean up and remove old build files:

[chris@thinkpad blink_led]$ make clean
Cleaning...
Files:
rm -rf ./src/entry.o ./src/main.o ./output/blink_led.ld
Build output:
rm -rf ./output
rm -rf bootstream-factory.raw
[chris@thinkpad blink_led]$

Now compile the example and create a bootsream file for the SD card:

[chris@thinkpad blink_led]$ make all
mkdir -p ./output
arm-none-eabi-gcc -c -O0 -g -Wall -I./include -I/include -nostdinc -fno-builtin -D__ASSEMBLY__ -o src/entry.o src/entry.S
arm-none-eabi-gcc -c -O0 -g -Wall -I./include -I/include -nostdinc -fno-builtin -o src/main.o src/main.c
arm-none-eabi-cpp -P -DBASE_ADDR=0x00002000 -o output/blink_led.ld src/blink_led.ld.in
arm-none-eabi-ld -o output/blink_led.elf ./src/entry.o ./src/main.o -static -nostdlib -T ./output/blink_led.ld -L/home/chris/local/gcc-arm-none-eabi-4_7-2013q1/bin/../lib/gcc/arm-none-eabi/4.7.3/ -lgcc
arm-none-eabi-objcopy -R -S -O binary -R .note -R .note.gnu.build-id -R .comment output/blink_led.elf output/blink_led.bin
[chris@thinkpad blink_led]$ make sb
./elftosb2 -V -z -c blink_led.db \
            -o ./output/blink_led_sb.bin
Boot Section 0x00000000:
        LOAD | addr=0x00002000 | len=0x00000034 | crc=0x14a547ad
        LOAD | addr=0x00002040 | len=0x000000d0 | crc=0xdf4d8bbb
        JUMP | addr=0x00002000 | arg=0x00000000
[chris@thinkpad blink_led]$ ./make_boot_image.sh
4+0 records in
4+0 records out
2048 bytes (2.0 kB) copied, 0.000238814 s, 8.6 MB/s
1+0 records in
1+0 records out
512 bytes (512 B) copied, 0.000215001 s, 2.4 MB/s
[chris@thinkpad blink_led]$

Preparing the SD card

To run the blink example we need to create a special partition on the SD card. If you have used the SD card previously to boot Linux (written a Linux image to the SD card) then you should already have this boot partition else you can create it with fdisk.

[chris@thinkpad blink_led]$ # list the current partitions of the SD card
[chris@thinkpad blink_led]$ sudo fdisk -l /dev/mmcblk0
Swipe your right index finger across the fingerprint reader

Disk /dev/mmcblk0: 3980 MB, 3980394496 bytes, 7774208 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xdcc08fa1

        Device Boot      Start         End      Blocks   Id  System
/dev/mmcblk0p1            2048       67583       32768   53  OnTrack DM6 Aux3
/dev/mmcblk0p2           67584     3164159     1548288   83  Linux
/dev/mmcblk0p3         3164160     3854335      345088   82  Linux swap / Solaris
[chris@thinkpad blink_led]$ # to start from scratch I will delete all partitions
[chris@thinkpad blink_led]$ # and then create only the "boot" partition
[chris@thinkpad blink_led]$ sudo fdisk /dev/mmcblk0
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): o
Building a new DOS disklabel with disk identifier 0x59f6b520.

Command (m for help): p

Disk /dev/mmcblk0: 3980 MB, 3980394496 bytes, 7774208 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x59f6b520

        Device Boot      Start         End      Blocks   Id  System

Command (m for help): n
Partition type:
  p   primary (0 primary, 0 extended, 4 free)
  e   extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-7774207, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-7774207, default 7774207): +16M
Partition 1 of type Linux and of size 16 MiB is set

Command (m for help): t
Selected partition 1
Hex code (type L to list all codes): 53
Changed type of partition 'Linux' to 'OnTrack DM6 Aux3'

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[chris@thinkpad blink_led]$ sudo fdisk -l /dev/mmcblk0

Disk /dev/mmcblk0: 3980 MB, 3980394496 bytes, 7774208 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x59f6b520

        Device Boot      Start         End      Blocks   Id  System
/dev/mmcblk0p1            2048       34815       16384   53  OnTrack DM6 Aux3
[chris@thinkpad blink_led]$

Now we can write the blink_led example program to the SD card:

[chris@thinkpad blink_led]$ sudo dd if=bootstream-factory.raw of=/dev/mmcblk0p1
5+0 records in
5+0 records out
2560 bytes (2.6 kB) copied, 0.0134655 s, 190 kB/s
[chris@thinkpad blink_led]$

Now remove the SD card, plug it into your OLinuXino and power it up. You should see a fast blinking LED.

Debugging the example

Connect the debugger to your OLinuXino, press the yellow button on the iMX233-SJTAG until its LED1 goes off or starts blinking too and then start OpenOCD:

[chris@thinkpad blink_led]$ openocd -f /usr/share/openocd/scripts/interface/olimex-jtag-tiny.cfg -f imx233.cfg
Open On-Chip Debugger 0.8.0-dev-00121-ga4d3446 (2013-08-24-17:09)
Licensed under GNU GPL v2
For bug reports, read
        http://openocd.sourceforge.net/doc/doxygen/bugs.html
Info : only one transport option; autoselect 'jtag'
trst_and_srst srst_pulls_trst srst_gates_jtag trst_push_pull srst_open_drain connect_deassert_srst
adapter speed: 800 kHz
dcc downloads are enabled
fast memory access is enabled
Info : clock speed 750 kHz
Info : JTAG tap: imx23.cpu tap/device found: 0x079264f3 (mfg: 0x279, part: 0x7926, ver: 0x0)
Info : Embedded ICE version 6
Info : imx23.cpu: hardware has 2 breakpoint/watchpoint units
Info : accepting 'gdb' connection from 3333
...

In another console start GDB (arm-none-eabi-gdb):

[chris@thinkpad blink_led]$ arm-none-eabi-gdb output/blink_led.elf
GNU gdb (GNU Tools for ARM Embedded Processors) 7.4.1.20130312-cvs
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=i686-linux-gnu --target=arm-none-eabi".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/chris/local/OLinuXino/Debugging/SJTAG-Example/blink_led/output/blink_led.elf...done.
(gdb) target extended-remote :3333
Remote debugging using :3333
0x00000000 in ?? ()
(gdb) monitor reset halt
JTAG tap: imx23.cpu tap/device found: 0x079264f3 (mfg: 0x279, part: 0x7926, ver: 0x0)
srst pulls trst - can not reset into halted mode. Issuing halt after reset.
target state: halted
target halted in ARM state due to debug-request, current mode: Supervisor
cpsr: 0x800000d3 pc: 0x00002068
MMU: disabled, D-Cache: disabled, I-Cache: disabled
(gdb) load
Loading section .start, size 0x34 lma 0x2000
Loading section .text, size 0xd0 lma 0x2040
Start address 0x2000, load size 260
Transfer rate: 42 KB/sec, 130 bytes/write.
(gdb) monitor reset init
JTAG tap: imx23.cpu tap/device found: 0x079264f3 (mfg: 0x279, part: 0x7926, ver: 0x0)
srst pulls trst - can not reset into halted mode. Issuing halt after reset.
target state: halted
target halted in ARM state due to debug-request, current mode: Supervisor
cpsr: 0x800000d3 pc: 0x00002064
MMU: disabled, D-Cache: disabled, I-Cache: disabled
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /home/chris/local/OLinuXino/Debugging/SJTAG-Example/blink_led/output/blink_led.elf
^C0x00002074 in delay_us (us=100000) at src/main.c:16
16         while (HW_DIGCTL_MICROSECONDS_RD() - start < us)
(gdb) break main.c:39
Breakpoint 1 at 0x20c4: file src/main.c, line 39.
(gdb) c
Continuing.

Breakpoint 1, main () at src/main.c:39
39              HW_PINCTRL_DOUT2_SET(LED);
(gdb) c
Continuing.

Breakpoint 1, main () at src/main.c:39
39              HW_PINCTRL_DOUT2_SET(LED);
(gdb) c
Continuing.

Breakpoint 1, main () at src/main.c:39
39              HW_PINCTRL_DOUT2_SET(LED);
(gdb) quit
A debugging session is active.

        Inferior 1 [Remote target] will be killed.

Quit anyway? (y or n) y
[chris@thinkpad blink_led]$

So manually debugging with GDB works, I’ve set a breakpoint in main.c at line 39 and every time the program hits this point the LED stops blinking and the program pauses until I enter c to continue.

Now we can try to automate this a little bit and use some GDB GUI.

Debugging the example with Nemiver

Nemiver is a standalone graphical debugger from the GNOME desktop environment.

In Fedora you can install it via:

[chris@thinkpad project0]$ sudo yum install nemiver

If OpenOCD is already running you can simply start Nemiver with the following command:

[chris@thinkpad blink_led]$ arm-none-eabi-gdb --batch --command=init.gdb output/blink_led.elf
0x00002060 in delay_us (us=100000) at src/main.c:16
16         while (HW_DIGCTL_MICROSECONDS_RD() - start < us)
JTAG tap: imx23.cpu tap/device found: 0x079264f3 (mfg: 0x279, part: 0x7926, ver: 0x0)
srst pulls trst - can not reset into halted mode. Issuing halt after reset.
target state: halted
target halted in ARM state due to debug-request, current mode: Supervisor
cpsr: 0x800000d3 pc: 0x00002074
MMU: disabled, D-Cache: disabled, I-Cache: disabled
Loading section .start, size 0x34 lma 0x2000
Loading section .text, size 0xd0 lma 0x2040
Start address 0x2000, load size 260
Transfer rate: 50 KB/sec, 130 bytes/write.
Breakpoint 1 at 0x2094: file src/main.c, line 23.

Breakpoint 1, main () at src/main.c:23
23          HW_PINCTRL_MUXSEL4_CLR(0x3FFF);
[chris@thinkpad blink_led]$ nemiver --remote=localhost:3333 --gdb-binary="$(which arm-none-eabi-gdb)" output/blink_led.elf

Sometimes you have to click onto “step into” in Nemiver until the “Continue or start” button works.

To start OpenOCD, GDB and Nemiver just run the debug.sh script:

[chris@thinkpad blink_led]$ ./debug.sh
0x00002324 in ?? ()
JTAG tap: imx23.cpu tap/device found: 0x079264f3 (mfg: 0x279, part: 0x7926, ver: 0x0)
srst pulls trst - can not reset into halted mode. Issuing halt after reset.
target state: halted
target halted in ARM state due to debug-request, current mode: Supervisor
cpsr: 0x800000d3 pc: 0x00002064
MMU: disabled, D-Cache: disabled, I-Cache: disabled
Loading section .start, size 0x34 lma 0x2000
Loading section .text, size 0xd0 lma 0x2040
Start address 0x2000, load size 260
Transfer rate: 42 KB/sec, 130 bytes/write.
Breakpoint 1 at 0x2094: file src/main.c, line 23.

Breakpoint 1, main () at src/main.c:23
23          HW_PINCTRL_MUXSEL4_CLR(0x3FFF);
[chris@thinkpad blink_led]$ cat debug.sh

#!/bin/bash

# start xterm with openocd in the background
xterm -e openocd -f /usr/share/openocd/scripts/interface/olimex-jtag-tiny.cfg -f imx233.cfg &

# save the PID of the background process
XTERM_PID=$!

# wait a bit to be sure the hardware is ready
sleep 2

# execute some initialisation commands via gdb
arm-none-eabi-gdb --batch --command=init.gdb output/blink_led.elf

# start the gdb gui
nemiver --remote=localhost:3333 --gdb-binary="$(which arm-none-eabi-gdb)" output/blink_led.elf

# close xterm when the user has exited nemiver
kill $XTERM_PID

Debugging the example with Eclipse

Eclipse seems to work too but I don’t like it that much because its slow and fat.

Debugging with Eclipse

Load a modified program into memory

For example edit src/main.c and change the blink speed from 100000 to 900000:

[chris@thinkpad blink_led]$ grep -i delay_us src/main.c
void delay_us (int us)
        delay_us(100000);
        delay_us(100000);
[chris@thinkpad blink_led]$ vim src/main.c
[chris@thinkpad blink_led]$ grep -i delay_us src/main.c
void delay_us (int us)
        delay_us(900000);
        delay_us(900000);
[chris@thinkpad blink_led]$ make all
mkdir -p ./output
arm-none-eabi-gcc -c -O0 -g -Wall -I./include -I/include -nostdinc -fno-builtin -o src/main.o src/main.c
arm-none-eabi-ld -o output/blink_led.elf ./src/entry.o ./src/main.o -static -nostdlib -T ./output/blink_led.ld -L/home/chris/local/gcc-arm-none-eabi-4_7-2013q1/bin/../lib/gcc/arm-none-eabi/4.7.3/ -lgcc
arm-none-eabi-objcopy -R -S -O binary -R .note -R .note.gnu.build-id -R .comment output/blink_led.elf output/blink_led.bin
[chris@thinkpad blink_led]$

Now change to your console with GDB, stop the running program and load the modified one:

(gdb) monitor halt
target state: halted
target halted in ARM state due to debug-request, current mode: Supervisor
cpsr: 0x800000d3 pc: 0x0000206c
MMU: disabled, D-Cache: disabled, I-Cache: disabled
(gdb) load output/blink_led.elf
`/home/chris/local/OLinuXino/Debugging/SJTAG-Example/blink_led/output/blink_led.elf' has changed; re-reading symbols.
Loading section .start, size 0x34 lma 0x2000
Loading section .text, size 0xd0 lma 0x2040
Start address 0x2000, load size 260
Transfer rate: 42 KB/sec, 130 bytes/write.
(gdb) monitor resume 0x2000
(gdb)

Without reading the documentation I’ve chosen monitor resume 0x2000 because it says earlier Loading section .start, size 0x34 lma 0x2000. We will see how this works with U-Boot or the Linux kernel.

As a result I have now a slower blinking LED.

Debugging the Linux kernel via SJTAG

So now that we have mastered to debug the simple blink_led example we can try to debug the Linux Kernel.

First we need to build a kernel with debugging information, to do this you can follow this quide and when configuring the kernel make sure that the following is selected:

Kernel hacking  --->
  [*] Compile the kernel with debug info

On newer kernels:

Kernel hacking  --->
  Compile-time checks and compiler options  --->
    [*] Compile the kernel with debug info

I’m using the same SD card as before with only one partition and no root filesysten and copied the kernel to the first partition:

[chris@thinkpad imx-bootlets-src-10.05.02]$ sudo dd if=sd_mmc_bootstream.raw of=/dev/mmcblk0p1
Swipe your right index finger across the fingerprint reader
8433+0 records in
8433+0 records out
4317696 bytes (4.3 MB) copied, 1.49275 s, 2.9 MB/s
[chris@thinkpad imx-bootlets-src-10.05.02]$

That’s why the following kernel panic is not surprising:

[    1.700000] stmp3xxx-rtc 8005c000.rtc: setting system clock to 1970-01-01 00:00:05 UTC (5)
[    1.730000]  mmcblk0: p1
[    1.760000] ALSA device list:
[    1.760000]   No soundcards found.
[    1.770000] VFS: Cannot open root device "mmcblk0p2" or unknown-block(179,2): error -6
[    1.780000] Please append a correct "root=" boot option; here are the available partitions:
[    1.790000] b300         3887104 mmcblk0  driver: mmcblk
[    1.790000]   b301           16384 mmcblk0p1 59f6b520-01
[    1.800000] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(179,2)
[    1.800000] [<c0014054>] (unwind_backtrace+0x0/0xf0) from [<c059e850>] (panic+0x98/0x1ec)
[    1.800000] [<c059e850>] (panic+0x98/0x1ec) from [<c0800c3c>] (mount_block_root+0x1a0/0x24c)
[    1.800000] [<c0800c3c>] (mount_block_root+0x1a0/0x24c) from [<c0800dcc>] (mount_root+0xe4/0x10c)
[    1.800000] [<c0800dcc>] (mount_root+0xe4/0x10c) from [<c0800f18>] (prepare_namespace+0x124/0x184)
[    1.800000] [<c0800f18>] (prepare_namespace+0x124/0x184) from [<c08008c4>] (kernel_init_freeable+0x170/0x1b4)
[    1.800000] [<c08008c4>] (kernel_init_freeable+0x170/0x1b4) from [<c059d22c>] (kernel_init+0x8/0xe4)
[    1.800000] [<c059d22c>] (kernel_init+0x8/0xe4) from [<c000ee28>] (ret_from_fork+0x14/0x2c)

Nevertheless we can try to debug it, get a full backtrace and step through the code until the panic occurs.

Update: I was not fast enough setting the breakpoints that’s why I start experimenting with a working kernel.

As before start OpenOCD:

[chris@thinkpad blink_led]$ openocd -f /usr/share/openocd/scripts/interface/olimex-jtag-tiny.cfg -f imx233.cfg
Open On-Chip Debugger 0.8.0-dev-00121-ga4d3446 (2013-08-24-17:09)
Licensed under GNU GPL v2
For bug reports, read
        http://openocd.sourceforge.net/doc/doxygen/bugs.html
Info : only one transport option; autoselect 'jtag'
trst_and_srst srst_pulls_trst srst_gates_jtag trst_push_pull srst_open_drain connect_deassert_srst
adapter speed: 800 kHz
dcc downloads are enabled
fast memory access is enabled
Info : clock speed 750 kHz
Info : JTAG tap: imx23.cpu tap/device found: 0x079264f3 (mfg: 0x279, part: 0x7926, ver: 0x0)
Info : Embedded ICE version 6
Info : imx23.cpu: hardware has 2 breakpoint/watchpoint units

Now switch to the kernel directory and start GDB:

[chris@thinkpad imx-bootlets-src-10.05.02]$ cd ../../kernel/linux-stable/
[chris@thinkpad linux-stable]$ arm-none-eabi-gdb vmlinux
GNU gdb (GNU Tools for ARM Embedded Processors) 7.4.1.20130312-cvs
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=i686-linux-gnu --target=arm-none-eabi".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/chris/local/OLinuXino/imx23-olinuxino/kernel/linux-stable/vmlinux...done.
(gdb) target remote :3333
Remote debugging using :3333
0x00000000 in ?? ()
(gdb) monitor halt
target state: halted
target halted in ARM state due to debug-request, current mode: Supervisor
cpsr: 0x60000093 pc: 0xc005d310
MMU: enabled, D-Cache: enabled, I-Cache: enabled
(gdb) hbreak heartbeat_trig_activate
Hardware assisted breakpoint 1 at 0xc042ee54: file drivers/leds/ledtrig-heartbeat.c, line 82.
(gdb) hbreak sys_sync
Hardware assisted breakpoint 2 at 0xc00fe528: file fs/sync.c, line 103.
(gdb) c
Continuing.

Now you should be able to hit a breakpoint by either running sync or echo heartbeat > /sys/class/leds/green/trigger on the serial console of your OLinuXino.

Have a look at the heartbeat_trig_activate function from kernel/linux-stable/drivers/leds/ledtrig-heartbeat.c and then compare it with the GDB output when stepping through the code:

static void heartbeat_trig_activate(struct led_classdev *led_cdev)
{
  struct heartbeat_trig_data *heartbeat_data;

  heartbeat_data = kzalloc(sizeof(*heartbeat_data), GFP_KERNEL);
  if (!heartbeat_data)
    return;

  led_cdev->trigger_data = heartbeat_data;
  setup_timer(&heartbeat_data->timer,
        led_heartbeat_function, (unsigned long) led_cdev);
  heartbeat_data->phase = 0;
  led_heartbeat_function(heartbeat_data->timer.data);
  led_cdev->activated = true;
}

Now continue the execution:

(gdb) c
Continuing.

Breakpoint 1, heartbeat_trig_activate (led_cdev=0xc3a7a814) at drivers/leds/ledtrig-heartbeat.c:82
82      {
(gdb) bt
#0  heartbeat_trig_activate (led_cdev=0xc3a7a814) at drivers/leds/ledtrig-heartbeat.c:82
#1  0xc042d75c in led_trigger_set (led_cdev=led_cdev@entry=0xc3a7a814, trig=trig@entry=0xc087296c) at drivers/leds/led-triggers.c:131
#2  0xc042db28 in led_trigger_store (dev=<optimized out>, attr=<optimized out>, buf=<optimized out>, count=10) at drivers/leds/led-triggers.c:58
#3  0xc02d1740 in dev_attr_store (kobj=<optimized out>, attr=<optimized out>, buf=<optimized out>, count=<optimized out>) at drivers/base/core.c:114
#4  0xc012f868 in flush_write_buffer (count=10, buffer=0xc2958d80, dentry=<optimized out>) at fs/sysfs/file.c:202
#5  sysfs_write_file (file=<optimized out>, buf=<optimized out>, count=<optimized out>, ppos=0xc293ff88) at fs/sysfs/file.c:236
#6  0xc00d486c in vfs_write (file=file@entry=0xc29f6b60, buf=0xb6f60000 "heartbeat\n", count=10, count@entry=0, pos=pos@entry=0xc293ff88) at fs/read_write.c:463
#7  0xc00d4cd8 in sys_write (fd=<optimized out>, buf=<optimized out>, count=0) at fs/read_write.c:510
#8  0xc000ed80 in kuser_cmpxchg32_fixup ()
#9  0xc000ed80 in kuser_cmpxchg32_fixup ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) step
85              heartbeat_data = kzalloc(sizeof(*heartbeat_data), GFP_KERNEL);
(gdb) step
kzalloc (flags=208, size=76) at drivers/leds/ledtrig-heartbeat.c:85
85              heartbeat_data = kzalloc(sizeof(*heartbeat_data), GFP_KERNEL);
(gdb) step
kmalloc (flags=32976, size=76) at drivers/leds/ledtrig-heartbeat.c:85
85              heartbeat_data = kzalloc(sizeof(*heartbeat_data), GFP_KERNEL);
(gdb) hbreak ledtrig-heartbeat.c:94
Hardware assisted breakpoint 3 at 0xc042eeb4: file drivers/leds/ledtrig-heartbeat.c, line 94.
(gdb) c
Continuing.
Warning:
Cannot insert hardware breakpoint 3.
Could not insert hardware breakpoints:
You may have requested too many hardware breakpoints/watchpoints.

(gdb) delete breakpoints 3
(gdb) delete breakpoints 2
(gdb) hbreak ledtrig-heartbeat.c:94
Hardware assisted breakpoint 4 at 0xc042eeb4: file drivers/leds/ledtrig-heartbeat.c, line 94.
(gdb) c
Continuing.

Breakpoint 4, heartbeat_trig_activate (led_cdev=0xc3a7a814) at drivers/leds/ledtrig-heartbeat.c:94
94              led_cdev->activated = true;
(gdb) delete breakpoints
Delete all breakpoints? (y or n) y
(gdb) hbreak sys_sync
Hardware assisted breakpoint 5 at 0xc00fe528: file fs/sync.c, line 103.
(gdb) c
Continuing.

Breakpoint 5, sys_sync () at fs/sync.c:103
103     {
(gdb) c
Continuing.

Software breakpoints seems to work too:

(gdb) c
Continuing.

^C
Program received signal SIGINT, Interrupt.
cpu_arm926_do_idle () at arch/arm/mm/proc-arm926.S:112
112             mcr     p15, 0, r1, c1, c0, 0           @ Restore ICache enable
(gdb) delete breakpoints
Delete all breakpoints? (y or n) y
(gdb) break sys_sync
Breakpoint 6 at 0xc00fe528: file fs/sync.c, line 103.
(gdb) c
Continuing.

Breakpoint 6, sys_sync () at fs/sync.c:103
103     {
(gdb) info breakpoints
Num     Type           Disp Enb Address    What
6       breakpoint     keep y   0xc00fe528 in sys_sync at fs/sync.c:103
        breakpoint already hit 1 time
(gdb) list
103     {
104             int nowait = 0, wait = 1;
105
106             wakeup_flusher_threads(0, WB_REASON_SYNC);
107             iterate_supers(sync_inodes_one_sb, NULL);
108             iterate_supers(sync_fs_one_sb, &nowait);
109             iterate_supers(sync_fs_one_sb, &wait);
110             iterate_bdevs(fdatawrite_one_bdev, NULL);
111             iterate_bdevs(fdatawait_one_bdev, NULL);
112             if (unlikely(laptop_mode))
(gdb)

Here are again the commands to trigger the breakpoints:

[root@olinuxino ~]# echo heartbeat >  /sys/class/leds/green/trigger
[root@olinuxino ~]#
[root@olinuxino ~]# sync

If you have any problems with setting the breakpoints you can also start OpenOCD in debug mode and watch for errors:

[chris@thinkpad blink_led]$ openocd -f /usr/share/openocd/scripts/interface/olimex-jtag-tiny.cfg -f imx233.cfg --debug=3

Debugging early kernel startup

I haven’t found a way to set a beakpoint at start_kernel() like it is possible with U-Boot so I’ve modified linux-stable/init/main.c and added a while loop at the beginning:

asmlinkage void __init start_kernel(void)
{
  char * command_line;
  extern const struct kernel_param __start___param[], __stop___param[];

  volatile char wait_for_debugger;
  wait_for_debugger = 1;
  while(wait_for_debugger);

  /*
  * Need to run as early as possible, to initialize the
  * lockdep hash:
  */
  lockdep_init();
  smp_setup_processor_id();
  debug_objects_early_init();

The start_kernel() function seems very interesting because a lot of kernel initialization happens here and we can step line-by-line through the code and see where the kernel hangs during boot.

If everything works as expected you will see something like:

HTLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLFC
PowerPrep start initialize power...
Battery Voltage = 0.68V
No battery or bad battery                                       detected!!!.Disabling battery                                   voltage measurements./7
EMI_CTRL 0x1C084040
FRAC 0x92926192
init_ddr_mt46v32m16_133Mhz
power 0x00820710
Frac 0x92926192
start change cpu freq
hbus 0x00000003
cpu 0x00010001
LLLLLLLFCLJUncompressing Linux... done, booting the kernel.

on the serial line and you can connect with GDB and step out of the while loop:

[chris@thinkpad linux-stable]$ arm-none-eabi-gdb vmlinux
GNU gdb (GNU Tools for ARM Embedded Processors) 7.4.1.20130312-cvs
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=i686-linux-gnu --target=arm-none-eabi".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/chris/local/OLinuXino/imx23-olinuxino/kernel/linux-stable/vmlinux...done.
(gdb) target remote :3333
Remote debugging using :3333
0x00000000 in ?? ()
(gdb) c
Continuing.
WARNING! The target is already running. All changes GDB did to registers will be discarded! Waiting for target to halt.
^C
Program received signal SIGINT, Interrupt.
start_kernel () at init/main.c:477
477       while(wait_for_debugger);
(gdb) list
472             char * command_line;
473             extern const struct kernel_param __start___param[], __stop___param[];
474
475       volatile char wait_for_debugger;
476       wait_for_debugger = 1;
477       while(wait_for_debugger);
478
479             /*
480              * Need to run as early as possible, to initialize the
481              * lockdep hash:
(gdb) list
482              */
483             lockdep_init();
484             smp_setup_processor_id();
485             debug_objects_early_init();
486
487             /*
488              * Set up the the initial canary ASAP:
489              */
490             boot_init_stack_canary();
491
(gdb) break init/main.c:483
Breakpoint 1 at 0xc08004cc: file init/main.c, line 483.
(gdb) set wait_for_debugger=0
(gdb) c
Continuing.
^C
Program received signal SIGINT, Interrupt.
start_kernel () at init/main.c:477
477       while(wait_for_debugger);
(gdb) whatis wait_for_debugger
type = volatile char
(gdb) set var wait_for_debugger=0
(gdb) c
Continuing.
^C
Program received signal SIGINT, Interrupt.
0xc08004c4 in start_kernel () at init/main.c:477
477       while(wait_for_debugger);
(gdb) p wait_for_debugger
$1 = 1 '\001'
(gdb) p &wait_for_debugger
Address requested for identifier "wait_for_debugger" which is in register $r0
(gdb) set $r0=0
(gdb) c
Continuing.

Breakpoint 1, start_kernel () at init/main.c:483
483             lockdep_init();
(gdb)

Now execute one line after each other and see what happens:

(gdb) n
484             smp_setup_processor_id();
(gdb) n
492             cgroup_init_early();
(gdb) n
494             local_irq_disable();
(gdb) n
495             early_boot_irqs_disabled = true;
(gdb) n
494             local_irq_disable();
(gdb) c
Continuing.

Debugging the kernel panic from the beginning

If we search within the kernel source for Cannot open root device we will find it in linux-stable/init/do_mounts.c within the function mount_block_root() so we can set a breakpoint there and step through the code until the panic happens:

[chris@thinkpad linux-stable]$ arm-none-eabi-gdb vmlinux
GNU gdb (GNU Tools for ARM Embedded Processors) 7.4.1.20130312-cvs
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=i686-linux-gnu --target=arm-none-eabi".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/chris/local/OLinuXino/imx23-olinuxino/kernel/linux-stable/vmlinux...done.
(gdb) target remote :3333
Remote debugging using :3333
0x00000000 in ?? ()
(gdb) c
Continuing.
WARNING! The target is already running. All changes GDB did to registers will be discarded! Waiting for target to halt.
^C
Program received signal SIGINT, Interrupt.
start_kernel () at init/main.c:477
477       while(wait_for_debugger);
(gdb) break mount_block_root
Breakpoint 1 at 0xc0800ab0: file init/do_mounts.c, line 374.
(gdb) set var wait_for_debugger=0
(gdb) set $r0=0
(gdb) c
Continuing.
^C
Program received signal SIGINT, Interrupt.
0xc08004c4 in start_kernel () at init/main.c:477
477       while(wait_for_debugger);
(gdb) set $r0=0
(gdb) c
Continuing.

Breakpoint 1, mount_block_root (name=name@entry=0xc071e6fc "/dev/root", flags=32768) at init/do_mounts.c:374
374     {
(gdb)

Now the kernel startup stops at:

[    1.780000] can: broadcast manager protocol (rev 20120528 t)
[    1.780000] can: netlink gateway (rev 20130117) max_hops=1
[    1.800000] Key type dns_resolver registered
[    1.800000] registered taskstats version 1
[    1.810000]  mmcblk0: p1
[    1.820000] stmp3xxx-rtc 8005c000.rtc: setting system clock to 1970-01-01 00:04:46 UTC (286)
[    1.860000] ALSA device list:
[    1.860000]   No soundcards found.

And if we continue we will trigger the panic:

(gdb) n
375             struct page *page = alloc_page(GFP_KERNEL |
(gdb) n
374     {
(gdb) n
375             struct page *page = alloc_page(GFP_KERNEL |
(gdb) n
377             char *fs_names = page_address(page);
(gdb) n
385             get_fs_names(fs_names);
(gdb) n
377             char *fs_names = page_address(page);
(gdb) n
385             get_fs_names(fs_names);
(gdb) n
377             char *fs_names = page_address(page);
(gdb) n
385             get_fs_names(fs_names);
(gdb) n
375             struct page *page = alloc_page(GFP_KERNEL |
(gdb) n
377             char *fs_names = page_address(page);
(gdb) n
385             get_fs_names(fs_names);
(gdb) n
387             for (p = fs_names; *p; p += strlen(p)+1) {
(gdb) n
388                     int err = do_mount_root(name, p, flags, root_mount_data);
(gdb) n
389                     switch (err) {
(gdb) n
404                     __bdevname(ROOT_DEV, b);
(gdb) n
406                     printk("VFS: Cannot open root device \"%s\" or %s: error %d\n",
(gdb) n
408                     printk("Please append a correct \"root=\" boot option; here are the available partitions:\n");
(gdb) n
410                     printk_all_partitions();
(gdb) n
415                     panic("VFS: Unable to mount root fs on %s", b);
(gdb) n
^C
Program received signal SIGINT, Interrupt.
__spin_lock_debug (lock=0xc0835500) at lib/spinlock_debug.c:114
114                     __delay(1);
(gdb) c
Continuing.
^C
Program received signal SIGINT, Interrupt.
__spin_lock_debug (lock=0xc0835500) at lib/spinlock_debug.c:111
111             for (i = 0; i < loops; i++) {
(gdb)

[   25.080000] VFS: Cannot open root device "mmcblk0p2" or unknown-block(179,2): error -6
[   25.110000] Please append a correct "root=" boot option; here are the available partitions:
[   27.060000] b300         3887104 mmcblk0  driver: mmcblk
[   27.060000]   b301           30720 mmcblk0p1 df299e2d-01
[   28.640000] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(179,2)
[   28.640000] [<c0014054>] (unwind_backtrace+0x0/0xf0) from [<c059e850>] (panic+0x98/0x1ec)
[   28.640000] [<c059e850>] (panic+0x98/0x1ec) from [<c0800c50>] (mount_block_root+0x1a0/0x24c)
[   28.640000] [<c0800c50>] (mount_block_root+0x1a0/0x24c) from [<c0800de0>] (mount_root+0xe4/0x10c)
[   28.640000] [<c0800de0>] (mount_root+0xe4/0x10c) from [<c0800f2c>] (prepare_namespace+0x124/0x184)
[   28.640000] [<c0800f2c>] (prepare_namespace+0x124/0x184) from [<c08008d8>] (kernel_init_freeable+0x170/0x1b4)
[   28.640000] [<c08008d8>] (kernel_init_freeable+0x170/0x1b4) from [<c059d22c>] (kernel_init+0x8/0xe4)
[   28.640000] [<c059d22c>] (kernel_init+0x8/0xe4) from [<c000ee28>] (ret_from_fork+0x14/0x2c)

Some GDB GUIs

arm-none-eabi-gdbtui --eval-command="target remote localhost:3333" vmlinux

ddd --eval-command="target remote localhost:3333" --debugger arm-none-eabi-gdb vmlinux

nemiver --remote=localhost:3333 --gdb-binary="$(which arm-none-eabi-gdb)" vmlinux

Using U-Boot with SJTAG support

First you need to build U-Boot with SJTAG support, therefore apply the following patch. For a full U-Boot build tutorial see here.

[chris@thinkpad u-boot.git]$ wget http://www.jann.cc/_downloads/serial_jtag_activation_for_u-boot.patch
[chris@thinkpad u-boot.git]$ patch -p1 < serial_jtag_activation_for_u-boot.patch
patching file arch/arm/cpu/arm926ejs/mxs/spl_power_init.c
Hunk #1 succeeded at 923 (offset 28 lines).
patching file arch/arm/include/asm/arch-mxs/regs-digctl.h
[chris@thinkpad u-boot.git]$

You can find U-Boot precompiled with SJTAG support here: http://sourceforge.net/projects/janncc/files/olinuxino/u-boot/uboot_v2013.04_sjtag/.

U-Boot begins booting as normal:

[   87.850000] System halted.
HTLLCLC

U-Boot 2013.04-dirty (Aug 27 2013 - 14:35:49)

CPU:   Freescale i.MX23 rev1.4 at 454 MHz
BOOT:  SSP SD/MMC #0
DRAM:  64 MiB
MMC:   MXS MMC: 0
*** Warning - bad CRC, using default environment

In:    serial
Out:   serial
Err:   serial
Net:   Net Initialization Skipped
No ethernet found.
Hit any key to stop autoboot:  0
mmc0 is current device
SD/MMC found on device 0
662 bytes read in 105 ms (5.9 KiB/s)
Importing environment from mmc (uEnv.txt)...
4299344 bytes read in 1396 ms (2.9 MiB/s)
8674 bytes read in 99 ms (85 KiB/s)
Booting from mmc ...
## Flattened Device Tree blob at 41000000
  Booting using the fdt blob at 0x41000000
  Loading Device Tree to 43b67000, end 43b6c1e1 ... OK

Starting kernel ...

Uncompressing Linux... done, booting the kernel.
[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 3.9.4-dirty (chris@thinkpad) (gcc version 4.7.3 20130312 (release) [ARM/embedded-4_7-branch revision 196615] (GNU Tools f3
[    0.000000] CPU: ARM926EJ-S [41069265] revision 5 (ARMv5TEJ), cr=00053177
...
[  OK  ] Reached target Multi-User System.
[  OK  ] Reached target Graphical Interface.

Arch Linux 3.9.4-dirty (ttyAMA0)

olinuxino login: root
Password:
Last login: Wed Dec 31 18:00:58 on ttyAMA0
[root@olinuxino ~]#
[root@olinuxino ~]# # first start the debugger and attach
[root@olinuxino ~]# echo heartbeat >  /sys/class/leds/green/trigger
[root@olinuxino ~]#

And after the second try (second boot) I was able to connect with OpenOCD without errors (without errors of importance):

[chris@thinkpad blink_led]$ openocd -f /usr/share/openocd/scripts/interface/olimex-jtag-tiny.cfg -f imx233.cfg
Open On-Chip Debugger 0.8.0-dev-00121-ga4d3446 (2013-08-24-17:09)
Licensed under GNU GPL v2
For bug reports, read
        http://openocd.sourceforge.net/doc/doxygen/bugs.html
Info : only one transport option; autoselect 'jtag'
trst_and_srst srst_pulls_trst srst_gates_jtag trst_push_pull srst_open_drain connect_deassert_srst
adapter speed: 800 kHz
dcc downloads are enabled
fast memory access is enabled
Info : clock speed 750 kHz
Info : JTAG tap: imx23.cpu tap/device found: 0x079264f3 (mfg: 0x279, part: 0x7926, ver: 0x0)
Info : Embedded ICE version 6
Info : imx23.cpu: hardware has 2 breakpoint/watchpoint units
Info : accepting 'gdb' connection from 3333
Warn : acknowledgment received, but no packet pending
undefined debug reason 7 - target needs reset
Error: Target not halted
Error: Target not halted
Error: Target not halted
Error: Target not halted
Warn : target not halted
target state: halted
target halted in ARM state due to debug-request, current mode: Supervisor
cpsr: 0x600000d3 pc: 0xc0019024
MMU: enabled, D-Cache: enabled, I-Cache: disabled
Warn : memory read caused data abort (address: 0x00000000, size: 0x4, count: 0x1)
Error: Not implemented: target_get_gdb_fileio_info_default
Error: Not implemented: target_get_gdb_fileio_info_default
Error: Not implemented: target_get_gdb_fileio_info_default
Error: Not implemented: target_get_gdb_fileio_info_default

If you see something like this:

Error: JTAG scan chain interrogation failed: all zeroes
Error: Check JTAG interface, timings, target power, etc.
Error: Trying to use configured scan chain anyway...
Error: imx23.cpu: IR capture error; saw 0x00 not 0x01
Warn : Bypassing JTAG setup events due to errors
Info : Embedded ICE version 0
Error: unknown EmbeddedICE version (comms ctrl: 0x00000000)
Info : imx23.cpu: hardware has 2 breakpoint/watchpoint units

Then you have a problem. You can try to power off your board, disconnect the debugger, connect it again, press the yellow button, power up your board and so forth.

Now set a breakpoint:

[chris@thinkpad linux-stable]$ arm-none-eabi-gdb vmlinux
GNU gdb (GNU Tools for ARM Embedded Processors) 7.4.1.20130312-cvs
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=i686-linux-gnu --target=arm-none-eabi".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/chris/local/OLinuXino/imx23-olinuxino/kernel/linux-stable/vmlinux...done.
(gdb) target remote :3333
Remote debugging using :3333
0x00000000 in ?? ()
(gdb) mon halt
target state: halted
target halted in ARM state due to debug-request, current mode: Supervisor
cpsr: 0x600000d3 pc: 0xc0019024
MMU: enabled, D-Cache: enabled, I-Cache: disabled
(gdb) break heartbeat_trig_activate
Breakpoint 1 at 0xc042ee54: file drivers/leds/ledtrig-heartbeat.c, line 82.
(gdb) c
Continuing.

Breakpoint 1, heartbeat_trig_activate (led_cdev=0xc367a814) at drivers/leds/ledtrig-heartbeat.c:82
82      {
(gdb) stepi
kmalloc_slab (size=76) at include/linux/slub_def.h:220
220             return kmalloc_caches[index];
(gdb) stepi
heartbeat_trig_activate (led_cdev=0xc367a814) at drivers/leds/ledtrig-heartbeat.c:82
82      {
(gdb) stepi
kmalloc_slab (size=76) at include/linux/slub_def.h:220
220             return kmalloc_caches[index];
(gdb) c
Continuing.
^C
Program received signal SIGINT, Interrupt.
cpu_arm926_do_idle () at arch/arm/mm/proc-arm926.S:112
112             mcr     p15, 0, r1, c1, c0, 0           @ Restore ICache enable
(gdb) c
Continuing.

Set a breakpoint at start_kernel()

If you hit any key on the serial console while booting you can stop U-Boot before loading the kernel:

HTLLCLC

U-Boot 2013.04-dirty (Aug 27 2013 - 14:35:49)

CPU:   Freescale i.MX23 rev1.4 at 454 MHz
BOOT:  SSP SD/MMC #0
DRAM:  64 MiB
MMC:   MXS MMC: 0
*** Warning - bad CRC, using default environment

In:    serial
Out:   serial
Err:   serial
Net:   Net Initialization Skipped
No ethernet found.
Hit any key to stop autoboot:  0
=>

Then start GDB and set your breakpoint and continue:

[chris@thinkpad linux-stable]$ arm-none-eabi-gdb vmlinux
GNU gdb (GNU Tools for ARM Embedded Processors) 7.4.1.20130312-cvs
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=i686-linux-gnu --target=arm-none-eabi".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/chris/local/OLinuXino/imx23-olinuxino/kernel/linux-stable/vmlinux...done.
(gdb) target remote :3333
Remote debugging using :3333
0x43f8a038 in ?? ()
(gdb) hbreak star
starget_for_each_device    start_dma_with_bch_irq     start_iaa_cycle            start_transaction          start_unlink_intr
start_2wr_probe            start_dma_without_bch_irq  start_kernel               start_tty                  start_worker
start_bandwidth_timer      start_free_itds            start_this_handle          start_unlink_async         started_after
(gdb) hbreak start
start_2wr_probe            start_dma_without_bch_irq  start_kernel               start_tty                  start_worker
start_bandwidth_timer      start_free_itds            start_this_handle          start_unlink_async         started_after
start_dma_with_bch_irq     start_iaa_cycle            start_transaction          start_unlink_intr
(gdb) hbreak start_kernel
Hardware assisted breakpoint 1 at 0xc08044b0: file init/main.c, line 471.
(gdb) c
Continuing.

Now tell U-Boot to boot the kernel:

=> boot
mmc0 is current device
SD/MMC found on device 0
662 bytes read in 104 ms (5.9 KiB/s)
Importing environment from mmc (uEnv.txt)...
4299344 bytes read in 1395 ms (2.9 MiB/s)
8674 bytes read in 99 ms (85 KiB/s)
Booting from mmc ...
## Flattened Device Tree blob at 41000000
  Booting using the fdt blob at 0x41000000
  Loading Device Tree to 43b67000, end 43b6c1e1 ... OK

Starting kernel ...

Uncompressing Linux... done, booting the kernel.

And voila we hit the breakpoint:

(gdb) c
Continuing.

Breakpoint 1, start_kernel () at init/main.c:471
471     {
(gdb) step
479             lockdep_init();
(gdb) step
lockdep_init () at kernel/lockdep.c:3982
3982            if (lockdep_initialized)
(gdb) step
3985            for (i = 0; i < CLASSHASH_SIZE; i++)
(gdb) s
3986                    INIT_LIST_HEAD(classhash_table + i);
(gdb) s
3985            for (i = 0; i < CLASSHASH_SIZE; i++)
(gdb) s
3986                    INIT_LIST_HEAD(classhash_table + i);
(gdb)

Now you should be able to debug the kernel from the beginning.

TODO

How to use JTAG to debug u-boot, imx-bootlets
How to load the kernel/u-boot over JTAG into memory and run it?

Debugging the Linux kernel via kgdb

Another possibility to debug the kernel is to use kgdb on a serial port console. With kgdb you may also be able to debug the kernel on your desktop PC If you have problems with new hardware and write a detailed bug report or fix the problem yourself.

You need to build the kernel with:

Kernel hacking  --->
  [*] Compile the kernel with debug info
  [*] KGDB: kernel debugger  --->
     <*>   KGDB: use kgdb over the serial console (NEW)

And add kgdboc=ttyAMA0,115200 to the kernel commandline:

[chris@thinkpad imx-bootlets-src-10.05.02]$ vim linux_prep/cmdlines/imx23_olinuxino_dev.txt
[chris@thinkpad imx-bootlets-src-10.05.02]$ cat linux_prep/cmdlines/imx23_olinuxino_dev.txt
noinitrd console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 root=/dev/mmcblk0p2 rw rootwait ssp1=mmc
noinitrd console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 root=/dev/mmcblk0p2 rw rootwait ssp1=mmc
noinitrd console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 root=/dev/mmcblk0p2 rw rootwait ssp1=mmc
[chris@thinkpad imx-bootlets-src-10.05.02]$

Build and start the kgdb agent proxy:

[chris@thinkpad Downloads]$ git clone git://git.kernel.org/pub/scm/utils/kernel/kgdb/agent-proxy.git
Cloning into 'agent-proxy'...
remote: Counting objects: 25, done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 25 (delta 11), reused 25 (delta 11)
Receiving objects: 100% (25/25), 24.16 KiB | 0 bytes/s, done.
Resolving deltas: 100% (11/11), done.
[chris@thinkpad Downloads]$ cd agent-proxy/
[chris@thinkpad agent-proxy]$ make
gcc -DAGENT_VER=1.96 -g -Wall -Wno-unused-parameter -Dlinux -c agent-proxy.c -o agent-proxy.o
gcc -DAGENT_VER=1.96 -g -Wall -Wno-unused-parameter -Dlinux -c agent-proxy-rs232.c -o agent-proxy-rs232.o
gcc -DAGENT_VER=1.96 -g -Wall -Wno-unused-parameter -Dlinux -o agent-proxy agent-proxy.o agent-proxy-rs232.o
[chris@thinkpad agent-proxy]$ ./agent-proxy 4440^4441 0 /dev/ttyUSB0,115200 &

Now connect via telnet to the serial console, log in and switch to kdbg:

[chris@thinkpad ~]$ telnet localhost 4440
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
...
olinuxino login: root
Password:
Last login: Wed Dec 31 18:59:54 on ttyAMA0
[root@olinuxino ~]# echo g > /proc/sysrq-trigger
[   65.500000] SysRq : DEBUG
[   65.500000] Entering KGDB

Start GDB in another console, set a breakpoint for the led heartbeat trigger and continue:

[chris@thinkpad linux-stable]$ arm-none-eabi-gdb
GNU gdb (GNU Tools for ARM Embedded Processors) 7.4.1.20130312-cvs
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=i686-linux-gnu --target=arm-none-eabi".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
(gdb) file vmlinux
Reading symbols from /home/chris/local/OLinuXino/imx23-olinuxino/kernel/linux-stable/vmlinux...done.
(gdb) target remote localhost:4441
Remote debugging using localhost:4441
kgdb_breakpoint () at kernel/debug/debug_core.c:1013
1013            arch_kgdb_breakpoint();
(gdb) break heartbeat_trig_activate
Breakpoint 1 at 0xc042ee54: file drivers/leds/ledtrig-heartbeat.c, line 82.
(gdb) continue
Continuing.

Now you should be able to use the serial console again, just type echo g > /proc/sysrq-trigger to switch back to the debugger.

Now lets try to hit the breakpoint:

[root@olinuxino ~]# echo heartbeat >  /sys/class/leds/green/trigger
$T05thread:8b;#40+$14a8a7c3d80b32c10000000054ee42c014a8a7c36c2987c0e0d9b7c3782987c05030a8c308aaa7c348af5dc0000000009c6596c0c8fe5fc15cd742c054ee42c000000000000000000000000000000000000000000000000000000000000000000000000000000000#92+$OK#9a+$38402de9#03

Now the consonle is unusable again, ignore the character salad, this is output for GDB.

If you switch back to the console with GDB you will see that we have hit the breakpoint and your can display a backtrace:

(gdb) break heartbeat_trig_activate
Breakpoint 1 at 0xc042ee54: file drivers/leds/ledtrig-heartbeat.c, line 82.
(gdb) continue
Continuing.

Breakpoint 1, heartbeat_trig_activate (led_cdev=0xc3a7a814) at drivers/leds/ledtrig-heartbeat.c:82
82      {
(gdb) bt
#0  heartbeat_trig_activate (led_cdev=0xc3a7a814) at drivers/leds/ledtrig-heartbeat.c:82
#1  0xc042d75c in led_trigger_set (led_cdev=led_cdev@entry=0xc3a7a814, trig=trig@entry=0xc087296c) at drivers/leds/led-triggers.c:131
#2  0xc042db28 in led_trigger_store (dev=<optimized out>, attr=<optimized out>, buf=<optimized out>, count=10) at drivers/leds/led-triggers.c:58
#3  0xc02d1740 in dev_attr_store (kobj=<optimized out>, attr=<optimized out>, buf=<optimized out>, count=<optimized out>) at drivers/base/core.c:114
#4  0xc012f868 in flush_write_buffer (count=10, buffer=0xc3afc800, dentry=<optimized out>) at fs/sysfs/file.c:202
#5  sysfs_write_file (file=<optimized out>, buf=<optimized out>, count=<optimized out>, ppos=0xc15fff88) at fs/sysfs/file.c:236
#6  0xc00d486c in vfs_write (file=file@entry=0xc3bdf460, buf=0xb6ffc000 "heartbeat\n", count=10, count@entry=0, pos=pos@entry=0xc15fff88) at fs/read_write.c:463
#7  0xc00d4cd8 in sys_write (fd=<optimized out>, buf=<optimized out>, count=0) at fs/read_write.c:510
#8  0xc000ed80 in kuser_cmpxchg32_fixup ()
#9  0xc000ed80 in kuser_cmpxchg32_fixup ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb)

Breakpoint at heartbeat_trig_activate

Experiments

Just playing around...

Playing a bit with OpenOCD

[chris@thinkpad linux-stable]$ telnet localhost 4444
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Open On-Chip Debugger
> pwd
/home/chris/local/OLinuXino/Debugging/SJTAG-Example/blink_led
> reset init
JTAG tap: imx23.cpu tap/device found: 0x079264f3 (mfg: 0x279, part: 0x7926, ver: 0x0)
srst pulls trst - can not reset into halted mode. Issuing halt after reset.
target state: halted
target halted in ARM state due to debug-request, current mode: Supervisor
cpsr: 0x800000d3 pc: 0x0000206c
MMU: disabled, D-Cache: disabled, I-Cache: disabled
> halt
> scan_chain
  TapName             Enabled  IdCode     Expected   IrLen IrCap IrMask
-- ------------------- -------- ---------- ---------- ----- ----- ------
0 imx23.cpu              Y     0x079264f3 0x079264f3     4 0x01  0x03
> resume
> poll
background polling: on
TAP: imx23.cpu (enabled)
target state: running
> poll
background polling: on
TAP: imx23.cpu (enabled)
target state: halted
target halted in ARM state due to debug-request, current mode: Supervisor
cpsr: 0x800000d3 pc: 0x00002070
MMU: disabled, D-Cache: disabled, I-Cache: disabled
> reg
===== ARM registers
(0) r0 (/32): 0x000186A0 (dirty)
(1) r1 (/32): 0x00002120
(2) r2 (/32): 0x000036CC
(3) r3 (/32): 0x06B4C3F7
(4) r4 (/32): 0x000056C0
(5) r5 (/32): 0x00002000
(6) r6 (/32): 0x0000621C
(7) r7 (/32): 0x00000000
(8) r8 (/32): 0x00000000
(9) r9 (/32): 0x00000000
(10) r10 (/32): 0xFFFF2248
(11) r11 (/32): 0x0000539C
(12) r12 (/32): 0x80018724
(13) sp_usr (/32)
(14) lr_usr (/32)
(15) pc (/32): 0x00002070 (dirty)
(16) r8_fiq (/32)
(17) r9_fiq (/32)
(18) r10_fiq (/32)
(19) r11_fiq (/32)
(20) r12_fiq (/32)
(21) sp_fiq (/32)
(22) lr_fiq (/32)
(23) sp_irq (/32)
(24) lr_irq (/32)
(25) sp_svc (/32): 0x00005388
(26) lr_svc (/32): 0x000020D8
(27) sp_abt (/32)
(28) lr_abt (/32)
(29) sp_und (/32)
(30) lr_und (/32)
(31) cpsr (/32): 0x800000D3
(32) spsr_fiq (/32)
(33) spsr_irq (/32)
(34) spsr_svc (/32): 0x00000010
(35) spsr_abt (/32)
(36) spsr_und (/32)
===== EmbeddedICE registers
(37) debug_ctrl (/6): 0x05
(38) debug_status (/10)
(39) comms_ctrl (/6)
(40) comms_data (/32)
(41) watch_0_addr_value (/32): 0x00000018
(42) watch_0_addr_mask (/32): 0x00000003
(43) watch_0_data_value (/32)
(44) watch_0_data_mask (/32): 0xFFFFFFFF
(45) watch_0_control_value (/9): 0x0000
(46) watch_0_control_mask (/8): 0xF7
(47) watch_1_addr_value (/32)
(48) watch_1_addr_mask (/32)
(49) watch_1_data_value (/32)
(50) watch_1_data_mask (/32)
(51) watch_1_control_value (/9)
(52) watch_1_control_mask (/8)
(53) vector_catch (/8)
> soft_reset_halt
requesting target halt and executing a soft reset
target state: halted
target halted in ARM state due to debug-request, current mode: Supervisor
cpsr: 0x800000d3 pc: 0x00000000
MMU: disabled, D-Cache: disabled, I-Cache: disabled
> resume
> poll
background polling: on
TAP: imx23.cpu (enabled)
target state: running
> halt
target state: halted
target halted in ARM state due to debug-request, current mode: Supervisor
cpsr: 0x400000d3 pc: 0x00000004
MMU: disabled, D-Cache: disabled, I-Cache: disabled
> arm disassemble 0x00000004 10
0x00000004      0xe3110001      TST r1, #0x1
0x00000008      0x0afffffc      BEQ 0x00000000
0x0000000c      0xee111e10      MRC p14, 0x00, r1, c1, c0, 0x00
0x00000010      0xe4801004      STR r1, [r0], #0x4
0x00000014      0xeafffff9      B 0x00000000
0x00000018      0x00002000      ANDEQ r2, r0, r0
0x0000001c      0x0000621c      ANDEQ r6, r0, r12, LSL r2
0x00000020      0x00000000      ANDEQ r0, r0, r0
0x00000024      0x00000000      ANDEQ r0, r0, r0
0x00000028      0x00000000      ANDEQ r0, r0, r0
> reset init
JTAG tap: imx23.cpu tap/device found: 0x079264f3 (mfg: 0x279, part: 0x7926, ver: 0x0)
srst pulls trst - can not reset into halted mode. Issuing halt after reset.
target state: halted
target halted in ARM state due to debug-request, current mode: Supervisor
cpsr: 0x800000d3 pc: 0x0000206c
MMU: disabled, D-Cache: disabled, I-Cache: disabled
> resume
> halt
target state: halted
target halted in ARM state due to debug-request, current mode: Supervisor
cpsr: 0x800000d3 pc: 0x00002070
MMU: disabled, D-Cache: disabled, I-Cache: disabled
> arm disassemble 0x00002070 20
0x00002070      0xe51b3010      LDR r3, [r11, #-0x10]
0x00002074      0xe1520003      CMP r2, r3
0x00002078      0x3afffff8      BCC 0x00002060
0x0000207c      0xe28bd000      ADD r13, r11, #0x0
0x00002080      0xe8bd0800      LDM r13!, {r11}
0x00002084      0xe12fff1e      BX r14
0x00002088      0x8001c0c0      ANDHI r12, r1, r0, ASR #0x1
0x0000208c      0xe92d4800      STMDB r13!, {r11, r14}
0x00002090      0xe28db004      ADD r11, r13, #0x4
0x00002094      0xe59f3054      LDR r3, [r15, #0x54]
0x00002098      0xe59f2054      LDR r2, [r15, #0x54]
0x0000209c      0xe5832000      STR r2, [r3]
0x000020a0      0xe59f3050      LDR r3, [r15, #0x50]
0x000020a4      0xe59f2050      LDR r2, [r15, #0x50]
0x000020a8      0xe5832000      STR r2, [r3]
0x000020ac      0xe59f304c      LDR r3, [r15, #0x4c]
0x000020b0      0xe3a02002      MOV r2, #0x2
0x000020b4      0xe5832000      STR r2, [r3]
0x000020b8      0xe59f3044      LDR r3, [r15, #0x44]
0x000020bc      0xe3a02002      MOV r2, #0x2
> step
target state: halted
target halted in ARM state due to breakpoint, current mode: Supervisor
cpsr: 0x800000d3 pc: 0x00002074
MMU: disabled, D-Cache: disabled, I-Cache: disabled
> step
target state: halted
target halted in ARM state due to breakpoint, current mode: Supervisor
cpsr: 0x800000d3 pc: 0x00002078
MMU: disabled, D-Cache: disabled, I-Cache: disabled
> step
target state: halted
target halted in ARM state due to breakpoint, current mode: Supervisor
cpsr: 0x800000d3 pc: 0x00002060
MMU: disabled, D-Cache: disabled, I-Cache: disabled
>

Debugging U-Boot

This is a bit more complicated because before we can use SJTAG it needs to be initialized by software.

This thread may be helpful: https://community.freescale.com/thread/302010

If you change to your U-Boot build directory you will find among others a file called u-boot.map that contains the following lines:

[chris@thinkpad linux-stable]$ cd /home/chris/local/OLinuXino/u-boot.git/
[chris@thinkpad u-boot.git]$ sed -n 14,28p u-boot.map
Memory Configuration

Name             Origin             Length             Attributes
*default*        0x00000000         0xffffffff

Linker script and memory map

                0x00000000                . = 0x0
                0x00000000                . = ALIGN (0x4)

.text           0x40000100    0x29d74
                0x40000100                __image_copy_start = .
arch/arm/cpu/arm926ejs/start.o(.text*)
.text          0x40000100      0x400 arch/arm/cpu/arm926ejs/start.o
                0x40000100                _start
[chris@thinkpad u-boot.git]$

So the start address is 0x40000100.

It seems I can at least load a new U-Boot into memory and resume at the start address:

[chris@thinkpad u-boot.git]$ telnet localhost 4444
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Open On-Chip Debugger
> halt
target state: halted
target halted in ARM state due to debug-request, current mode: Supervisor
cpsr: 0x200000d3 pc: 0x43f8a038
MMU: enabled, D-Cache: enabled, I-Cache: enabled
> load_image /home/chris/local/OLinuXino/u-boot.git/u-boot.bin 0x40000100
240368 bytes written at address 0x40000100
downloaded 240368 bytes in 3.831031s (61.272 KiB/s)
> resume 0x40000100
> resume 0x40000100
target not halted
in procedure 'resume'
> halt
target state: halted
target halted in ARM state due to debug-request, current mode: Supervisor
cpsr: 0x600000d3 pc: 0x43f7f3dc
MMU: enabled, D-Cache: enabled, I-Cache: enabled
> resume 0x40000100
>

But then U-Boot fails to resume:

U-Boot 2013.04-dirty (Aug 31 2013 - 14:23:58)

CPU:   Freescale i.MX23 rev1.4 at 454 MHz
BOOT:  SSP SD/MMC #0
DRAM:  64 MiB
## Can't malloc 46 bytes
Hit any key to stop autoboot:  0
ERROR : memory not allocated

If we want to use GDB we need to set the correct relocation address for the symbol file, in the u-boot console type bdinfo to get the relocation address:

=> bdinfo
arch_number = 0x00001009
boot_params = 0x40000100
DRAM bank   = 0x00000000
-> start    = 0x40000000
-> size     = 0x04000000
current eth = unknown
ip_addr     = <NULL>
baudrate    = 115200 bps
TLB addr    = 0x43FF0000
relocaddr   = 0x43F72000
reloc off   = 0x03F71F00
irq_sp      = 0x43B6DF40
sp start    = 0x43B6DF30
=>

Then we have at least correct debug symbols to debug the already running U-Boot:

[chris@thinkpad u-boot.git]$ arm-none-eabi-gdb u-boot
GNU gdb (GNU Tools for ARM Embedded Processors) 7.4.1.20130312-cvs
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=i686-linux-gnu --target=arm-none-eabi".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/chris/local/OLinuXino/u-boot.git/u-boot...done.
(gdb) target remote :3333
Remote debugging using :3333
0x43f8a038 in ?? ()
(gdb) c
Continuing.
^C
Program received signal SIGINT, Interrupt.
0x43f8a038 in ?? ()
(gdb) symbol-file
Discard symbol table from `/home/chris/local/OLinuXino/u-boot.git/u-boot'? (y or n) y
No symbol file now.
(gdb) add-symbol-file u-boot 0x43F72000
add symbol table from file "u-boot" at
        .text_addr = 0x43f72000
(y or n) y
Reading symbols from /home/chris/local/OLinuXino/u-boot.git/u-boot...done.
(gdb) c
Continuing.
^C
Program received signal SIGINT, Interrupt.
pl01x_serial_getc () at serial_pl01x.c:184
184             return pl01x_getc (CONSOLE_PORT);
(gdb) break puts
Breakpoint 3 at 0x43f7b578: file console.c, line 468.
(gdb) c
Continuing.

Breakpoint 3, puts (s=0x43b6d9ec "H") at console.c:468
468             if (!gd->have_console)
(gdb) c
Continuing.

Breakpoint 3, puts (s=0x43b6d9ec "e") at console.c:468
468             if (!gd->have_console)
(gdb) c
Continuing.

Breakpoint 3, puts (s=0x43b6d9ec "l") at console.c:468
468             if (!gd->have_console)
(gdb) c
Continuing.

Breakpoint 3, puts (s=0x43b6d9ec "l") at console.c:468
468             if (!gd->have_console)
(gdb) c
Continuing.

Breakpoint 3, puts (s=0x43b6d9ec "o") at console.c:468
468             if (!gd->have_console)
(gdb) c
Continuing.
^C
Program received signal SIGINT, Interrupt.
pl01x_serial_getc () at serial_pl01x.c:184
184             return pl01x_getc (CONSOLE_PORT);
(gdb) load u-boot
Loading section .text, size 0x29d74 lma 0x40000100
Loading section .rodata, size 0x95f6 lma 0x40029e74
Loading section .hash, size 0x40 lma 0x4003346c
Loading section .data, size 0x1908 lma 0x400334ac
Loading section .got.plt, size 0xc lma 0x40034db4
Loading section .u_boot_list, size 0x730 lma 0x40034dc0
Loading section .rel.dyn, size 0x5650 lma 0x400354f0
Loading section .dynsym, size 0xb0 lma 0x4003ab40
Start address 0x40000100, load size 240366
Transfer rate: 52 KB/sec, 11446 bytes/write.
(gdb) set $pc = 0x40000100
(gdb) c
Continuing.

But it fails to boot again.

Acknowledgments

I would like to thank Bones AG for providing me the iMX233-SJTAG, the J-Link and some other equipment. My next task is now to port Linux to the Bones iMX233 Audio Development Board.

References/Further Reading

Pingback:

Last updated: 2013-09-11

Posted by Christian Jann

Filed under: English

Tags: OLinuXino, Freescale, iMX233, ARM, HOWTO, Olimex, OpenOCD, JTAG, Embedded Linux, Linux Kernel, U-Boot, Debugging

Christian's Blog

Linux, programming, hacking, electronics, Python… These are the things I love.