Linux kernel crash analysis example

look_w

Issue reported:

When USB is connected as Mass Storage mode, copy file from external sdcard to clipboard.
The disconnect USB and try to paste clipboard file into internal sdcard but it will fail.
Reconnect USB and target crash several minutes later.


Crash Context:

• <4>[32284.249267] C0 [      swapper/0] CPU: 0    Tainted: G        W     (3.4.5 #1)  
• <4>[32284.256499] C0 [      swapper/0] PC is at DWC_WORKQ_SCHEDULE+0xd4/0x108  
• <4>[32284.263152] C0 [      swapper/0] LR is at DWC_WORKQ_SCHEDULE+0x94/0x108  
• <4>[32284.269866] C0 [      swapper/0] pc : [<c0332fc4>]    lr : [<c0332f84>]    psr: 800001d3  
• <4>[32284.269897] C0 [      swapper/0] sp : c0947dc0  ip : dc050000  fp : 00000000  
• <4>[32284.285064] C0 [      swapper/0] r10: 00000000  r9 : 00000000  r8 : db376e04  
• <4>[32284.292174] C0 [      swapper/0] r7 : db370000  r6 : c0318050  r5 : db2d6a40  r4 : d972e140  
• <4>[32284.300597] C0 [      swapper/0] r3 : 00000000  r2 : caa91840  r1 : d972e158  r0 : 0000007f  
• <4>[32284.309020] C0 [      swapper/0] Flags: Nzcv  IRQs off  FIQs off  Mode SVC_32  ISA ARM  Segment kernel  
• <4>[32284.318389] C0 [      swapper/0] Control: 10c53c7d  Table: 8291806a  DAC: 00000015  
• <4>[32284.326018] C0 [      swapper/0]   
• <4>[32284.326049] C0 [      swapper/0] PC: 0xc0332f44:  
• <4>[32284.334014] C0 [      swapper/0] 2f44  e595000c ebfffc5c e3a00044 ebffff3d e1a04000 e88000c0 e5805008 e59f007c  
• <4>[32284.344451] C0 [      swapper/0] 2f64  ebffff3a e59f1078 e59f2078 e584000c e1a03000 e59f0070 e58d4000 ebfffd7b  
• <4>[32284.354858] C0 [      swapper/0] 2f84  e3a03c05 e5843018 e284301c e584301c e5843020 e2841018 e59f3050 e5843024  
• <4>[32284.365264] C0 [      swapper/0] 2fa4  e2853010 e5843010 e5952014 e5842014 e5952010 e1520003 05854010 15953014  
• <4>[32284.375671] C0 [      swapper/0] 2fc4  15834010 e5854014 e5950000 ebf4e615 e28dd018 e8bd40f0 e28dd004 e12fff1e  
• <4>[32284.386077] C0 [      swapper/0] 2fe4  c0a71d34 c063fef4 c076b31a c07a722f c03326a8 e92d4038 e1a04000 e59f501c  
• <4>[32284.396484] C0 [      swapper/0] 3004  ea000003 e5953004 e2444001 e59f0010 e12fff33 e3540000 1afffff9 e8bd8038  
• <4>[32284.406890] C0 [      swapper/0] 3024  c09944e0 066665b0 e92d4008 e59f3008 e5933008 e12fff33 e8bd8008 c09944e0  
• <4>[32284.417358] C0 [      swapper/0]   
• <4>[32284.417358] C0 [      swapper/0] LR: 0xc0332f04:  
• <4>[32284.425353] C0 [      swapper/0] 2f04  e1a06001 e1a07002 e3a01080 e59d202c e59f00c8 e58d300c ebfc2336 e5950008  
• <4>[32284.435790] C0 [      swapper/0] 2f24  e28d1010 ebfffd28 e5953004 e59d1010 e2833001 e5950008 e5853004 eb0b2c0d  
• <4>[32284.446197] C0 [      swapper/0] 2f44  e595000c ebfffc5c e3a00044 ebffff3d e1a04000 e88000c0 e5805008 e59f007c  
• <4>[32284.456634] C0 [      swapper/0] 2f64  ebffff3a e59f1078 e59f2078 e584000c e1a03000 e59f0070 e58d4000 ebfffd7b  
• <4>[32284.467040] C0 [      swapper/0] 2f84  e3a03c05 e5843018 e284301c e584301c e5843020 e2841018 e59f3050 e5843024  
• <4>[32284.477477] C0 [      swapper/0] 2fa4  e2853010 e5843010 e5952014 e5842014 e5952010 e1520003 05854010 15953014  
• <4>[32284.487884] C0 [      swapper/0] 2fc4  15834010 e5854014 e5950000 ebf4e615 e28dd018 e8bd40f0 e28dd004 e12fff1e  
• <4>[32284.498321] C0 [      swapper/0] 2fe4  c0a71d34 c063fef4 c076b31a c07a722f c03326a8 e92d4038 e1a04000 e59f501c  

System Triage Procedure:1) Find call stack and locate the DWC_WORKQ_SCHEDULE() API
2) Get the assembly code via objdump for offending API
3) ARM assembly code is listed

• EXPORT_SYMBOL(DWC_WORKQ_FREE);  
• 
  
• void DWC_WORKQ_SCHEDULE(dwc_workq_t *wq, dwc_work_callback_t work_cb,  
•             void *data, char *format, ...)  
• {  
•     <strong><span style="color:#FF0000;">107c</span></strong>:  e52d3004    push    {r3}        ; (str r3, [sp, #-4]!)  
•     1080:   e92d40f0    push    {r4, r5, r6, r7, lr}                <span style="color:#FF0000;">// 0x107c + 0xD4 offset = > 0x1150</span>
  
•     1084:   e24dd018    sub sp, sp, #24  
•     1088:   e1a05000    mov r5, r0  
•     int64_t flags;  
•     work_container_t *container;  
•     static
  char name[128];  
• 
  
•     va_list args;  
•     va_start(args, format);  
•     108c:   e28d3030    add r3, sp, #48 ; 0x30  

4) Check offset=0x1150 code

• #ifdef DEBUG
  
•     DWC_CIRCLEQ_INSERT_TAIL(&wq->entries, container, entry);  
•     1130:   e2853010    add r3, r5, #16  
•     1134:   e5843010    str r3, [r4, #16]  
•     1138:   e5952014    ldr r2, [r5, #20]  
•     113c:   e5842014    str r2, [r4, #20]  
•     1140:   e5952010    ldr r2, [r5, #16]  
•     1144:   e1520003    cmp r2, r3  
•     1148:   05854010    streq   r4, [r5, #16]  
•     114c:   15953014    ldrne   r3, [r5, #20]  
•     <span style="color:#FF0000;"><strong>1150</strong></span>:  15834010    strne   r4, [r3, #16]                           // 0xc0332f44  
  
•     1154:   e5854014    str r4, [r5, #20]  
• #endif
  

5) Check ARM instruction against crash context

• <4>[32284.375671] C0 [      swapper/0] 2fc4  <span
  style="color:#FF0000;"><strong>15834010 </strong></span>e5854014 e5950000 ebf4e615 e28dd018 e8bd40f0 e28dd004 e12fff1e  

6) Now we can conclude that null pointer is caused in offset=0x1150.7) Check against source c code DWC_CIRCLEQ_INSERT_TAIL()

• #define DWC_CIRCLEQ_INSERT_TAIL(head, elm, field) do {            \
  
•     (elm)->field.cqe_next = DWC_CIRCLEQ_END(head);            \  
•     (elm)->field.cqe_prev = (head)->cqh_last;            \  
•     if ((head)->cqh_first == DWC_CIRCLEQ_END(head))            \  
•         (head)->cqh_first = (elm);                \  
•     else                                \  
•         (head)->cqh_last->field.cqe_next = (elm);    \  
•     (head)->cqh_last = (elm);    \  
• } while (0)  

8) Analyze Crash Context

• r3 : 00000000  

So we know R3=(head)->cqh_last and it is NULL pointer.
9) Add protection code for NULL pointer.