gmane.linux.kernel.crash-dump.crash-utility

Re: [Patch v4] Show module taint flags

Dave Anderson — 2013-05-21T18:36:57


Hi Aaron,

With respect to your v4 version of the patch, I just found a few
issues that need addressing.

The module.gpgsig_ok "unsigned" check was only made if module.taints
exists, and not checked if module.license_gplok exists.  Earlier
kernels have license_gplok/gpgsig_ok combinations.

I'm not sure how you were able to get this sample display from your 
help page:

   crash> mod -T
     NAME                 TAINT
     dm_mod               G
     scsi_tgt             G
     serio_raw            G
     dm_log               G
     ata_generic          G
     qla2xxx              P(U)
     dm_region_hash       G
     enclosure            G
     pata_acpi            G

because the module.taints field would be 0 for all but the
qla2xxx module, and your code gates the tnts[] true/false display 
by "if (taints)"?  In my tests of your patch, the 'G' would only 
be shown if some *other* bit in the bitfield were set. 

Anway, I don't feel that it's necessary to print the 'G' for GPL'd
modules.  We could probably

[Patch v4] Show module taint flags

Aaron Tomlin — 2013-05-19T16:51:19

Hi Dave,


Since v4:

 - Updated help_mod[] help page
 - User is notified if no tainted modules exists
 - Added the '-t' option, to display the hexadecimal value of a module's "taint" flag

 
Examples:

crash> mod -T
NOTE: modules have changed on this system -- reinitializing
NAME                     TAINT
test                     GFO

crash> mod -t
NAME                     TAINT
test                     0x1002

crash> mod -T
NAME                 TAINT
vxfs                 P(U)
vxspec               P(U)
dmpaa                P(U)
dmpap                P(U)
dmpjbod              P(U)
fdd                  P(U)
vxportal             P(U)
vxdmp                P(U)
vxio                 P(U)
llt                  P(U)
gab                  P(U)
vxfen                P(U)
amf                  P(U)
vxodm                P(U)

crash> mod -t
NAME                 TAINT
vxfs                 0x1
vxspec               0x1
dmpaa                0x1
dmpap                0x1
dmpjbod              0x1
f

Re: EPPIC: fails to build in crash-7.0.0/gdb-7.6environment

Luc Chouinard — 2013-05-17T20:34:39

Dave - I just pushed the epic.c fix.
When you release 7.0.1, I will work through the warnings.
And thanks!


On Fri, May 17, 2013 at 4:07 PM, Dave Anderson <anderson< at >redhat.com> wrote:

EPPIC: fails to build in crash-7.0.0/gdb-7.6environment

Dave Anderson — 2013-05-17T20:07:23

Luc et al,

As I mentioned in the crash-7.0.0 announcement, the build 
of the eppic extension module fails due to changes in the
newly-embedded gdb-7.6 tree:

$ make extensions
...
Cloning into 'eppic'...
remote: Counting objects: 154, done.
remote: Finding sources: 100% (154/154), done.
remote: Total 154 (delta 65)
Receiving objects: 100% (154/154), 173.79 KiB | 267 KiB/s, done.
Resolving deltas: 100% (65/65), done.
cd eppic/libeppic && make
bison -peppic -v -t -d eppic.y
eppic.y: conflicts: 253 shift/reduce, 20 reduce/reduce
cc -O0 -g -fPIC   -c -o eppic_util.o eppic_util.c
cc -O0 -g -fPIC   -c -o eppic_node.o eppic_node.c
cc -O0 -g -fPIC   -c -o eppic_var.o eppic_var.c
cc -O0 -g -fPIC   -c -o eppic_func.o eppic_func.c
cc -O0 -g -fPIC   -c -o eppic_str.o eppic_str.c
cc -O0 -g -fPIC   -c -o eppic_op.o eppic_op.c
cc -O0 -g -fPIC   -c -o eppic_num.o eppic_num.c
cc -O0 -g -fPIC   -c -o eppic_stat.o eppic_stat.c
cc -O0 -g -fPIC   -c -o eppic_builtin.o eppic_builtin.c
cc -O0 -g -fPIC   -c -o eppic_type.o eppic_

Re: [ANNOUNCE] crash gcore command version 1.2.1 is released

Dave Anderson — 2013-05-14T14:50:07


----- Original Message -----


Thanks Daisuke -- I've posted crash-gcore-command-1.2.1.tar.gz on the
extensions page:
 
 http://people.redhat.com/anderson/extensions.html#GCORE

Dave

[ANNOUNCE] crash gcore command,version 1.2.1 is released

HATAYAMA Daisuke — 2013-05-14T04:15:51

This is the release of crash gcore command, version 1.2.1.

ChangeLog:

 - Fix failure of coredump at accessing memory for VSYSCALL page due
   to wrong conversion of uvtop which wrongly treats address
   VSYSCALL_START as belongs to kernel direct mapping region. This fix
   executes uvtop in verbose mode to make it always paging and
   retrieves the correct physical address from its output. Without
   this fix, VSYSCALL page fails to be collected and core dump process
   is aborted; though VSYSCALL page is done in the last so allmost all
   corefile is already generated.
   (d.hatayama< at >jp.fujitsu.com)

 - Skip page-faulted pages by lseek() rather than writing zero-filled
   pages. By this, generated core file has holes in the corresponding
   positions for each page-faulted pages if filesystem supports sparse
   files. This is highly useful when the target process has huge
   virtual memory space such as qemu process that has huge physical
   memory of KVM guest machine.
   (d.hatayama< at >jp.fujitsu.com)

 - F

[ANNOUNCE] crash-7.0.0 is available

Dave Anderson — 2013-05-10T20:19:35

The upgrade from gdb-7.3.1 to gdb-7.6 has been accomplished with
this release.  It was painful as usual, and as history has proven
in the past, there are going to be regressions.  I've addressed 
several of them, but it's highly likely that others will crop up.

Most notably, the eppic extension module no longer builds, but
that needs to be addressed in the eppic git tree.

Download from: http://people.redhat.com/anderson

Changelog:

 - Updated the embedded gdb version to FSF gdb-7.6, which was officially
   released by the Free Software Foundation on http://www.gnu.org on 
   4/26/13. The primary motivation for upgrading from gdb-7.3.1 is for 
   future ARM64 support, but there are also issues with respect to 
   kernels built with gcc-4.8.0.  The relevant pieces of gdb-7.3.1.patch
   were forward-ported to the gdb-7.6.patch, and the GDB_7_6 #define has
   been applied in the top-level sources where appropriate.
   (anderson< at >redhat.com)

 - Continued incremental steps for support of the ARM64 architecture

Re: crash read error: kernel virtual address / vmcore address mismatch ?

Anand Raj Manickam — 2013-05-08T06:12:36

After rebooting to the primary / live kernel

   I have attached the output for "crash -d8 vmlinux /data/vmcore" in d8.txt


  You are right , the System Kernel means the Primary kernel and Debug-
  Secondary/Relocated kernel.


 Here the crash utility does not core dump , its the Kernel which does
 not vmcore on the above mentioned address .

 I did increase the crashkernel to 256M but it results the same .

 Thanks,
 Anand

crash 6.1.6
Copyright (C) 2002-2013  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter

Re: crash read error: kernel virtual address / vmcore address mismatch ?

Anand Raj Manickam — 2013-05-08T05:53:20

Do yourself a favor and copy /proc/vmcore to somewhere on disk.  Then
reboot the system into the original kernel, and run crash on the saved
vmcore.  I've never seen crash run on a /proc/vmcore file from the
secondary kernel.

Also, after rebooting the the original kernel, please confirm that
crash runs OK on the live system.



Please show the full output of "crash -d8 vmlinux vmcore".

I have attached the output . d8.txt


I don't know what you mean by "System Kernel" and "Debug Kernel".

With respect to kdump and the crash utility, the only kernel that is
of interest is the vmlinux that was running when the primary system
crashed.


Again, the "Debug" kernel is irrelevant -- presuming by "Debug" you mean
the relocated "secondary" kernel that was kexec'd after the primary kernel
crashed.

You are right , the System Kernel means the Primary kernel and Debug-
Secondary/Relocated kernel.


What "core dumps", the crash utility?

Here the crash utility does not core dump , its the Kernel which does
not vmcore

Re: [PATCH v3] Show module taint flags

Dave Anderson — 2013-05-07T19:07:47


----- Original Message -----

Right -- it seems reasonable to at least print a hexadecimal value for the
earlier kernels.  The "help mod" output should state something to the effect that
the relevant kernel sources should be consulted for the meaning of either the 
letter(s) or hexadecimal bit-value(s). 

Also, this piece had me scratching my head a little bit:

                if (is_module_taint(lm, buf2, &maxtnts_len)) {
                        fprintf(fp, "%s  ", mkstring(buf1, maxnamelen,
                                LJUST, lm->mod_name));
                        fprintf(fp, "%s\n", mkstring(buf2, maxtnts_len,
                                LJUST, buf2));
                }

Why bother with the "maxtnts_len" business and the mkstring() call for the
taints string?  I understand that the lm->mod_name should be passed to 
mkstring() to keep that field length common (i.e. maxnamelen), but for
the taint string, you can just print it as-is without bothering to do
any kind of justification or worrying abo

[PATCH v3] Show module taint flags

Aaron Tomlin — 2013-05-07T17:47:57

Hi Dave,


Sorry about the delay. I've finally found the time to attempt another version.

As per your request, I've updated *help_mod[] to document the -T option.
Also in this particular version, I omit ' ' and '-', when false, as I'm under 
the impression that we're only interested in true values as per module_flags()
(since 2.6.25).
I suspect that there shouldn't be a situation where a module isn't tainted since
every module is either proprietary license or GPL. However, if this is not 
acceptable I can change this behaviour. 

For module.gpgsig_ok case (as seen in kernel-2.6.32-1.el6), I've decided to follow
the same logic to highlight "(U)" to the user, as per print_modules().
With regards to module.sig_ok, this is handled, as seen under 3.8.9-200.fc18:

nf_nat                   G 
bnx2i                    G 
ip6t_REJECT              G 
nf_defrag_ipv6           G 
be2iscsi                 G 
tun                      G 
test                     GFO

For pre 2.6.28 kernels, I'll work on a solution

HEADS-UP: Linux kernel 3.9 debuginfo issues

Dave Anderson — 2013-04-30T16:13:56

Something has changed with the vmlinux debuginfo file such that 
the currently-embedded gdb-7.3.1 version in the crash utility
can no longer can find the debuginfo data for text symbols in
Linux 3.9 kernels.

I don't know whether it's due to a change in the kernel build procedure
arguments or maybe the tools used, because there is no such problem as
recently as 3.8.8-100.fc17.

Taking crash out of the picture, the simple gdb "whatis" command
should show the debuginfo data for kernel text symbols.  For example,
here things work OK with gdb-7.5.1-34.el7: 
  
  # gdb /usr/lib/debug/lib/modules/3.9.0-0.55.el7.x86_64/vmlinux
  GNU gdb (GDB) Red Hat Enterprise Linux (7.5.1-34.el7)
  Copyright (C) 2012 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
  and "show warranty" for details.
  This GDB was configured

Re: How do I get IA64 register R32 and above?

Seymour, Shane M — 2013-04-29T00:28:07

Hi Dave,


They will only be the contents of the input registers in a local function after they were rotated during a function call at the point in time where they were saved/spilt to the RSE. What they may contain is really determined by the ABI/compiler/calling convention and I don't know if Linux or the compiler options used to compile the kernel dictate that the input registers can be reused or not. 

When a function starts usually the first instruction is an alloc instruction which allocates input, local, output registers in the RSE (input and local are really combined). When use the call instruction the RSE automatically renumbers the rotating registers so the output registers start from r32 and the caller uses alloc to formalize its RSE usage (defining the callers outputs as inputs to the callee). 

Although r32 is the first argument register if the callee wants to reuse the register it could, for example it only needs the argument initially and after that it can be reused as a general register (if th

Re: How do I get IA64 register R32 and above?

Dave Anderson — 2013-04-23T20:28:43


----- Original Message -----

The "bt -f" option uses the rse_function_params() function backported
from the kernel's ia64 unwind facility.  And they do appear to be as
you described, i.e., at the BSP address location.  For example:

 crash> bt -f
  PID: 1      TASK: e0000100ff118000  CPU: 0   COMMAND: "init"
   #0 [BSP:e0000100ff119358] schedule at a000000100678b90
      (void)
   #1 [BSP:e0000100ff119328] schedule_timeout at a00000010067a1b0
      (1388)
   #2 [BSP:e0000100ff119250] do_select at a0000001001b0090
      (e0000100fe518180, e0000100ff11fce0, e0000100ff11fe10)
   #3 [BSP:e0000100ff1191f0] core_sys_select at a0000001001b0990
      (b, 60000fffffdf77a8, 0, 0, e0000100ff11fe10)
   #4 [BSP:e0000100ff119160] sys_select at a0000001001b18b0
      (b, 60000fffffdf77a8, 0, 0, 60000fffffdf7828)
   #5 [BSP:e0000100ff119160] ia64_ret_from_syscall at a00000010000be40
    EFRAME: e0000100ff11fe40
        B0: 400000000000aee0      CR_IIP: a000000000010620
   CR_IPSR: 00001213085a6010      CR_IFS: 0000000000

Re: How do I get IA64 register R32 and above?

Dave Anderson — 2013-04-23T21:15:53

Correction: actually rse_function_params() was written by me,
but what I meant was that it uses the kernel's unw_access_gr()
function to read the argument registers.  Here's the part of
unw_access_gr() that that accesses registers 32 and above:


        } else {
                /* access a stacked register */
                addr = ia64_rse_skip_regs((unsigned long *) info->bsp,
regnum - 32);
                nat_addr = ia64_rse_rnat_addr(addr);
                if ((unsigned long) addr < info->regstk.limit
                    || (unsigned long) addr >= info->regstk.top)
                {
                        error(INFO, "unwind: ignoring attempt to access
register outside of rbs\n");
                        return -1;
                }
                if ((unsigned long) nat_addr >= info->regstk.top)
                        nat_addr = &info->sw->ar_rnat;
                nat_mask = (1UL << ia64_rse_slot_num(addr));
        }

I don't remember much about the ia64 backing store stuff, but
I guess it is poss

Re: How do I get IA64 register R32 and above? [RESEND]

Dave Anderson — 2013-04-23T20:47:55

(my first response from my redhat.com address has been sitting in the
"sent" bucket for ~20 minutes...)

----- Original Message -----

The "bt -f" option uses the rse_function_params() function backported
from the kernel's ia64 unwind facility.  And they do appear to be as
you described, i.e., at the BSP address location.  For example:

 crash> bt -f
  PID: 1      TASK: e0000100ff118000  CPU: 0   COMMAND: "init"
   #0 [BSP:e0000100ff119358] schedule at a000000100678b90
      (void)
   #1 [BSP:e0000100ff119328] schedule_timeout at a00000010067a1b0
      (1388)
   #2 [BSP:e0000100ff119250] do_select at a0000001001b0090
      (e0000100fe518180, e0000100ff11fce0, e0000100ff11fe10)
   #3 [BSP:e0000100ff1191f0] core_sys_select at a0000001001b0990
      (b, 60000fffffdf77a8, 0, 0, e0000100ff11fe10)
   #4 [BSP:e0000100ff119160] sys_select at a0000001001b18b0
      (b, 60000fffffdf77a8, 0, 0, 60000fffffdf7828)
   #5 [BSP:e0000100ff119160] ia64_ret_from_syscall at a00000010000be40
    EFRAME: e0000100ff11fe40

Re: How do I get IA64 register R32 and above?

Bob Montgomery — 2013-04-23T20:03:13

Jay, here's an excerpt from an email I answered in 2006.  It's not clear
that I remember anything about this now :-)   Hope it helps.

Bob Montgomery

--------------------------------



There are three big secrets:

1) The saved values of the stack frame registers (r32, r33, ... up to
the limit set by the alloc statement at the beginning of the function
for that frame) are located at the Backing Store Pointer (BSP) shown for
the frame.  If you want to see r32, r33, r34, r35 for frame #8, do
        crash> x/4xg 0xe00000404ae490e8

2) The saved values of the stack frame registers are their current
values, not necessarily the values passed in to the function (for those
registers that were used as input registers, for example).  In other
words, the first parameter is passed as r32, but r32 might be modified
by that function before the point in the code represented by the stack
trace.  Check for this by disassembling:
        crash> dis mdc_replay_open | grep r32
and looking to see where (if ever) r32 is modif

Re: How do I get IA64 register R32 and above?

Robin Holt — 2013-04-23T06:35:26

The r32... registers are rotating registers.  They change with the
stack frame.  I don't recall how gdb(crash) does it, but they should
be associated with the stack frame.  The registers have special meaning
depending upon the first "instruction" of the function which tells the
processor where to advance the rotating point to.

Take a look at the backtrace.  I think the registers will show up there.

I will look at the ia64 book when I get to the office and see if I
can remember how the rotate works, where the backing store is for
the registers, etc.

Good Luck Jay,
Robin

How do I get IA64 register R32 and above?

Jay Lan — 2013-04-22T23:38:02

Hi,

I got an IA64 vmcore. The stack backtrace only
printed registers up to R31. How do I get the contents
of R32 and above?

Thanks,
Jay

Re: fix bug of struct command

Dave Anderson — 2013-04-12T14:26:31


----- Original Message -----

Luckily it's a relatively rare scenario, and in this particular case,
"task -R fs" still works...

Queued for crash-6.1.7.

Thanks,
  Dave

Re: Threaded crash tool? Is it time?

Dave Anderson — 2013-04-12T13:52:33


----- Original Message -----

Right, I've felt your pain...

But the problem with "kmem -s" on huge systems is that there are typically 
one or more absurdly large individual caches like the inode or buffer head
caches that consume most of the time.  And that's because the command has
to traverse linked lists containing hundreds of thousands of slab structures, 
verifying each one along the way.  In those extreme cases, it's almost worth
just setting aside a crash session window for the one command, and doing any
work in another.

I should also mention that there is an unadvertised workaround for "kmem -s" 
to skip/ignore one or more problematic caches.  For example:

  crash> kmem -s -I buffer_head,inode_cache
  CACHE            NAME                 OBJSIZE  ALLOCATED     TOTAL  SLABS  SSIZE
  ffff88020d648000 nf_conntrack_ffff88020e258000 304     0         0      0     8k
  ffff880209a14100 nfs_direct_cache         208          0         0      0     8k
  ffff880209a14000 nfs_write_data           904