gmane.linux.kernel.perf.user

Re: perf record always record the same kernel symbol

Chao Xu — 2013-05-10T20:43:49

Thanks David. This indeed gave more samples. Do you know why the
symbol  [k] 0xffffffff81043fea occured so often? As shown in the
pasted result, both "swapper" and "perf" had this symbol. And if I
"perf record dd if=/dev/zero of=/dev/null" or "perf record ping", they
all showed this symbol. I pasted these results in the end of this
email.
I guess my real concern is the symbols cannot be mapped into
meaningful function names. These symbols cannot be found in either
/proc/kallsyms nor the vmlinux I built from the kernel source tree.
Any idea? Thanks.

result of sudo perf record -a -- sleep 1
# Samples: 104  of event 'cycles'
# Event count (approx.): 61197061

#
# Overhead          Command               Shared Object  Symbol
# ........  ...............  ..........................  ......
#
    47.49%          swapper  [kernel.kallsyms]           [k] 0xffffffff81043fea
    11.78%  chromium-browse  [kernel.kallsyms]           [k] 0xffffffff81095950
    11.46%  chromium-browse  chromium-browser            [.] 0x00

Re: perf record always record the same kernel symbol

David Ahern — 2013-05-10T20:22:42

take a look at the output of report -v and report -D. For me those 
commands work just fine:

# perf record -a -- sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.104 MB perf.data (~4527 samples) ]

# perf report

  38.39%       swapper  [kernel.kallsyms]   [k] intel_idle
   9.26%          perf  [kernel.kallsyms]   [k] generic_exec_single
   5.58%       swapper  [kernel.kallsyms]   [k] cpuidle_wrap_enter
   4.73%       swapper  [kernel.kallsyms]   [k] hpet_legacy_next_event
   4.47%       swapper  [kernel.kallsyms]   [k] _raw_spin_lock_irqsave
...

David

Re: perf record always record the same kernel symbol

David Ahern — 2013-05-10T19:12:42

you are profiling the 'sleep' command which is not very exciting. Try:
   sudo perf record -a -- sleep 1
   sudo perf report

David

perf record always record the same kernel symbol

Chao Xu — 2013-05-10T19:04:10

Hi,

I issued the following commands:
sudo perf record sleep 1
sudo perf report

Here is a snippet from the output of perf report:
# Samples: 8  of event 'cycles'
# Event count (approx.): 1609581
#
# Overhead  Command      Shared Object  Symbol
# ........  .......  .................  ......
#
   100.00%    sleep  [kernel.kallsyms]  [k] 0xffffffff81043fea

I also tried to record other commands such as dd, ping, the symbol
showed in the output is always "[k] 0xffffffff81043fea", and this was
the only kernel symbol. I checked /proc/kallsyms, this symbol did not
exist in the file.

I built my kernel from linux source tree with the default config. I
tried on both 3.9.0 and 3.9.1. I built perf from
linux-source-tree/tools/perf with a simple "make".

Could someone suggest where did I go wrong? Thanks a bunch.
--
Regards,
Chao Xu

Re: How to monitor sampling over time with Perf

Guru Prasad — 2013-04-28T12:47:51

I don't think this functionality is available in perf..I could be wrong..
I think you would have to add a small piece of code in perf stat to
achieve this..
Follow the execution from cmd_stat until run_perf_stat(I think..)

I will try to answer your question better once I go back to my lab..I
don't have access to the kernel sources currently..



On Sun, Apr 28, 2013 at 8:01 AM, Donitta <sabra_gr< at >yahoo.fr> wrote:

How to monitor sampling over time with Perf

Donitta — 2013-04-28T12:01:24

Hi,
I would like to monitor perf results over time. For example, I would like to 
have all the occurrences of an event and their timestamp of occurrence.
How can I have those infos? I read somewhere in the documentation that perf 
script command provides timestamps for each occurrence, but when I tried this 
command I didn't understand it's output, and there is no documentation. 
Could someone plead provide me examples on how to use perf utilities to answer 
my needs?

Many thnaks

Re:7

dark_footix — 2013-04-26T15:23:36

Sat, 27 Apr 2013 19:21:36 I highly recommend the following site  http://yofuku09.web.fc2.com/default/79It%20s%20p0ssible%20t0%20bec0me%20rich%20w0rking%200n-line47

Re: perf script timestamp

David Ahern — 2013-04-24T13:10:48

times are <seconds>.<microseconds>

David

Re: NumaTOP 1.0 launched

zhou jencce — 2013-04-24T09:00:56

2013/4/24 Jin, Yao <yao.jin< at >intel.com>:

That's OK,  I'd like to try.


Thanks
XIong

perf script timestamp

OSDepend — 2013-04-24T08:22:05

Hi guys,
I just try to use perf script, and get some thing like below:
            java 15893 [015] 96380.588871: timer_cancel: timer=0xffff8804379d6698
            java 15721 [010] 96380.588872: timer_cancel: timer=0xffff8800be3d6698
            java 15893 [015] 96380.588875: timer_expire_entry: timer=0xffff8804379d6698 function=delayed_work_timer_fn now=4391194790
            java 15721 [010] 96380.588876: timer_expire_entry: timer=0xffff8800be3d6698 function=delayed_work_timer_fn now=4391194790

I want to know exactly about the time units of the events, such as 96380.588871, and 96380.588872.
The number befor "." is second or millisecond? and  Is the number after "." nanoseconds or microseconds?
I'm a little confused about the documation for ktime_t, please help me out!
 
Best,
Chen


2013-04-24

simultaneous counter start

Chris Freehill — 2013-04-18T03:37:47

Hi,

Is perf stat supposed to start all specified counters simultaneously?

For example, in  the command:

perf stat -a -e rXXXX,rYYYY sleep 5

I would think that perf would configure for counting XXXX and YYYY,
and then start counting for both, so that the 2 events will have been
collected for the same amount of time.

However, looking at the perf code (and adding some logging) I see that
the first event counter is configured, then started before it then
configures and starts the 2nd event counter. Additionally, each time
it configures and starts, a global disable ("pmu_disable") is called.
So if you have many event specified, there would be a lot of starting
and stopping of all counters.

It seems like you would always want to start all the counters
simultaneously. Am I missing something? Or is there an option I can
specify to make that happen?

thanks,
chris

RE: NumaTOP 1.0 launched

Jin, Yao — 2013-04-18T02:20:02

Hi, 

https://01.org/numatop/ is slightly faster to access.
Or please access https://github.com/01org/numatop to get the source directly. 

Thanks
Jin Yao

-----Original Message-----
From: Jin, Yao 
Sent: Wednesday, April 17, 2013 10:30 AM
To: 'lwn< at >lwn.net'; 'linux-kernel< at >vger.kernel.org'; 'linux-perf-users< at >vger.kernel.org'
Subject: NumaTOP 1.0 launched

We are pleased to announce today that the NumaTOP project has been added to 01.org. 

Performance analysis engineers know that NUMA can seriously impact performance and that NUMA performance analysis can be challenging. We've realized that currently there isn't an easy-to-use tool that lets us easily observe whether NUMA-related issues exist and, if so, where the NUMA bottleneck(s) reside. It can be quite challenging, especially in complex server environments. 

We decided to create a tool that automatically performs the typical steps in NUMA analysis and provides a good starting point to dive in and fix NUMA-related bottlenecks. That's NumaTOP!

NumaTOP is

NumaTOP 1.0 launched

Jin, Yao — 2013-04-17T02:29:31

We are pleased to announce today that the NumaTOP project has been added to 01.org. 

Performance analysis engineers know that NUMA can seriously impact performance and that NUMA performance analysis can be challenging. We've realized that currently there isn't an easy-to-use tool that lets us easily observe whether NUMA-related issues exist and, if so, where the NUMA bottleneck(s) reside. It can be quite challenging, especially in complex server environments. 

We decided to create a tool that automatically performs the typical steps in NUMA analysis and provides a good starting point to dive in and fix NUMA-related bottlenecks. That's NumaTOP!

NumaTOP is an observation tool for runtime memory locality characterization and analysis of processes and threads running on a NUMA system. It helps the user characterize the NUMA behavior of processes and threads and identify where the NUMA-related performance bottlenecks reside. It uses Intel performance counter sampling technologies and associates the performance

Re: Issue about raw_syscalls!

David Ahern — 2013-04-16T16:53:14

perf-trace uses libaudit to do the conversion. Perhaps something can be 
done for perf-script as well.

David

Issue about raw_syscalls!

OSDepend — 2013-04-16T15:17:22

Hi all,

I recently use perf "-e  raw_syscalls:sys_enter  -e raw_syscalls:sys_exit" to trace all the syscall entry and exit.
After i perf script the data, I get result like below:
            perf 10258 [013] 260748.823215: sys_exit: NR 0 = 0
            perf 10258 [013] 260748.823244: sys_enter: NR 59 (7fffd5b085bf, 7fffd5b0bb10, 12f7030, 3266018240, 65726f632d667265, 2811)
            perf 10258 [013] 260748.823280: sys_exit: NR 59 = -2
            perf 10258 [013] 260748.823283: sys_enter: NR 59 (7fffd5b085b9, 7fffd5b0bb10, 12f7030, 3266018240, 6e69622f75706363, 2811)
         runspec 10258 [019] 260748.823995: sys_exit: NR 59 = 0
         runspec 10258 [019] 260748.824028: sys_enter: NR 12 (0, 41b590, 0, 64, 7fffa13ff000, 37f)
         runspec 10258 [019] 260748.824031: sys_exit: NR 12 = 7761920
         runspec 10258 [019] 260748.824139: sys_enter: NR 9 (0, 1000, 3, 22, ffffffff, 0)
         runspec 10258 [019] 260748.824147: sys_exit: NR 9 = 140288080863232
         runspec 10258 [019] 260748.824167: s

how perf handles (or should handle) off-core events

Freehill Christopher-RAT063 — 2013-04-12T14:52:55

Hi,

I am seeing inconsistencies when I use perf to measure off core events with the -a option, compared with using a probe. For example

$ perf -a -A -e rXXXX sleep 5

I am running a simple program that copies arrays. The rXXXX event above counts memory accesses.

For comparison, I have a probe from which I can measure the events. To see if the measurements are equal, I am comparing events/second collected by each method, while the simple array-copy program is running. What I am seeing is that the counts provided by perf are 2X to 3X smaller for the memory accesses.

My theory is that for non-core events, when perf is handling counter overflows via interrupt in one of the cores, the other cores are still active, but the events they are generating are not being counted, because perf is busy handling a counter overflow. 

I've tried various tests, but have not been able to prove/disprove this theory. 

If the theory is correct, is there already a way to deal with this issue? Seems like you would have to susp

Re: No tracepoint event in the perf list output

Namhyung Kim — 2013-04-11T12:20:22

Hi Donitta,

On Mon, 8 Apr 2013 09:51:32 +0000 (UTC), Donitta wrote:

Do you have CONFIG_EVENT_TRACING enabled on your kernel?

Thanks,
Namhyung

No tracepoint event in the perf list output

Donitta — 2013-04-08T09:51:32

Hi,

 According to the perf documentation, when running "perf list" command we 
should have all pre-defined events that could be hardware, software, hardware 
cache, raw hardware event descriptor, hardware breakpoint or tracepoint event.
In my case, when I executed the perf list command I had all type of events 
listed except the tracepoint event. 
What does this mean?

PS: I'm using perf on an embedded OS on ARM cortex 9 core.

Best regards

Re: Fw: Re: Re: Howto make perf probe work

Namhyung Kim — 2013-04-04T06:05:54

Hi,

On Thu, 4 Apr 2013 01:28:34 +0800, OSDepend wrote:

Hmm.. it seems not, currently.

Maybe we can change perf to output the symbol was not able to be probed
after checking symbol address and it's between
__kprobes_text_{start,end}.

I'll take a look at it later.

Thanks,
Namhyung

RE: perf event attrributes

Freehill Christopher-RAT063 — 2013-04-03T22:26:38

Thanks Namhyung. Installed Ubuntu 12.10 (with 3.5.0-17-generic) on VirtualBox. I'm not seeing this folder is it later than this?

Also, are you aware of any documentation on the usage?

Thanks,
Chris

Re: Re: Fw: Re: Re: Howto make perf probe work

OSDepend — 2013-04-03T17:28:34

Hi,



That's works nicely for me! 
Thanks a lot!!

Is there any guide for us to decide which tracepoint (or other kernel symbol) canbe used, and which cannot? 
For some of the kallsyms cannot be used directly. Such as  do_raw_spin_lock.
perf probe -a test=do_raw_spin_lock
    Fatal: Kernel symbol 'do_raw_spin_lock' not found - probe not added. 

Best,
Chen