gmane.linux.oprofile

Revive this old patch

Mark Pearson — 2013-05-21T19:54:33

I'd like to revive this old patch from 2008:

    diff -u a/utils/opcontrol b/utils/opcontrol
    --- a/utils/opcontrol    2008-11-11 14:31:25.000000000 +0100
    +++ b/utils/opcontrol        2008-11-11 12:30:42.000000000 +0100
    < at >< at > -187,7 +187,7 < at >< at >
                     fi
             fi
             mkdir /dev/oprofile >/dev/null 2>&1
    -       grep oprofilefs /etc/mtab >/dev/null
    +       grep oprofilefs /proc/mounts >/dev/null
             if test "$?" -ne 0; then
                     mount -t oprofilefs nodev /dev/oprofile >/dev/null
             fi
    < at >< at > -1607,7 +1607,7 < at >< at >
      do_deinit()
      {
             # unmount /dev/oprofile if it is mounted
    -       OPROF_FS=`grep /dev/oprofile /etc/mtab`
    +       OPROF_FS=`grep /dev/oprofile /proc/mounts`
             if test -n "$OPROF_FS"; then
                     umount /dev/oprofile
             fi
    < at >< at > -1705,7 +1705,7 < at >< at >
      check_version()
      {
             OPROFILE_AVAILABLE=no
    -       grep oprofilefs /etc/mtab >/dev/null

Re: [PATCH] Add support for Intel Netburst (e.g., Pentium P4) to operf

Maynard Johnson — 2013-05-21T17:36:39

See http://oprofile.git.sourceforge.net/git/gitweb-index.cgi for a list of all 4 sub-projects of oprofile.  The testsuite can be obtained using the following command:

    git clone git://oprofile.git.sourceforge.net/gitroot/oprofile/oprofile-tests

This looks very similar to what I see on my Intel Core 2 Duo when comparing opcontrol results to operf results -- 1) the number of operf samples in memcpyt.c:main is about triple what I get from opcontrol; 2) about 30% more vmlinux samples with operf; 3) number of samples in libc:memcpy is roughly equivalent between operf and opcontrol.  So I think the P4 patch looks to be good, so far.  I will commit soon unless I get some review comments from others in the community.  Thanks again for testing it.

-Maynard


------------------------------------------------------------------------------
Try New Relic Now & We'll Send You this Cool Shirt
New Relic is the only SaaS-based application performance monitoring service 
that delivers powerful full stack analytics. Optim

Re: [PATCH] Add support for Intel Netburst (e.g., Pentium P4) to operf

Gilles Allard — 2013-05-21T17:22:53


Is this testsuite included in the master branch of official repository ? I 
didn't find it in the copy I got using "git clone" ( I just found a few test 
programs in some lib* sub-directories )


Using the "memcpyt" test program, the results printed by "opreport --symbols" 
are as follows

First "opcontrol based" profiler:

CPU: P4 / Xeon, speed 2e+06 MHz (estimated)
Counted GLOBAL_POWER_EVENTS events (time during which processor is not 
stopped) with a unit mask of 0x01 (mandatory) count 100000
samples  %        image name               symbol name
813263   97.3844  libc-2.11.1.so           memcpy
12138     1.4535  no-vmlinux               /no-vmlinux
9699      1.1614  memcpyt                  main
2        2.4e-04  ld-2.11.1.so             do_lookup_x
1        1.2e-04  ld-2.11.1.so             _dl_catch_error
1        1.2e-04  ld-2.11.1.so             _dl_relocate_object
1        1.2e-04  ld-2.11.1.so             malloc
1        1.2e-04  libc-2.11.1.so           __rpc_thread_destroy

Then "operf based" o

Re: [PATCH] Add support for Intel Netburst (e.g., Pentium P4) to operf

Maynard Johnson — 2013-05-21T12:31:41

Gilles,
The oprofile-tests have some simple workloads used in the oprofile testsuite.  Attached is the "memcpyt.c" workload, slightly modified to run a bit longer than normal.  This little testcase should give you better repeatable results.  If you can find time to run opcontrol and operf against it, I would appreciate it.
I'm not quite sure I understand exactly what problem you're having, but a simple test I did of adding a SIGUSR1 handler to the memcpyt.c program seemed to work OK (when I sent the USR1 signal to memcpyt while operf was profiling it).  Please work up a minimal testcase that exhibits the problem and post it as a new thread to this list, with all details we'll need to reproducee.

Thanks.

-Maynard


/* memcpyt.c
 *  Copyright (C) 2012 IBM

 * This file is free software; you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation; either version 2 of the License, or
 * (at your option) any later version.
 *
 * T

Re: [PATCH] Add support for Intel Netburst (e.g., Pentium P4) to operf

Gilles Allard — 2013-05-21T12:12:33

I cannot say that the results in both cases ( "operf-based" & "opcontrol-
based" profiler ) are pretty close : overall number of samples & number of 
samples for each binary file are different for each test ( at least for the 
lines showing a great number of samples ) but the percentages for each binary 
file are rather similar.
I think that this can be partly explained by the fact that stopping the 
application is done by a mouse click on a button ( this application is a small 
Qt widget )

I can send a copy of the results printed by opreport if needed ( about 210 
lines each )

But another problem occurs during the test with operf. In the first release, 
the application processes its argument when it receives "SIGUSR1"; its 
behaviour was as expected with "opcontrol" but the test program doesn't 
receive the signal when under control of "operf" ( so I had to slightly modify 
the code to get some results ). The Oprofile manual doesn't say anything about 
this point and I may have missed something.

Let me

[PATCH] Fix Coverity errors found on May 20, 2013 git snapshot

Maynard Johnson — 2013-05-20T22:41:54

Fix Coverity errors found on May 20, 2013 git snapshot

Coverity identified the following errors on scans run from May 7 through
May 20, 2013:

Type,Category,File,Function
Wrapper object use after free,Memory - illegal accesses,/agents/jvmpi/jvmpi_oprofile.cpp,compiled_method_load(JVMPI_Event *)
Unchecked return value,Error handling issues,/daemon/opd_mangling.c,opd_open_sample_file
Dereference after null check,Null pointer dereferences,/daemon/opd_sfile.c,sfile_hash
Uninitialized scalar field,Uninitialized members,/gui/oprof_start_config.cpp,config_setting::config_setting()
Division or modulo by zero,Integer handling issues,/libdb/db_stat.c,odb_hash_stat
Resource leak,Resource leaks,/libop/op_cpu_type.c,_auxv_fetch
Resource leak,Resource leaks,/libop/op_cpu_type.c,fetch_at_hw_platform
Negative array index read,Memory - illegal accesses,/libop/op_events.c,_is_um_valid_bitmask
Write to pointer after free,Memory - corruptions,/libop/op_events.c,read_events
Read from pointer after free,Memory - illegal accesses

Re: [PATCH] Add support for Intel Netburst (e.g., Pentium P4) to operf

Maynard Johnson — 2013-05-20T13:36:42

Great!
You should be able to compare opcontrol and operf results pretty easily by doing something like the following:

1. rm /root/.oprofile/daemonrc (delete opcontrol cache file so we're starting from scratch)
2. rm -rf oprofile_data (to make sure opreport doesn't "see" profile data collected by previous runs of operf)
3. opcontrol --reset
4. opcontrol --start --no-vmlinux --separate=lib,kernel -e GLOBAL_POWER_EVENTS:100000
5. Run a specific benchmark or application that you can trust will yield repeatable results.
6. opcontrol --deinit
7. opreport --symbols > op1.out

8. operf -e GLOBAL_POWER_EVENTS:100000 <your benchmark or app>
9. opreport --symbols > op2.out
10. Compare op1.out and op2.out -- the number of samples per binary file should be pretty close.

Thanks!
-Maynard


------------------------------------------------------------------------------
AlienVault Unified Security Management (USM) platform delivers complete
security visibility with the essential security capabilities. Easily and
efficientl

Re: [PATCH] Add support for Intel Netburst (e.g., Pentium P4) to operf

Gilles Allard — 2013-05-19T15:43:18

I just inserted the patch you provided into the Oprofile code ( taken from the 
official repository )
Running the previous simple test ( operf using default event on P4 ) and with 
that patch, operf runs fine : no problem at startup, usable results. Operf 
behaves as expected.

But I don't see how I can compare results of "opcontrol based" & "operf based" 
profilers; they seem to me too different even if I run the test in the same 
conditions ( same application, same preloaded libraries, same "opreport" 
command ). Can you suggest a way to make such a comparison ?

And I intend very shortly to do some other tests ( with different events ).

Gilles Allard


------------------------------------------------------------------------------
AlienVault Unified Security Management (USM) platform delivers complete
security visibility with the essential security capabilities. Easily and
efficiently configure, manage, and operate all of your security controls
from a single console and one unified framework. Download a

Re: [PATCH] Add support for architected events for IBM ppc64architecture (for ISA 2.07)

Carl E. Love — 2013-05-18T00:39:37

OK, so for powerN and powerN+ the are the same for the OProfile events
any we return the base type.  That is true for Power7 and Power7+.
Hopefully that will be true from here on out. That wasn't always true
but for the older processors where it wasn't true oprof isn't supported
so that doesn't matter here.  

If you get powerN+1 running in compat mode powerN the string lengths are
equal so you use the CPU_PPC64_ARCH_V1 events.  There was no compat mode
for PowerN and PowerN+.  The processor type returned when you have
PowerN+1 running in compat mode is PowerN.  As long as that stays true
for the HW designs in the future, this should all work since the string
lengths will be equal. Eventually we may get to power10 running in
compat mode of power9 and you will be in trouble.  

When adding new Processor type, especially for the + revs of a
processor, this code will need to be looked at to make sure the hardware
is obeying the implied assumptions in this code.  For now it should be
fine, we will have to deal

Re: [PATCH] Add support for architected events for IBM ppc64architecture (for ISA 2.07)

Maynard Johnson — 2013-05-18T00:25:16

Carl,
I found another not-so-minor problem. Please review this patch.

Thanks!

-Maynard

-----------------------------------

Fix IBM architected events to work on IBM POWER7+

In situations where oprofile is running on a kernel that does not
have full native support for an ISA 2.07-based ppc64 processor,
but does have the base level architected support, oprofile userspace
code uses the auxiliary vector of the operf program and compares
AT_PLATFORM and AT_BASE_PLATFORM values to see whether or not it
should use the new ppc64 architected events.  However, if
AT_PLATFORM="power7" and AT_BASE_PLATFORM="power7+", the oprofile
code was erroneously identifying this as a situation where the
architected events should be used.  This patch fixes that problem.

Signed-off-by: Maynard Johnson <maynardj< at >us.ibm.com>

Index: oprof-work/libop/op_cpu_type.c
===================================================================
--- oprof-work.orig/libop/op_cpu_type.c
+++ oprof-work/libop/op_cpu_type.c
< at >< at > -254,8 +254,22 < at >< at > stati

Re: [PATCH] Add support for Intel Netburst (e.g., Pentium P4) to operf

Maynard Johnson — 2013-05-17T16:43:44

Can someone with a Pentium P4 processor please test this patch?

Thanks.

-Maynard

-------------------------------------------------------------------------
On 05/15/2013 10:35 AM, Maynard Johnson wrote:


------------------------------------------------------------------------------
AlienVault Unified Security Management (USM) platform delivers complete
security visibility with the essential security capabilities. Easily and
efficiently configure, manage, and operate all of your security controls
from a single console and one unified framework. Download a free trial.
http://p.sf.net/sfu/alienvault_d2d

Re: [PATCH] Add support for architected events for IBM ppc64architecture (for ISA 2.07)

Maynard Johnson — 2013-05-17T15:32:15

Thanks!  Fixed.

-Maynard


------------------------------------------------------------------------------
AlienVault Unified Security Management (USM) platform delivers complete
security visibility with the essential security capabilities. Easily and
efficiently configure, manage, and operate all of your security controls
from a single console and one unified framework. Download a free trial.
http://p.sf.net/sfu/alienvault_d2d

Re: [PATCH] Add support for architected events for IBM ppc64architecture (for ISA 2.07)

Carl E. Love — 2013-05-17T15:21:28

Maynard:

Looks good with just one minor typo.  The copyright for the new events
file is updated to 2013, see below.  But the copyright on the new unit
masks is 2009, I think you intended to make it 2013.  :-)

               Carl Love

<snip>


<snip>




------------------------------------------------------------------------------
AlienVault Unified Security Management (USM) platform delivers complete
security visibility with the essential security capabilities. Easily and
efficiently configure, manage, and operate all of your security controls
from a single console and one unified framework. Download a free trial.
http://p.sf.net/sfu/alienvault_d2d

[PATCH] Add support for architected events for IBM ppc64 architecture(for ISA 2.07)

Maynard Johnson — 2013-05-17T14:05:56

The patch below is ppc64-specific functionality. As the ppc64
maintainer for oprofile, I have taken the liberty of committing
it already.  But as usual, I'd be happy to respond to any
review comments.


---------------------------------------------------------

Add support for architected events for IBM ppc64 architecture

The Power ISA 2.07, recently published at http://power.org/,
formally defines base performance monitoring facilities which
must be provided by any processor implementation of the ISA.
Specific implementations may provide additional features, but
must include the standard architected features.

This patch creates a generic ppc64 cpu type called
"ppc64/architected_events_v1" that has a list of events which
are defined in the ISA 2.07 performance monitoring unit
architecture section.  This new generic type will only be
supported by operf.  It will *not* be supported by the legacy
oprofile kernel driver and opcontrol-based profiler.  This
new cpu type can be used in situations where oprofile i

Re: [PATCH] Fix Coverity issues identified against oprofile 0.9.8release

Maynard Johnson — 2013-05-15T17:41:54

I decided I should review the latest Coverity scan that I ran last week to see if
any of the issues identified in Will's scan (from Jan 15) that I tried to address in this
patch are still problematic.  And in fact, I did find a few.  See below.

I fixed those issues (I hope) and committed the patch.  I'll re-run Coverity to see what's left to fix.

-Maynard


Defining and using OP_JIT_HEADER_SIZE did not stop Coverity from making the following complaint
about opagent.c:

 Uninitialized value "header": field "header"."bfd_target" is uninitialized when calling "fwrite(...)

Our code is doing the right thing -- I just don't know how to silence Coverity, except to mark it
as a "false positive". So I will revert the above changes in jitdump.h, as well as the change
in opagent.c where OP_JIT_HEADER_SIZE was being used.

Should be 'delete [] buffer'
Oops!  This close() should be under label 'out', not 'out_fail'.

Also need to set ;section=entries_address_ascending[0]->section'.


---------------------------------

[PATCH] Add support for Intel Netburst (e.g., Pentium P4) to operf

Maynard Johnson — 2013-05-15T15:35:05

Hello, Giles,
I don't have a Pentium P4 to test out this patch, so can you please apply it to
the latest oprofile source and test it out.  You can get the latest oprofile
source from:
git clone git://oprofile.git.sourceforge.net/gitroot/oprofile/oprofile

Please compare results with the opcontrol-based profiler.

Thanks for your patience.

-Maynard

----------------------------------------------------
Add support for Intel Netburst (e.g., Pentium P4) to operf

The "legacy" oprofile kernel driver has special "p4" handling. There's
a map of event codes to ESCR/CCCR values.  Unfortunately, the P4 event
codes (stored in events/i386/p4/events) that are used by the oprofile
kernel driver don't match what perf_events kernel code expects.  This
patch adds some p4-specific event code handling to operf in order to
generate the correct encoding to pass to perf_events kernel subsystem.

Signed-off-by: Maynard Johnson <maynardj< at >us.ibm.com>
---
 libop/Makefile.am      |    4 +-
 libop/op_netburst.c    | 1597 +++++++++++++

Re: Some help needed about operf

Andi Kleen — 2013-05-10T15:56:39

p4 and p4_ht2 I think

There were more P4 based micro architectures, but they all seem to map
to those (williamette, northwood, prescott, presler)

-Andi

------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and 
their applications. This 200-page book is written by three acclaimed 
leaders in the field. The early access version is available now. 
Download your free book today! http://p.sf.net/sfu/neotech_d2d_may

Re: Some help needed about operf

Maynard Johnson — 2013-05-10T14:56:45

The "old driver" will, in fact, go away at some point in the future . . . but will be around for at least one more release.

TI think this would be a fairly easy problem to fix.  The main issue (for me, anyway -- not knowing Intel arch very well) is which Intel cpu_types other than i386/p4 need the special "p4" handling.

-Maynard


------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and 
their applications. This 200-page book is written by three acclaimed 
leaders in the field. The early access version is available now. 
Download your free book today! http://p.sf.net/sfu/neotech_d2d_may

Re: [PATCH] Fix Coverity issues identified against oprofile 0.9.8release

Maynard Johnson — 2013-05-10T12:38:47

Review comments are welcome.  If I don't get any responses by early next week, I'll go ahead and apply the patch.

-Maynard


------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and 
their applications. This 200-page book is written by three acclaimed 
leaders in the field. The early access version is available now. 
Download your free book today! http://p.sf.net/sfu/neotech_d2d_may

Re: Some help needed about operf

Andi Kleen — 2013-05-09T17:00:07

Sorry I don't know much about P4 and don't have capabilities to support it.

My suggestion would be to just let it use the old driver, which is not
going away.

-Andi

------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and 
their applications. This 200-page book is written by three acclaimed 
leaders in the field. The early access version is available now. 
Download your free book today! http://p.sf.net/sfu/neotech_d2d_may

Re: Some help needed about operf

Maynard Johnson — 2013-05-08T21:37:22

For most architectures (including x86*), operf has been using the existing event codes as a basis for what we pass to the perf_events kernel subsystem.  Looking at the "legacy" oprofile kernel driver, I see special "p4" handling. There's a map of event codes to ESCR/CCCR values.  Unfortunately, the event codes used by the oprofile kernel driver don't match what perf_events kernel code expects.

We'll need to have some special event handling code for operf for P4 and the like -- potentially use libpfm to get the code.  *Andi and Suravee* are our Intel/x86_64 experts in the community, so I will let one of them drive this problem, but I can certainly help as needed.

-Maynard


------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and 
their applications. This 200-page book is written by three acclaimed 
leaders in the field. The early access version is available now. 
D