gmane.linux.kernel.numa

[PATCH] Fix segfault in read_distance_table when no NUMA

Ian Wienand — 2013-06-19T11:56:08

Hi

Debian bug #712692 reported a segfault on systems with no numa info.
In this case, read_distance_table would exit with "0" which was not
picked up as an error by numa_distance(), leading to a dereference of
distance_table.

Signed-off-by: Ian Wienand <ian< at >wienand.org>
---
 distance.c |    4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/distance.c b/distance.c
index 4a26972..ec423c4 100755
--- a/distance.c
+++ b/distance.c
< at >< at > -58,15 +58,13 < at >< at > static int read_distance_table(void)
 int *table = NULL;
 int err = -1;
 
-for (nd = 0;; nd++) {
+for (nd = 0; nd<maxnode; nd++) {
 char fn[100];
 FILE *dfh;
 sprintf(fn, "/sys/devices/system/node/node%d/distance", nd);
 dfh = fopen(fn, "r");
 if (!dfh) {
 if (errno == ENOENT)
-err = 0;
-if (!err && nd<maxnode)
 continue;
 else
 break;

NUMA performance, optimal settings for multithreaded, memory mapped file access

Andreas Mueller — 2013-05-22T09:22:52

Hello,

I have rather strange test results with different numa settings. My application uses multiple threads and (simplified) accesses data in a memory mapped file. I tried to find optimal settings and see the following effects with 40 threads (on a NUMA hardware with 8 nodes with 6 CPUs per node):

Independently of the number of assigned nodes, the runtime is nearly the same (25 +/- 3 seconds). The used CPU (showed in top) is nearly 100% * number of processors (e.g. 600% for one node up to 4000% for 8 nodes, here not 4800, because I use only 40 threads).

The user CPU (measured with time) is around 1m20 (+/-5s). The system  CPU is 1m20 with 1 node and increases (3min for 2 nodes, 6min for 3 nodes etc. up to 15min for 8 nodes).
So every gain with additional nodes/cpus is completely wasted in system CPU.

I made some other tests with only CPU assignments. If I assign only 1 CPU, the test needs 47s. Optimum is reached at 4 CPUs (on 2 different nodes) with 16s. This is even much better than any of the results

Re: How does numa_parse_nodestring +0-1 works/should work

Andi Kleen — 2013-05-16T12:11:26

cpuset refers to the cpusets in 
http://man7.org/linux/man-pages/man7/cpuset.7.html
You seem to think it refers to the plain task affinity, which is not the
case.

In general generic relative NUMA policy descriptions would be nice
(like in OpenMP4) but we only have them with cpusets currently.

-andi

How does numa_parse_nodestring +0-1 works/should work

Andreas Mueller — 2013-05-16T12:00:40

Hi,

could someone please explain the behaviour of a nodestring "+0-1" or similar for numa_parse_nodestring?

According to the manual, "a leading + can be used to indicate that the node numbers in the list are relative to the task's cpuset".

What I expected and want to achieve is, that if a process was started on node 3, I will get a node mask of 3,4.
What I get (on a kernel 2.6.32) is always 0,1 independently of the node the process was started (either by chance or by assigning a specific node with numactl).

Background: I want to limit a multithreaded process to a subset of the available nodes, but want to include the node it currently runs on, because I assume that the memory (stack/heap) already in use by the process is mainly on this node, so it makes sense to include this node. I know I could simply start the process with numactl, but I want to implement a useful default behavior and for the kind or process this is neither achieved by limiting to one node nor to all nodes.

Thanks in advance for any h

[PATCH] libnuma: prototype for numa_bitmask_weight()

Cliff Wickman — 2013-02-07T21:51:32

From: Cliff Wickman <cpw< at >sgi.com>

Just a minor tweak. An omission pointed out to me.
The function numa_bitmask_weight() is missing a prototype in numa.h

Signed-off-by: Cliff Wickman <cpw< at >sgi.com>
---
 numa.h |    1 +
 1 file changed, 1 insertion(+)

Index: numactl-test/numa.h
===================================================================
--- numactl-test.orig/numa.h
+++ numactl-test/numa.h
< at >< at > -53,6 +53,7 < at >< at > struct bitmask *numa_bitmask_clearall(st
 struct bitmask *numa_bitmask_setbit(struct bitmask *, unsigned int);
 struct bitmask *numa_bitmask_clearbit(struct bitmask *, unsigned int);
 unsigned int numa_bitmask_nbytes(struct bitmask *);
+unsigned int numa_bitmask_weight(const struct bitmask *);
 struct bitmask *numa_bitmask_alloc(unsigned int);
 void numa_bitmask_free(struct bitmask *);
 int numa_bitmask_equal(const struct bitmask *, const struct bitmask *);

Urgent: numactl-2.0.8 - make does not create numastat binary

Andreas Hollmann — 2013-01-10T14:23:25

Hi,

I've noticed that make doesn't create a binary
for numastat with numactl-2.0.8.

The tar file contains already an empty file "numastat"
with a timestamp as recent as in the c file. Therefore,
make doesn't create a binary for numastat. Making
numastat works after running "make clean".

First distributions like Arch ship already numactl-2.0.8
where numastat is not working. So there should be a
new release without this issue.

btw: The new numastat is quite nice!

Best regards,
Andreas Hollmann

libnuma - data types problems

Fabio Tordini — 2013-01-07T10:46:36

Hello,

I'm using libnuma for an enhanced NUMA-aware allocator. There are some 
functions which do not work as explained in the 'numa man page' or 
libnuma APIs. Precisely, I'm having problems with these functions: 
|numa_get_run_node_mask()| and |numa_node_to_cpus()|.

Concerning the former, reading the API and the manual, it returns a 
|struct bitmask *|, while the compiler asks for a |nodemask_t|.

The latter is supposed to require two parameters, as from API 
definition, but the compiler asks for three. If I pass three arguments 
as it asks - |(int node, unsigned long *buffer, int bufferlen) -| I get 
a /segfault/ every time, no matter the /bufferlen/ I use.

Maybe I'm doing something wrong, but I can't see what, since I'm 
trivially following APIs directives. I wonder whether there are some 
discrepancies between APIs and source code that have changed through 
libnuma versions. (btw, how do I check which libnuma/numactl is running 
on the machine? I working remotely on a server and |numactl| doesn't 
se

Re: [PATCH] Custom interleaved allocation

Vasileios Karakasis — 2012-11-27T16:39:37

Hi,

On 11/26/2012 10:21 PM, Andi Kleen wrote:

Definitely, this was just a proof-of-concept of my patch.


void *numa_alloc_interleaved_custom(size_t size, size_t *parts, size_t
nr_parts, int *nodes);

User-specified custom interleaved allocation.

This function allocates a continuous region of `size' bytes (rounded up
to the system's page size) that are interleaved in `nr_parts' partitions
across the system's memory nodes according to the interleaving
specification passed in the `parts' and `nodes' arrays. The size of each
partition in bytes and the node that it must lie on are passed in the
`parts' and `nodes' arrays, respectively, both of size `nr_parts'. For
example, the i-th partition of size `parts[i]' will be placed on the
node `nodes[i]'. This function takes care of the correct alignment of
each partition to the system's page size and the `parts' array is
updated accordingly. The page at the boundary of two partitions is
assigned to the partition owning the largest part, provided that each
partition

Re: [PATCH] Custom interleaved allocation

Andi Kleen — 2012-11-26T20:21:06

You cannot handle errors like this in the library.

It's not fully clear to me what the interface of your proposed new function 
is. Can you send a spec of the new user interface in a manpage like format?

Also it would be good to have some more information on the use case.

My first reaction is that you can create any interleaving scheme you 
want anyways using the low level functions.

-Andi

[PATCH] Custom interleaved allocation

Vasileios Karakasis — 2012-11-21T18:54:56

Hi,

I'm submitting a patch that I found quite useful in my case. It supports
a user-specified custom interleaved allocation scheme. More
specifically, apart from the required allocation size, the user
specifies an arbitrary set of partitions and the set of nodes, where
each partition must lie on. The interleaved allocator will allocate the
required region with a single mmap() call and will then bind the
different partitions on the specified nodes, taking care of the correct
alignment of the partitions and the possible merging of very small ones.

The key motivation behind this patch is the seamless conversion of
existing multithreaded shared-memory-based code to NUMA-aware. For
example, suppose you have a multithreaded code acting on a large dataset
that you split using your own static load balancing scheme. Converting
this code to NUMA-aware using only numa_alloc_onnode() calls would
require considerable refactoring, since you need to alter your algorithm
toward a more distributed logic. The allocation sch

Re: numactl using it's own affinity to determine CPU set

Petr Holasek — 2012-10-23T21:17:08

Thank you!

I'll check it.

Petr

Re: numactl using it's own affinity to determine CPU set

Cliff Wickman — 2012-10-23T14:19:18

Hi Petr,

On Tue, Oct 23, 2012 at 02:21:40PM +0200, Petr Holasek wrote:

Memory nodes allowed was made cpuset-aware by the below patch.
I'm not quite sure if this is the change you are asking about.

patch 0804_ls_patch2
From: Lee Schermerhorn <lee.schermerhorn< at >hp.com>
Date: Wed, 02 Apr 2008 14:34:37 -0400

Depends-on:  MPOL_F_MEMS_ALLOWED kernel patch
Provide libnuma API for MPOL_F_MEMS_ALLOWED flag.
Return nodes allowed by the application's current cpuset context
via new API numa_get_mems_allowed().

+.BR numa_get_mems_allowed()
+returns the mask of nodes from which the process is allowed to allocate
+memory in it's current cpuset context.

-Cliff

Re: numactl using it's own affinity to determine CPU set

Petr Holasek — 2012-10-23T12:21:40

Hi everyone,

On Mon, 22 Oct 2012, Jon Stanley wrote:

I've added new interface for parsing cpusets/nodesets with _all() suffix
in 2.0.8-rc5 libnuma, but numactl utility still uses old ones with cpuset
awareness. I didn't want to dig into some libnuma internals and change
behaviour of dependent utils in such way.


I like the idea of some overriding option proposed by Andi.

btw, Cliff, can you remember what was the reason for this change in some
version after 1.0?

Petr

Re: numactl using it's own affinity to determine CPU set

Jon Stanley — 2012-10-22T20:12:45

I'll happily admit to not being an expert here :)

However, I thought that the concept of cpusets was that if you
attempted to set affinity outside of your cpuset, that would silently
fail. I assume that the reason that libnuma didn't want to set outside
of it's own affinity mask is to avoid such a silent failure?

Also, I question the Real World(TM) prevalence of cpusets.


Not sure if I'd call it a bug since it was intentional, but sure would
call it an unexpected change in behavior :)

Re: numactl using it's own affinity to determine CPU set

Andi Kleen — 2012-10-22T19:41:41

That was done intentional at some point to handle cpusets.
numactl 1.0 or so didn't have that problem.

However it's unclear if it's really bug.

Presumably could have an option to override.

-Andi

numactl using it's own affinity to determine CPU set

Jon Stanley — 2012-10-22T19:35:42

If you set the affinity of your shell to be a subset of the available
CPUs, and then attempt to use numactl to bind to something that is not
in that affinity mask, the attempt will fail. I thought that this was
fixed in the current numactl 2.0.8, but it doesn't seem to be.

A simple reproducer on a 2-node 8-core system is as such:

$ numactl --physcpubind=5 ls
<see directory goodness>
$ taskset -p f $$
$ numactl --physcpubind=5 ls
libnuma: Warning: cpu argument 5 is out of range
<snip>

Let me know if more info is required.

Re: [PATCH] Do not use cpu_to_node() to find an offlined cpu's node.

Peter Zijlstra — 2012-10-19T11:18:04

Would this mean we have to remap (and memcpy) per-cpu memory on
node-plug?


Yeah, none of this should be anywhere near ACPI, its got nothing to do
with ACPI. Furthermore it should be be same across all archs, not just
be weird and wonderful for x86.

Re: [PATCH] Do not use cpu_to_node() to find an offlined cpu's node.

Tang Chen — 2012-10-18T02:51:31

Hi David,

We are working on this problem. Since it is complicated, it really
takes us some time. Sorry for the delay. :)

Actually, we intend to clear cpu-to-node mappings when a whole node is
removed. But the node hot-plug code is still under development, so I
think Wen will send a fix patch soon. :)

Thanks.

Re: [PATCH] Do not use cpu_to_node() to find an offlined cpu's node.

David Rientjes — 2012-10-18T00:52:51


Ok, so it's been a week and these patches are still in -mm.  This is what 
I was afraid of: patches that both Peter and I nacked sitting in -mm and 
allow a NULL pointer dereference because no alternative patch exists yet 
to fix the issue correctly.

Tang and Wen, are you intending on addressing these problems (i.e. not 
touching the acpi code at all and rather clearing cpu-to-node mappings at 
node hot-remove) as we've discussed or do I need to do it myself?

Thanks.

Re: [PATCH] Rewrite numastat to also show per-node process stats

Cliff Wickman — 2012-10-12T16:49:03

Hi Andi,

On Fri, Oct 12, 2012 at 05:49:35PM +0200, Andi Kleen wrote:

Yes, test/regress-io still fails for me.

testing ./node-parse file:.
libnuma: Warning: Cannot find block device 7:7 in sysfs for `.'
Failed to convert `file:.'
failed
testing ./node-parse ip:8.8.8.8
libnuma: Warning: Kernel does not know node mask for device
`/devices/pci0001:00/0001:00:04.0/'
Failed to convert `ip:8.8.8.8'
failed
...

I could use some help to fix that.  I haven't dug into it.
But IMO that shouldn't hold up all the other fixes that have been
accumulated in 2.0.8.

Re: [PATCH] Rewrite numastat to also show per-node process stats

Andi Kleen — 2012-10-12T15:49:35

I'm not sure, did the IO affinity ever work on your SLES system?
iirc there were still some issues.

-Andi