gmane.linux.kernel.virtualization.lguest

Re: [PATCH 2/2] kvm-s390: implement config_changed forvirtio on s390

Rusty Russell — 2008-11-18T14:34:40

Thanks, both applied. Rusty.

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatch changes

H. Peter Anvin — 2008-11-14T03:22:24

The reason for the multiple curves is to show the range of uncertainty. There are three sets of graphs in there: black (current mainline, 16-byte stubs), red (4-byte stubs with a double jump), and blue (8-byte stubs.) The fact that the system is idle is pretty much a case which should favor "blue" over "red"; realistically I think the graphs show that they are identical within the limits of measurement, and both are significantly better than "black". Since this is pretty much the optimal case for "blue" and it doesn't show any performance win over "red", I implemented the "red" option and pushed it into tip:x86/irq. Same here, which is why I wanted to check them both out. -hpa

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatch changes

Matt Mackall — 2008-11-14T02:29:36

Still confused. If, say, the top blue line on the left represents the same number of interrupts as the bottom red one, then at some point it must cross under the red one as it goes to the right, which it does not appear to do. Thus, it does not appear the scale on the left is actually in units of constant probability, no? Though I'll agree that even if they're not scaled so that the area under the curve sums to a probability of 1, the centerpoint of the vertical drop is what matters. But that's rather hard to read off this chart, as the blue line I mentioned has a center point point well above the red one, so while it looks like a shift of 80 cycles, it's more like 30. Is there any theoretical reason you can't just sum the histograms for runs of the same code and then divide by event count? Is there some sort of alignment/cache-coloring issue across boots? That's what I'd expect on an idle system, certainly. Anyway, I'm actually surprised your red graph is visibly better than your blue one. FWIW, I was

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatch changes

H. Peter Anvin — 2008-11-14T01:20:08

Last I heard it was still a dozen-ish cycles even on Nehalem. Even if a CPU came out *today* that had zero-cost locks we'd have to worry about it for at least another 5-10 years. The good news is that we're doing pretty good with it for now, but I don't believe in general we can avoid the fact that improving LOCK performance helps everything when you're dealing with large numbers of cores/threads. -hpa

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatch changes

H. Peter Anvin — 2008-11-14T01:18:21

No, they reflect individual runs. They start at 1 at the top left and drop to 0 at the far right in each case. What matters is the horizontal position of large vertical drops. All interrupts, but rather inherently the difference between interrupt handlers is going to be bigger than the differences between implementations of the same handler. I *believe* all the interrupts you're seeing in that graph are probably timer interrupts. The other major interrupt source that was active on the system was USB. -hpa

work

Matias Zabaljauregui — 2008-11-11T14:13:03

Hello! I'd like to ask if I can go on with my lguest to-do list or if PAE is causing any problems (if this is the case, please let me know and I'll try to fix it). regards Matias

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatch changes

Ingo Molnar — 2008-11-11T09:54:39

* Alexander van Heukelum public.gmane.org> wrote: a high-pass filter should be applied in any case, to filter out the "nothing happened" baseline. Eliminating every delta below 500-1000 cycles would do the trick i think, all IRQ costs are at least 1000 cycles. then a low-pass filter should be applied to eliminate non-irq noise such as scheduling effects or expensive irqs (which are both uninteresting to such analysis). and then _that_ double-filtered dataset should be normalized: the number of events should be made the same. (just clip the larger dataset to the length of the smaller dataset) i was only looking at before/after duos, for the same basic type of workload. Idle versus hackbench is indeed apples to oranges. Ingo

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatch changes

H. Peter Anvin — 2008-11-11T05:00:53

Okay, after spending most of the day trying to get something that isn't completely like white noise (interesting problem, otherwise I'd have given up long ago) I did, eventually, come up with something that looks like it's significant. I did a set of multiple runs, and am looking for the "waterfall points" in the cumulative statistics. http://www.zytor.com/~hpa/baseline-hpa-3000-3600.pdf This particular set of data points was gathered on a 64-bit kernel, so I didn't try the segment technique. It looks to me that the collection of red lines is enough to the left of the black ones that one can assume there is a significant effect, probably by about a cache miss worth of time. -hpa

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatch changes

H. Peter Anvin — 2008-11-10T23:34:59

Okay, I've stared at a bunch of different transformations of this data and I'm starting to think that it's getting lost in the noise. The difference between your "idleticks" and "idleticks2" data sets, for example, is as big as the differences between any two data sets that I can see. Just for reference, see this graph where I have filtered out events outside the [30..1000] cycle range and renormalized. http://www.zytor.com/~hpa/hist.pdf I don't know how to even figure out what a realistic error range looks like, other than repeating each run something like 100+ times and do an "eye chart" kind of diagram. -hpa

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatch changes

H. Peter Anvin — 2008-11-10T22:21:16

I believe you need to remove the obvious null events at the low end (no interrupt happened) and renormalize to the same scale for the histograms to make sense. As it is, the difference in the number of events that actually matters dominate the graphs; for example, there are 142187 events >= 12 in hack10ticks, but 136533 in hack10ticks_hpa. -hpa

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatch changes

Alexander van Heukelum — 2008-11-10T21:44:18

On Mon, 10 Nov 2008 07:39:22 -0800, "H. Peter Anvin" public.gmane.org> said: I did the rdtsctest again for the irqstubs patch you sent. The data is at http://heukelum.fastmail.fm/irqstubs/ and the latency histogram is http://heukelum.fastmail.fm/irqstubs/latency_hpa.png Greetings, Alexander

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatch changes

Alexander van Heukelum — 2008-11-10T21:35:07

On Mon, 10 Nov 2008 14:07:09 +0100, "Ingo Molnar" public.gmane.org> said: I thought so ;). The total number of measured intervals (between two almost-adjacent rdtsc's) is exactly the same for all histograms (10^10). Almost all measurements are of the "nothing happened" type, i.e., around 11 clock cycles on this machine. The user time spent inside the rdtsctest program is almost independent of the load, but it measures time spent outside of the program... But what should be attributed to what effect is unclear to me at the moment. Basically the difference between the "idle" and "hack10" versions should indicate the effect of extra interrupts (timer) and additional exceptions and cache effects due to context switching. Thanks, Alexander

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatch changes

H. Peter Anvin — 2008-11-10T15:39:22

For what it's worth, I tested this out, and I'm pretty sure you need to run a uniprocessor configuration (or system) for it to make sense -- otherwise you end up missing too many of the interrupts. I first tested this on an 8-processor system and, well, came up with nothing. I'm going to try this later on a uniprocessor, unless Alexander beats me to it. -hpa

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatch changes

Ingo Molnar — 2008-11-10T13:07:09

* Alexander van Heukelum public.gmane.org> wrote: yeah, sorry! You describe and did exactly the kind of histogram that i wanted to see done ;-) I'm not sure i can read out the same thing from the result though. Firstly, it seems the 'after' histograms are better, because there the histogram shifted towards shorter delays. (i.e. lower effective irq entry overhead) OTOH, unless i'm misreading them, it's a bit hard to compare them visually: the integral of the histograms does not seem to be constant, they dont seem to be normalized. It should be made constant for them to be comparable. (i.e. the total number of irq hits profiled should be equal - or should be normalized with the sum after the fact) Ingo

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatch changes

Alexander van Heukelum — 2008-11-10T12:44:21

On Mon, 10 Nov 2008 09:58:46 +0100, "Ingo Molnar" public.gmane.org> said: Hi Ingo, I guess you just stopped reading here? I should have presented the second benchmark as the first I guess. I really just used hackbench as a workload. I gathered it would give a good amount of exceptions like page faults and maybe others. It would be nice to have a simple debug switch in the kernel to make it generate a lot of interrupts, though ;). See the rest of the mail you replied to and its attachment. I've put the programs I used and the histogram in http://heukelum.fastmail.fm/irqstubs/ I think rdtsctest.c is pretty much what you describe. Greetings, Alexander

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatch changes

Ingo Molnar — 2008-11-10T08:58:46

* Alexander van Heukelum public.gmane.org> wrote: hackbench is _way_ too noisy to measure such cycle-level differences as irq entry changes cause. It also does not really stress interrupts - it only stresses networking, the VFS and the scheduler. a better test might have been to generate a ton of interrupts, but even then it's _very_ hard to measure it properly. The best method is what i've suggested to you early on: run a loop in user-space and observe irq costs via RDTSC, as they happen. Then build a histogram and compare the before/after histogram. Compare best-case results as well (the first slot of the histogram), as those are statistically much more significant than a noisy average. Measuring such things in a meaningful way is really tricky business. Using hackbench to measure IRQ entry micro-costs is like trying to take a photo of a delicate flower at night, by using an atomic bomb as the flash-light: you certainly get some sort of effect to report, but

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatch changes

H. Peter Anvin — 2008-11-10T01:29:45

Hi Alexander, First of all, great job on the timing analysis. I believe this confirms the concerns that I had about this technique. Here is a prototype patch of the compressed IRQ stubs -- this patch compresses them down to 7 stubs per 32-byte cache line (or part of cache line) at the expense of a back-to-back jmp which has the potential of being ugly on some pipelines (we can only get 4 stubs into 32 bytes without that). Would you be willing to run your timing test on this patch? This isn't submission-quality since it commingles multiple changes, and it needs some cleanup, but it should be useful for testing. As a side benefit it eliminates some gratuitous differences between the 32- and 64-bit code. -hpa _______________________________________________ Lguest mailing list Lguest-mnsaURCQ41sdnm+yROfE0A< at >public.gmane.org https://ozlabs.org/mailman/listinfo/lguest

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatch changes

Alexander van Heukelum — 2008-11-09T15:16:45

On Tue, 4 Nov 2008 21:44:00 +0100, "Ingo Molnar" public.gmane.org> said: Hi all, I have spent some time trying to find out how expensive the segment-switching patch was. I have only one computer available at the time: a "Sempron 2400+", 32-bit-only machine. Measured were timings of "hackbench 10" in a loop. The average was taken of more than 100 runs. Timings were done for two seperate boots of the system. The second test measures the time difference in ticks between two (almost) consecutive rdtsc instructions and makes a histogram from that. The (cumulative) histograms are plotted for an otherwise unused system and for a system running "hackbench 10" in a loop. The histogram is plotted on a log-log scale. On the horizontal acces is the length of the latency, and on the vertical axis the number of occurrences of a measured time difference _more than_ the value on the horizontal axis. before: hackbench 10: 1.698(2)s 1.712(1)s hackbench 10 + rdtsctest: hackbench 1.736(2)s 1.712(3)s aft

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatch changes

Ingo Molnar — 2008-11-06T09:30:20

* H. Peter Anvin public.gmane.org> wrote: heh, yes of course :-) Ingo

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatch changes

H. Peter Anvin — 2008-11-06T09:25:58

8 bytes, rather.

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatch changes

Ingo Molnar — 2008-11-06T09:19:29

* Alexander van Heukelum public.gmane.org> wrote: the cost is 6 cycles instead of 1 cycles. In a codepath that takes thousands of cycles and is often cache-limited. we really want to keep the stack frame consistent between all the context types. We can do things like return-to-userspace-from-irq or schedule-from-irq-initiated-event, etc. - so crossing between these context frames has to be standard and straightforward. Ingo