gmane.linux.kernel.virtualization.lguest

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatchchanges

Ingo Molnar — 2008-12-01T10:49:04

* Avi Kivity public.gmane.org> wrote: that is what happened three weeks ago already on Nov 11, we applied Peter's patches to tip/x86/irq: 939b787: x86: 64 bits: shrink and align IRQ stubs b7c6244: x86: 32 bits: shrink and align IRQ stubs it's all in the x86 tree and in linux-next as well. Alexander and Peter are working on this together, not against each other. Alexander was still running some numbers to make sure we made the right decision. Ingo

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatch changes

Avi Kivity — 2008-12-01T10:41:53

What I mean is that hpa's patch makes the kernel better, so it should be applied. I'm not sure what else Alexander is working on, but I do hope the improvements will be more concrete. (Alexander, I apologise for being so discouraging; but I really feel these improvements are marginal)

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatch changes

Avi Kivity — 2008-12-01T09:24:33

Four bytes was the smallest sane option. Three bytes involved instruction opcodes overlap. Once it's done there's no reason not to commit it. But the effort expended to do it is gone, without any measurable return.

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatchchanges

Ingo Molnar — 2008-12-01T08:00:23

* Rusty Russell public.gmane.org> wrote: Yeah, the efforts from Alexander, Peter and Cyrill are fantastic. The most positive effects of it isnt just the optimizations and code compression. What is important is the fact that this piece of code (entry_*.S) has not gotten systematic attention for a decade, and has become a rather crufty patchwork of hit-and-run changes. It has become very hard to review and it has become a reoccuring source of nasty bugs. It is now being reworked and re-measured. It does not matter how small and hard to measure the gain is in one specific case - we are mainly concerned about not creating measurable _harm_, and we are after the cleanups and the increase in maintainability. Once the code is cleaner, improvements are much easier to do and bugs are much easier to fix. Ingo

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatch changes

Rusty Russell — 2008-12-01T04:32:35

Sure, but smallest cache wins. Which is why I thought hpa chose the 3 byte option. Respectfully disagree. I wouldn't do it, but it warms my heart that others are. It's are not subtractive from other optimization efforts. Cheers, Rusty.

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatch changes

Avi Kivity — 2008-11-29T18:21:22

This is noise. 0.05% cpu on a 1GHz machine servicing 1000 interrupt/sec boils down to 500 cycles/interrupt. These changes shouldn't amount to so much (and I doubt you have 1000 interrupts/sec with a single disk).. I'm sorry, but the whole effort is misguided, in my opinion. If you want to optimize, try reducing the number of interrupts that occur rather than saving a few cycles in the interrupt path.

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatch changes

Avi Kivity — 2008-11-29T18:22:41

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatch changes

Alexander van Heukelum — 2008-11-29T15:45:16

I now did the benchmarks for the same -rc6 with hpa's 4-byte stubs too. Same machine. It's significantly better than the other two options in terms of speed. It takes about 7% less cpu to handle the interrupts. (0.64% cpu instead of 0.69%.) I have to run now, I'll let interpreting the histogram to someone else ;). Greetings, Alexander _______________________________________________ Lguest mailing list Lguest-mnsaURCQ41sdnm+yROfE0A< at >public.gmane.org https://ozlabs.org/mailman/listinfo/lguest

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatch changes

Alexander van Heukelum — 2008-11-28T20:48:10

Hi all, We started the discussion with doing away with the whole jump array entirely, by changing the value of the CS index in the IDT. This needs the GDT to be extended with 256 entries, but an entire page (space for 512 entries) was already reserved anyhow! I think there is still some problem with the patch I sent due to some code depending on certain values of the CS index, but the system I've benchmarked on seemed to behave. I did a set of benchmarks on an 8-way Xeon in 64-bit mode. The system was loaded with an instance of bonnie++ pinned to processor 0, and all 8 processors were running a program doing (almost) adjacent rdtsc's. Bonnie++ causes interrupts and the latencies due to these show up as larger time intervals. Complete runs of bonnie++ in fast mode were sampled this way for a current -rc6 kernel and an -rc6 kernel plus my patch. The total sampling time was 30 minutes for each run. Per kernel I did one run as a warm-up and another two runs to measure the latencies. The results for measured latencies between 5 and 1000 microseconds are shown in the attached graph. Above 1000 microseconds there is only one big contribution: at 40000 microseconds ;). The surface below the graph is a measure of time. Observations (for this test load!): Near 200, 250 and 350 microseconds, the peaks shift to longer latencies for the cs-changing code by about 10 microseconds, but the total time spent is pretty much constant. The highest latencies for the cs-changing code are near 600 and 650 microseconds. The highest latencies for the current code are near 800 and 850 microseconds. The total surface of the graphs between 5 and 1000 microseconds is within an error estimate of 1% equal for both cases, and is about 0.69% of the total time. Most time is spent measuring 'latencies' of less than 5 micro- seconds, since bonnie++ is taking only about 5% cpu time on a single cpu most of the time, and only up to 50% on a single cpu during a short time in the file creation benchmark. Greetings, Alexander _______________________________________________ Lguest mailing list Lguest-mnsaURCQ41sdnm+yROfE0A< at >public.gmane.org https://ozlabs.org/mailman/listinfo/lguest

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatch changes

Andi Kleen — 2008-11-27T10:56:35

That would be still one byte, otherwise you wouldn't get a unique index. -Andi

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatch changes

H. Peter Anvin — 2008-11-27T00:03:59

Yes, I would consider that a theoretical exercise only :) On the entirely silly level... CC xx -hpa

Re: [PATCH 2/2] kvm-s390: implement config_changed forvirtio on s390

Rusty Russell — 2008-11-18T14:34:40

Thanks, both applied. Rusty.

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatch changes

H. Peter Anvin — 2008-11-14T03:22:24

The reason for the multiple curves is to show the range of uncertainty. There are three sets of graphs in there: black (current mainline, 16-byte stubs), red (4-byte stubs with a double jump), and blue (8-byte stubs.) The fact that the system is idle is pretty much a case which should favor "blue" over "red"; realistically I think the graphs show that they are identical within the limits of measurement, and both are significantly better than "black". Since this is pretty much the optimal case for "blue" and it doesn't show any performance win over "red", I implemented the "red" option and pushed it into tip:x86/irq. Same here, which is why I wanted to check them both out. -hpa

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatch changes

Matt Mackall — 2008-11-14T02:29:36

Still confused. If, say, the top blue line on the left represents the same number of interrupts as the bottom red one, then at some point it must cross under the red one as it goes to the right, which it does not appear to do. Thus, it does not appear the scale on the left is actually in units of constant probability, no? Though I'll agree that even if they're not scaled so that the area under the curve sums to a probability of 1, the centerpoint of the vertical drop is what matters. But that's rather hard to read off this chart, as the blue line I mentioned has a center point point well above the red one, so while it looks like a shift of 80 cycles, it's more like 30. Is there any theoretical reason you can't just sum the histograms for runs of the same code and then divide by event count? Is there some sort of alignment/cache-coloring issue across boots? That's what I'd expect on an idle system, certainly. Anyway, I'm actually surprised your red graph is visibly better than your blue one. FWIW, I was leaning towards your simpler variant (and away from the magical segment register proposal). I'd be happy to see either of your versions submitted.

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatch changes

H. Peter Anvin — 2008-11-14T01:20:08

Last I heard it was still a dozen-ish cycles even on Nehalem. Even if a CPU came out *today* that had zero-cost locks we'd have to worry about it for at least another 5-10 years. The good news is that we're doing pretty good with it for now, but I don't believe in general we can avoid the fact that improving LOCK performance helps everything when you're dealing with large numbers of cores/threads. -hpa

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatch changes

H. Peter Anvin — 2008-11-14T01:18:21

No, they reflect individual runs. They start at 1 at the top left and drop to 0 at the far right in each case. What matters is the horizontal position of large vertical drops. All interrupts, but rather inherently the difference between interrupt handlers is going to be bigger than the differences between implementations of the same handler. I *believe* all the interrupts you're seeing in that graph are probably timer interrupts. The other major interrupt source that was active on the system was USB. -hpa

work

Matias Zabaljauregui — 2008-11-11T14:13:03

Hello! I'd like to ask if I can go on with my lguest to-do list or if PAE is causing any problems (if this is the case, please let me know and I'll try to fix it). regards Matias

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatch changes

Ingo Molnar — 2008-11-11T09:54:39

* Alexander van Heukelum public.gmane.org> wrote: a high-pass filter should be applied in any case, to filter out the "nothing happened" baseline. Eliminating every delta below 500-1000 cycles would do the trick i think, all IRQ costs are at least 1000 cycles. then a low-pass filter should be applied to eliminate non-irq noise such as scheduling effects or expensive irqs (which are both uninteresting to such analysis). and then _that_ double-filtered dataset should be normalized: the number of events should be made the same. (just clip the larger dataset to the length of the smaller dataset) i was only looking at before/after duos, for the same basic type of workload. Idle versus hackbench is indeed apples to oranges. Ingo

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatch changes

H. Peter Anvin — 2008-11-11T05:00:53

Okay, after spending most of the day trying to get something that isn't completely like white noise (interesting problem, otherwise I'd have given up long ago) I did, eventually, come up with something that looks like it's significant. I did a set of multiple runs, and am looking for the "waterfall points" in the cumulative statistics. http://www.zytor.com/~hpa/baseline-hpa-3000-3600.pdf This particular set of data points was gathered on a 64-bit kernel, so I didn't try the segment technique. It looks to me that the collection of red lines is enough to the left of the black ones that one can assume there is a significant effect, probably by about a cache miss worth of time. -hpa

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatch changes

H. Peter Anvin — 2008-11-10T23:34:59

Okay, I've stared at a bunch of different transformations of this data and I'm starting to think that it's getting lost in the noise. The difference between your "idleticks" and "idleticks2" data sets, for example, is as big as the differences between any two data sets that I can see. Just for reference, see this graph where I have filtered out events outside the [30..1000] cycle range and renormalized. http://www.zytor.com/~hpa/hist.pdf I don't know how to even figure out what a realistic error range looks like, other than repeating each run something like 100+ times and do an "eye chart" kind of diagram. -hpa

Re: [PATCH RFC/RFB] x86_64,i386: interrupt dispatch changes

H. Peter Anvin — 2008-11-10T22:21:16

I believe you need to remove the obvious null events at the low end (no interrupt happened) and renormalize to the same scale for the histograms to make sense. As it is, the difference in the number of events that actually matters dominate the graphs; for example, there are 142187 events >= 12 in hack10ticks, but 136533 in hack10ticks_hpa. -hpa