gmane.linux.kernel.kernelnewbies

Re: Locks and the FSB

Elad Lahav — 2008-12-01T22:00:47

I'm wondering if the high event counts following spin_unlock_irqrestore() are due to an OProfile quirk. OProfile depends on interrupts, which are disabled inside the critical section protected by the spin lock. It may end up accounting for all events that occurred inside the critical section immediately after the processor exits this section. --Elad -- To unsubscribe from this list: send an email with "unsubscribe kernelnewbies" to ecartis< at >nl.linux.org Please read the FAQ at http://kernelnewbies.org/FAQ

Re: Locks and the FSB

Elad Lahav — 2008-12-01T15:50:10

Sorry for the late response. Not in this case. I have 4 web server processes running, each pinned to a different CPU. The interrupts are also pinned, such that interrupts for a specific NIC occur only on the CPU handling the data relevant to that web server process. Yes, this may be a case of false sharing, I'll check that. Here are the top functions with respect to global_power_events in the 3 processor case: samples % samples % samples % samples % app name symbol name 141924 9.3228 182254 11.5263 178498 11.2869 621626 38.7608 vmlinux cpu_idle 138963 9.1283 175842 11.1208 172319 10.8962 588353 36.6861 vmlinux poll_idle 79616 5.2299 103271 6.5312 101016 6.3875 338251 21.0913 vmlinux __rcu_process_callbacks 43093 2.8307 42444 2.6843 42888 2.7119 0 0 e1000.ko e1000_xmit_frame 42270 2.7767 42223 2.6703 43035 2.7212 0 0 vmlinux __inet_check_established 39980 2.6262 40492 2.5608 39348 2.4881 2 1.2e-04 e1000.ko e1000_clean_rx_irq 36378 2.3896 35166 2.2240 36595 2.3140 0 0 vmlinux tcp_ack and the same for the 4 processor case: samples % samples % samples % samples % image name app name symbol name 64036 4.1148 65463 4.0258 63911 3.9295 63304 3.8920 e1000.ko e1000.ko e1000_clean_rx_irq 62808 4.0359 64516 3.9675 64566 3.9698 63963 3.9325 vmlinux vmlinux __inet_check_established 61813 3.9720 62271 3.8295 61574 3.7858 61609 3.7878 e1000.ko e1000.ko e1000_xmit_frame 47354 3.0429 49379 3.0366 49729 3.0575 49858 3.0653 vmlinux vmlinux tcp_ack 44598 2.8658 43340 2.6653 43713 2.6876 43097 2.6496 vmlinux vmlinux sys_epoll_ctl 32774 2.1060 35096 2.1583 32988 2.0282 35085 2.1571 vmlinux vmlinux do_sendfile 30658 1.9700 31268 1.9229 31685 1.9481 31291 1.9238 vmlinux vmlinux tcp_transmit_skb As you can see, there is considerable per-processor idle time in the 3 processor case, that almost disappears when moving to 4 processors. If we take e1000_xmit_frame as an example, its average time share rises from ~2.7% per processor, to ~3.8%. The following is the most interesting part of this function's annotated assembly code: 26 0.0603 31 0.0730 22 0.0513 0 0 : 53ea: call ff0 117 0.2715 130 0.3063 116 0.2705 0 0 : 53ef: mov 0x1c(%esp),%eax 311 0.7217 260 0.6126 191 0.4453 0 0 : 53f3: mov 0x34(%esp),%edx 0 0 4 0.0094 7 0.0163 0 0 : 53f7: call 53f8 19227 44.6175 19109 45.0217 19229 44.8354 0 0 : 53fc: xor %eax,%eax 175 0.4061 169 0.3982 171 0.3987 0 0 : 53fe: add $0x7c,%esp 61 0.1416 44 0.1037 48 0.1119 0 0 : 5401: pop %ebx The instruction following a call to spin_unlock_irqrestore() accounts for 19227 sampled events out of a total of 43093 for the entire function (I'm only citing the numbers for the first processor, but the rest are roughly the same). When using all 4 processor, we end up with the following: 33 0.0534 29 0.0466 32 0.0520 39 0.0633 : 53ea: call ff0 182 0.2944 187 0.3003 207 0.3362 155 0.2516 : 53ef: mov 0x1c(%esp),%eax 440 0.7118 521 0.8367 338 0.5489 416 0.6752 : 53f3: mov 0x34(%esp),%edx 5 0.0081 6 0.0096 3 0.0049 1 0.0016 : 53f7: call 53f8 35568 57.5413 35241 56.5930 34853 56.6034 35700 57.9461 : 53fc: xor %eax,%eax 187 0.3025 209 0.3356 207 0.3362 206 0.3344 : 53fe: add $0x7c,%esp 63 0.1019 69 0.1108 53 0.0861 68 0.1104 : 5401: pop %ebx A dramatic increase to 35568 events out of a total of 61813. Now, the number of per-processor L2 cache misses for this function is exactly the same in the 3 and 4 processor scenarios. On the other hand, the number of FSB read/write events grows from 2029 to 2978, for which more than half are accredited to the xor instruction that follows the call to spin_unlock_irqrestore(). --Elad -- To unsubscribe from this list: send an email with "unsubscribe kernelnewbies" to ecartis< at >nl.linux.org Please read the FAQ at http://kernelnewbies.org/FAQ

Re: GCC 4.4 hacks in the Linux kernel

Frédéric Weisbecker — 2008-12-01T14:02:02

2008/12/1 Peter Teoh gmail.com>: Thanks! Very interesting :-) -- To unsubscribe from this list: send an email with "unsubscribe kernelnewbies" to ecartis< at >nl.linux.org Please read the FAQ at http://kernelnewbies.org/FAQ

Re: question on priority imbalance in Linux SMP scheduler!

Henrik Austad — 2008-12-01T12:54:14

Hello Note, this article is rather outdated as it describes the O(1) scheduler, and not the new CFS (which was introduced in 2.6.23) One of the major differences, is that normal tasks, i.e. tasks with priority between 139 and 100, are handled by CFS (SCHED_NORMAL, SCHED_BATCH and SCHED_IDLE). pri 99 to 1 (or 0, if you count in the top-priority real-time kernel priority not accessible from user-space), is handled by SCHED_RR/SCHED_FIFO. Each CPU has its own runqueue (struct rq), which have a pointer to the real-time runqueue and to cfs_rq, the red-black tree containing all normal tasks. The run-queue for real-time tasks is very similar to the old O(1) scheduler, except that it has 100 slots instead of 140, and consist of only one queue, not 2 (active and expired) as O(1). does anyone know how the old O(1) handled real-time tasks wrt to the 2 queues and active/expired swapping? Well, for a starter, to balance tasks perfectly on n processors, is NP-hard. This is due to the bin-packing problem, as the scheduler must take all tasks, calculate the load and try to distribute them across all processors, setting the capacity of each CPU to total_load / CPUs Every so often, a scheduler tick will happen (even though CFS is tickless), and when this happens, the currently running task may be reinserted into the run-queue and a new task selected, if this is appropriate. And this is basically the crux of the load-balancing - the scheduler will not try to balance the load if it has tasks to run, only when it is idle! The load_balance *does not* push a task to another CPU, it is used to retrieve a task from another CPU. So, when CPU A is idle, it will see if CPU B has a task it can take in order to balance the load. Well, to confuse you a bit, tasks *can* be pushed, by migration threads (see active_load_balance in sched.c), but it's more realistic that scheduler_tick will case the load_balancing functionality on an idle CPU. From kernel/sched.c: /* * Trigger the SCHED_SOFTIRQ if it is time to do periodic load balancing. * * In case of CONFIG_NO_HZ, this is the place where we nominate a new * idle load balancing owner or decide to stop the periodic load balancing, * if the whole system is idle. */ static inline void trigger_load_balance(struct rq *rq, int cpu) { *snip* ... On top of this, the balancing routine will first look at the group and then the enclosing scheduling domain, recursively untill the global/topmost domain is reached. Normal rules apply - I'm a newbie myself, so others may want to correct me :-)

Re: timer interrupts during boot

Stephane Lambert — 2008-12-01T08:34:40

Syed Khader a écrit : Hi Thanks for your reply. I'm porting uClinux on the CORTUS APS3 processor. Concerning the init task (init process), it is created in rest_init() by calling the kernel_thread() function. kernel_thread() |---> system_call() |---> sys_clone() |---> do_fork() Then, the init() function is called and performs a new kernel_thread() and runs the kthread() function. However, kthread() invokes the scheduler (it's at this point that (bad: scheduling from the idle thread!) printk message is printed..). The scheduler switch to the process previously created (switch_to() ) and run ksoftirqd . But the cpu remains in a endless loop into ksoftirqd: while (!kthread_should_stop()) { 493 preempt_disable(); 494 if (!local_softirq_pending()) { 495 preempt_enable_no_resched(); 496 schedule(); 497 preempt_disable(); 498 } } Concerning the timer interrupts: Yes I have implemented the timer in my arch-specific directory. The gdb-trace of the boot shows that a timer interrupt occurs each times do_fork() returns. Does it correct? What do you mean by machine_desc? Do you know if the timer interrupts need to be disabled until cpu_idle? Thanks in advance. -- To unsubscribe from this list: send an email with "unsubscribe kernelnewbies" to ecartis< at >nl.linux.org Please read the FAQ at http://kernelnewbies.org/FAQ

Re: how to calculate time required for the execution of the entry point of the driver ?

Mulyadi Santosa — 2008-11-30T14:47:49

On Sun, Nov 30, 2008 at 2:51 PM, Michael Blizek michaelblizek.twilightparadox.com> wrote: yes, you're correct, I forgot about frequency scaling that certainly affects TSC. NB: This is why I like this kind of discussion, it makes everybody smarter... regards, Mulyadi. -- To unsubscribe from this list: send an email with "unsubscribe kernelnewbies" to ecartis< at >nl.linux.org Please read the FAQ at http://kernelnewbies.org/FAQ

Re: how to calculate time required for the execution of the entry point of the driver ?

Mulyadi Santosa — 2008-11-30T14:42:37

Re: how to calculate time required for the execution of the entrypoint of the driver ?

Michael Blizek — 2008-11-30T07:51:55

Hi! On 13:43 Sun 30 Nov , Mulyadi Santosa wrote: ... This sounds great. See http://en.wikipedia.org/wiki/Time_Stamp_Counter This seems to have some implications: - The timestamp counter is not synchronized on all CPUs/cores ==> set the CPU affinity before starting the Benchmark. - The time unit is CPU clock cycles or something else (processor dependant), not microseconds. Frequency scaling and maybe APM_CPU_IDLE are probably not such a great idea... -Michi

Re: how to calculate time required for the execution of the entry point of the driver ?

Mulyadi Santosa — 2008-11-30T06:43:23

Hi All On Sat, Nov 29, 2008 at 4:12 AM, Michael Blizek michaelblizek.twilightparadox.com> wrote: Michael got some points. And since I believe "timing" is very broad subject to cover, I'd like to share my thougths too. I've done few timing for personal benchmarking and here's my conclusion: 1. run your timing test in run level 1. This way, you won't be hogged down with daemon running in the background which could interfere. 2. I suggest to use TSC (TimeStamp Clock) if you use x86 (32 and 64 bit). I don't say gettimeofday() won't meet your need, but I think reading TSC register is the fastest code path you can get to measure timing. 3. beware of preemption. Your timing could be preempted anywhere. I'd suggest to disable full kernel level preemption and enable voluntary preemption instead. Or better, disable kernel level preemption at all. This way, kernel code path goes uninterrupted by another non interrupt handler code path. 4. HZ could mean bring hassles too. Try to use lowest HZ as possible. This fit nicely if you use TSC, since gettimeofday relies on jiffies counting and that indirectly relies on timer tick. All in all, like other statistics work, use as many samples as you can and show the reader the standart deviation etc etc. regards, Mulyadi. -- To unsubscribe from this list: send an email with "unsubscribe kernelnewbies" to ecartis< at >nl.linux.org Please read the FAQ at http://kernelnewbies.org/FAQ

Re: how to calculate time required for the execution of the entrypoint of the driver ?

Michael Blizek — 2008-11-28T21:12:48

Hi! On 14:40 Fri 28 Nov , yogeshwar sonawane wrote: I would call the syscalls multiple times. This way, there is no requirement for a ultra high resolution clock. gettimeofday() will probably be enough. But there are some other pitfalls: - The time the ioctl, ... takes depends on whether the required things are already cached in the CPU cache or not. If you want to measure how long it takes with a cold cache, you can try to trash it between runs. But then you cannot easily call gettimeofday() before and after a loop of the ioctl(). You can call gettimeofday(), ioctl(), gettimeofday(), trash the cpu cache, then do it again. But - gettimeofday() itself takes some time. This means even if the resolution is high enough or if you average of a lot samples, this time always adds to the result. If you trash the CPU cache, you may also alter the time gettimeofday() takes. You may want to offset this by calling a gettimeofday() and ignoging the result to get the data into the cache. You can also try to offset the gettimeofday() times by measuring the time gettimeofday() takes and substracting this of the result. - Do not run any other process to make sure that the benchmark process does get the cpu. - Something else may come up. -Michi

Re: kmalloc() 's virtual address

JiSheng Zhang — 2008-11-28T16:44:46

I test it many times, the result( __pa(...) != look through page table ) appeared only once. replace k_pgd = pgd_offset_k(ka); with k_pgd = pgd_offset(current->mm, ka); the result should be OK. -- To unsubscribe from this list: send an email with "unsubscribe kernelnewbies" to ecartis< at >nl.linux.org Please read the FAQ at http://kernelnewbies.org/FAQ

RE: Detecting infinite loops

Hayim Shaul — 2008-11-28T15:42:25

You can't. This is called 'The halting problem' and Alan Turing proved it is impossible in 1936. http://en.wikipedia.org/wiki/Halting_problem ________________________________ From: kernelnewbies-bounce< at >nl.linux.org on behalf of Pranav Peshwe Sent: Fri 28/11/2008 5:08 PM To: Kernel Subject: Re: Detecting infinite loops Hi, On Fri, Nov 28, 2008 at 3:49 PM, yogeshwar sonawane gmail.com> wrote: Hi, Thanks for correction. Main idea:- I am talking about a program running in normal mode, not in gdb mode(not using GDB). Suppose the expected output should come in 10 min. Now, i run that program. It is running for quite some time now, like say 1hr. Still no output. That code has some infinite loop like explained in previous mail. But i dont know details. Now, by some means (like attaching GDB & doing something) can i make sure that the program is just looping for condition-check ? IMO, there is no general way to do that. More so, since you say 'i dont know details' about the program. If you attach a debugger and manually step through the code, then, with a lot of patience, you could check whether the same set of statements are being executed again and again i.e whether it is in a loop. AFAIK, there is no tool which can detect an infinite loop in a running program and that too, without knowing specific details about it. Putting some log statements in loop, will cause it, to print no. of times. Even after some looping, the expected condition occurs, still messages are going to be printed. Not necessarily, you could have something akin to - flag = 1; while(condition is not true) { if(flag == 1) log(condition not yet true); flag = 0; } log(condition now true); You can include timestamp in the logs to indicate how much time was spent looping. HTH. Corrections welcome.. Best regards, Pranav

Re: Detecting infinite loops

Pranav Peshwe — 2008-11-28T15:08:25

Hi, On Fri, Nov 28, 2008 at 3:49 PM, yogeshwar sonawane gmail.com>wrote: IMO, there is no general way to do that. More so, since you say 'i dont know details' about the program. If you attach a debugger and manually step through the code, then, with a lot of patience, you could check whether the same set of statements are being executed again and again i.e whether it is in a loop. AFAIK, there is no tool which can detect an infinite loop in a running program and that too, without knowing specific details about it. Not necessarily, you could have something akin to - flag = 1; while(condition is not true) { if(flag == 1) log(condition not yet true); flag = 0; } log(condition now true); You can include timestamp in the logs to indicate how much time was spent looping. HTH. Corrections welcome.. Best regards, Pranav

question on priority imbalance in Linux SMP scheduler!

Thirupathiah Annapureddy — 2008-11-28T15:08:31

Hi All, I am new to linux kernel. I was refering to an interesting article titled "Inside the Linux scheduler", published at http://www.ibm.com/developerworks/linux/library/l-scheduler/. I have a questions related to "priority imbalance". The article mentions that each CPU has a runqueue made up of 140 priority lists i.e. per CPU ready queue. Hence it is possible that a runnable high priority task waits for a CPU in ready queue while a lower priority task is running on a different CPU. This can lead to priority imbalance/priority inversion. How does linux SMP scheduler handle this priority imbalance? Thanks in advance for you answer. Regards

Re: Detecting infinite loops

megistos triskataratos — 2008-11-28T10:13:49

You might be able to do it with a spinlock.. you can hold a lock on your prog and repeatedly check whatever you want for example u can use jiffies or something, so that if it loops for like 1-2 secs you can release lock or whatever links i jumped into: http://en.wikipedia.org/wiki/Spinlock http://linux.ubuntu.com.tw/2008/04/linux-kernel-spin-locks-vs-semaphores.html http://www.linuxgrill.com/anonymous/fire/netfilter/kernel-hacking-HOWTO-5.html

Re: Detecting infinite loops

yogeshwar sonawane — 2008-11-28T10:19:43

Hi, Thanks for correction. Main idea:- I am talking about a program running in normal mode, not in gdb mode(not using GDB). Suppose the expected output should come in 10 min. Now, i run that program. It is running for quite some time now, like say 1hr. Still no output. That code has some infinite loop like explained in previous mail. But i dont know details. Now, by some means (like attaching GDB & doing something) can i make sure that the program is just looping for condition-check ? To give a practical example, suppose i am checking some counter register of a PCI card. Withing 10-20 times checking, the register is updated with new value. This is some kind of event for application. Counter value is stored. Then this counter value is checked for further increments. Suppose in such situations, every 10-20 times checking results in one counter increment. But what if, no increment happens ? how to detect this ? My focus is not on the coding choices & other stuff. My focus is on how to detect such situations using any tools available ? Putting some log statements in loop, will cause it, to print no. of times. Even after some looping, the expected condition occurs, still messages are going to be printed. Regards, Yogeshwar On Fri, Nov 28, 2008 at 3:12 PM, Pranav Peshwe gmail.com> wrote: -- To unsubscribe from this list: send an email with "unsubscribe kernelnewbies" to ecartis< at >nl.linux.org Please read the FAQ at http://kernelnewbies.org/FAQ

Re: Detecting infinite loops

Pranav Peshwe — 2008-11-28T09:42:58

If i'm not wrong, you mean 'while (expected condition has not occurred) ;' (note the semicolon) Otherwise, it does not make sense logically. IMHO, busy waiting like this would not be recommended in any app, and not at all inside kernel code. In case, there is absolutely no choice but to wait on a while(), the simplest way to let the user know about the internal state of your app/library is by having (a) log statement(s) inside the loop. HTH. CMIIW... Best regards, Pranav http://pranavsbrain.peshwe.com

Re: Detecting infinite loops

sandeep lahane — 2008-11-28T09:15:49

How about gdb attach process_id ? Or I am missing something? Regards, Sandeep. On Fri, Nov 28, 2008 at 2:25 PM, yogeshwar sonawane gmail.com> wrote: -- To unsubscribe from this list: send an email with "unsubscribe kernelnewbies" to ecartis< at >nl.linux.org Please read the FAQ at http://kernelnewbies.org/FAQ

Re: Info on driver code profiling tools

Belisko Marek — 2008-11-28T09:14:13

Hi, On Fri, Nov 28, 2008 at 10:03 AM, yogeshwar sonawane gmail.com> wrote: some google results: http://www.win.tue.nl/~aeb/linux/lk/lk-2.html (issue 2.10) hope this helps, thanks Marek

how to calculate time required for the execution of the entry point of the driver ?

yogeshwar sonawane — 2008-11-28T09:10:32

Hi all, 1) I have a driver code providing ioctl,read, write, mmap etc. entry points. Now, i want to find the execution time of each entry point for comparison. How this can be done ? for ex. take time_stamp1 call entry point take time_stamp2 The above code from a user application can measure the execution time ? Timing functions in user space are close to the real time ? 2) Which timing functions should be used to get the, close to correct time, in user & kernel space ? Mainly the purpose is to calculate timing interval. Kindly help me. Any link/reference will be helpful. With regards, Yogeshwar -- To unsubscribe from this list: send an email with "unsubscribe kernelnewbies" to ecartis< at >nl.linux.org Please read the FAQ at http://kernelnewbies.org/FAQ

Info on driver code profiling tools

yogeshwar sonawane — 2008-11-28T09:03:33

Hi all, I need some information on profilers which can profile driver codes for linux. Mainly, it should be able to pin-point the functions which are taking more time, plus some other info which can be helpful in making code efficient. Kindly help me. Any reference/link will be useful. With regards, Yogeshwar -- To unsubscribe from this list: send an email with "unsubscribe kernelnewbies" to ecartis< at >nl.linux.org Please read the FAQ at http://kernelnewbies.org/FAQ