gmane.linux.rt.user

Re: [PATCH] mm: fix up a spurious page fault whenever it happens

Steven Rostedt — 2013-05-23T17:36:13

And as it's a boot time change only, it's not quite in the category of
jump_labels and function tracing.

Kernel 3.6.11.3-rt35 : umount blocks for more than 200seconds

Koehrer Mathias (ETAS/ESS2 — 2013-05-22T12:00:48

Hi everybody,

I am still searching for the reason why the umount of a hdd partition takes very often more than 200seconds.

I regularly got a kernel message "task umount blocked for more than 120 seconds":
The addresses within the message are always the very same. This seems to be the indication that the very same issue occurs again and again.
After nearly 5 minutes the umount command  finally finishes (according to the time output).

--------------- BEGIN MESSAGE ------------------
INFO: task umount:4923 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
umount          D 00000000     0  4645   4465 0x00000000
 e2cc9e3c 00000086 f3ca8160 00000000 00000000 00000000 f5cdd980 c095e980
 c095e980 00000000 e2cc9df4 f5cdd980 e6319950 e2cc9e50 00000000 e2cc9e94
 c01e50db 00000000 0000000e f5d99980 00000000 00000000 00000001 00000000
Call Trace:
 [<c01e50db>] ? write_cache_pages+0xdb/0x3b0
 [<c061c7dd>] ? _raw_spin_unlock_irqrestore+0x1d/0x40
 [<c01e4750>] ?

Re: [PATCH V2 Resend 4/4] timer: Migrate running timer

Viresh Kumar — 2013-05-22T09:23:07

Sure, once the initial draft is approved we can get it optimized to the
best of our capabilities. :)
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo< at >vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Tracers+cyclictest causing kernel oops

Tom Cook — 2013-05-21T15:20:20

On Tue, May 21, 2013 at 12:16 PM, Sebastian Andrzej Siewior
<bigeasy< at >linutronix.de> wrote:

Sorry.

[snip]

No.  I've also tried compiling without kgdb and enabling tracers still
causes a crash (see below).

Again, I'm far from expert here, but as near as I can tell, a fast
interrupt exception handler is causing a data abort exception.  Do the
tracers use fast interrupts to wake up?  Is there some tracer-related
memory that's getting swapped out?

pi< at >raspberrypi:~/rt-tests$ sudo ./cyclictest -p95 -m -f -b 2000
# /dev/cpu_dma_latency set to 0us
[  199.186167] Bad mode in data abort handler detected
[  199.186194] Internal error: Oops - bad mode: 0 [#1] PREEMPT ARM
[  199.186246] Modules linked in: snd_bcm2835 snd_pcm snd_seq
snd_timer snd_seq_device snd snd_page_alloc
[  199.186260] CPU: 0    Not tainted  (3.6.11-rt31+ #1)
[  199.186297] PC is at ring_buffer_lock_reserve+0x7c/0x144
[  199.186326] LR is at vfs_write+0x140/0x188
[  199.186341] pc : [<c009486c>]    lr : [<c00efda4>]    psr: 600001d1
[  199.18634

Re: Tracers+cyclictest causing kernel oops

Sebastian Andrzej Siewior — 2013-05-21T11:16:55

please CC the list.

On 05/11/2013 06:43 PM, Tom Cook wrote:

Not known until you brought that up. I just booted x86 with kgdb=y and
did "echo function > current_tracer" with no side effects. I don't have
ARM at hand right now. Do you do anything kgdb related besides enabling
it in the kernel?


Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo< at >vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Kernel 3.6.x RT_PREEMPT: Issues with sync / umount of hdd partitions

Koehrer Mathias (ETAS/ESS2 — 2013-05-21T09:12:07

Hi all,

Since changing to the 3.6.x series of RT_PREEMPT kernel (it was 2.6.33.x before) I have some strange issues with mounting/umounting/syncing of hard disk partitions.

What I do is, I format a partition of the hard disk, I mount this partition, extract an tgz image (about 750 MB) into that freshly formatted partition and umount it again:
      mkfs.ext3 ....
      mount ...
      tar xf ...
      umount  ...

Doing the umount causes occasionally kernel panics (unable to handle kernel paging request...  in ksoftirqd/0).
When I  add a "sync" before the umount, I never got the kernel panic.
However, I can observe that the time needed for the "sync" varies dramatically.
Very often it takes around 6-8 seconds, however sometimes it takes up to 53 seconds!
Apart from the test script that performs this actions, the PC is idle.

Kernel: 3.6.11.3-rt35, x86 (32 bit mode no PAE), 4 GB RAM, CPU core i7.
Kernel boot parameters:    root=/dev/sda6 vga=0x31a isolcpus=1-31 

I have attached the kernel config and the re

3.8.10-rt6 : Observing high latency as preempt_schedule_irq:__schedule is not getting called

Jain Priyanka-B32167 — 2013-05-21T07:01:24

Setup: Single core PowerPC, 32-bit platform , Linux3.8.10-rt6

'preempt_schedule_irq()'  is a function which generally gets called while exiting from IRQ context to call __schedule so that preemption opportunities do not get missed.
I have added print just above call to __schedule()

(START)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 3891f97..1f423c5 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
< at >< at > -3350,6 +3350,7 < at >< at > asmlinkage void __sched preempt_schedule_irq(void)
        do {
                add_preempt_count(PREEMPT_ACTIVE);
                local_irq_enable();
+               printk("%s\n",__FUNCTION__);
                __schedule();
                local_irq_disable();
                sub_preempt_count(PREEMPT_ACTIVE);
(END)

As none of instance of this print is coming in kernel dump, it seems that 'preempt_schedule_irq:__schedule' is not getting called.
So preemption opportunities while exiting from IRQ context(exception handling) are getting missed leading to increase

RE: 3.8.10-rt6 : Observing high latency as timer_interrupt is taking longer to exit

Jain Priyanka-B32167 — 2013-05-21T05:42:37

Furthur debug analysis

I am running cylictest with following settings: 
./cyclictest -a -n -p 99 -c 1 -d 0 -O latency-format -b 600 --tracer=function_graph


It seems that even if 'N' (need rescheduled bit) is set in current task and preemption disabled count is zero, context switching to highest priority task which is cyclictest thread in this case is not happening immediately. It is getting delayed adding to latency. Trace capture attached.

Please help me in identifying what can be the issue.

Regards
Priyanka

[PATCH - sort of] x86: Livelock in handle_pte_fault

Stanislav Meduna — 2013-05-17T08:42:51

Hi all,

I don't know whether this is linux-rt specific or applies to
the mainline too, so I'll repeat some things the linux-rt
readers already know.

Environment:

- Geode LX or Celeron M
- _not_ CONFIG_SMP
- linux 3.4 with realtime patches and full preempt configured
- an application consisting of several mostly RR-class threads
- the application runs with mlockall()
- there is no swap

Problem:

- after several hours to 1-2 weeks some of the threads start to loop
  in the following way

  0d...0 62811.755382: function:  do_page_fault
  0....0 62811.755386: function:     handle_mm_fault
  0....0 62811.755389: function:        handle_pte_fault
  0d...0 62811.755394: function:  do_page_fault
  0....0 62811.755396: function:     handle_mm_fault
  0....0 62811.755398: function:        handle_pte_fault
  0d...0 62811.755402: function:  do_page_fault
  0....0 62811.755404: function:     handle_mm_fault
  0....0 62811.755406: function:        handle_pte_fault

  and stay in the loop until the RT throttling gets a

RE: 3.8.10-rt6 : Observing high latency as timer_interrupt is taking longer to exit

Jain Priyanka-B32167 — 2013-05-15T10:19:01

Hi,

I have further debug the issue. 

The increase in latency issue is observed when kworker is in execution and timer_interrupt(hardirq) arrives.
Upon completion of timer_interrupt, previous context should have immediately got restored.
But a large time-gap (varying from 50us to 200us) is observed in restoration of kworker task.
e.g in below context, 195us gap is observed in timer_interrupt_exit and timer_start (kworker).

In complete trace, I have seen multiple entries of timer_interrupt.
But exit is taking large time gap only when it arrives in context of kworker.

Looking for some pointers as help to debug the issue.

Regards
Priyanka



--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo< at >vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V5 0/5] Queue work on power efficient wq

Viresh Kumar — 2013-05-15T05:48:59

Thanks. All looks good.
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo< at >vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[ANNOUNCE] 3.6.11.3-rt35

Steven Rostedt — 2013-05-14T15:22:13

Dear RT Folks,

I'm pleased to announce the 3.6.11.3-rt35 stable release.


This release is just an update to the new stable 3.6.11.3 version
and no RT specific changes have been made.


You can get this release via the git tree at:

  git://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-stable-rt.git

  branch: v3.6-rt
  Head SHA1: aa40d8a6b739c1356e2f1643dc01b9fad7d2def5


Or to build 3.6.11.3-rt35 directly, the following patches should be applied:

  http://www.kernel.org/pub/linux/kernel/v3.x/linux-3.6.tar.xz

  http://www.kernel.org/pub/linux/kernel/v3.x/patch-3.6.11.xz

  http://www.kernel.org/pub/linux/kernel/projects/rt/3.6/stable/patch-3.6.11.3.xz

  http://www.kernel.org/pub/linux/kernel/projects/rt/3.6/patch-3.6.11.3-rt35.patch.xz




Enjoy,

Re: [PATCH] sched: don't clear PF_THREAD_BOUND in select_fallback_rq

Luis Claudio R. Goncalves — 2013-05-14T13:08:21

| This is revert of "sched-clear-pf-thread-bound-on-fallback-rq.patch"
| (commit 0d939066acdcb in v3.4-rt),.
| 
| Select_fallback_rq() can be easilly called during system boot, because
| select_task_rq_fair() just return task_cpu(p) for bounded kernel threads,
| which is 0 during system boot and not in tsk_cpus_allowed, so
| select_fallback_rq() is called and PF_THREAD_BOUND is cleared. In my
| box, 1/3 bounded kernel threads will clear that flag after boot.
| 
| And it will cause problems, for example:
| # for pid in `ps -e -o pid`; do taskset -p -c 0-15 $pid; done
| this command will cause system hung.
| 
| What's more, I don't see why we need to clear this flag any more,
| because "cpu/rt: Rework cpu down for PREEMPT_RT" already remove the
| optimization for PF_THREAD_BOUND on migrate_disable/enable.
| 
| Signed-off-by: Qiang Huang <h.huangqiang< at >huawei.com>
| ---
|  kernel/sched/core.c |    6 ------
|  1 files changed, 0 insertions(+), 6 deletions(-)
| 
| diff --git a/kernel/sched/core.c b/kernel/sched/co

Highmem support on ppc32 plattform with PREEMPT_RT_FULL option

Heiko Schumann — 2013-05-14T11:04:34

Hello,

i´ve a ppc32 processor with 2 GB Ram. I use a 3.8.4 Kernel with vendor
and 3.8.4rt2 patch. To support  2 GB of RAM i´ve to set the kernel
option HIGHMEM. But this option is not available when PREEMPT_RT_FULL
is selected.
Is there a workaround or additional patch to use more than 1GB RAM on
a ppc32 with the option PREEMPT_RT_FULL?

Thanks
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo< at >vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

3.8.10-rt6 : Observing high latency as timer_interrupt is taking longer to exit

Jain Priyanka-B32167 — 2013-05-14T10:40:29

Hi,

I am running PREEMPT_RT  kernel on single-core PowerPC based platform and using cyclictest tool to measure latency.

Earlier I have used 2.6.33.9-rt31, With that kernel, latency has always remained below 50us irrespective of the traffic. 
With 3.8.10-rt6, latency is shooting beyond 200us irrespective of the traffic - no load, heavy traffic conditions included. Issue is generally reproduced within 10 minutes of run

When I tried to debug the issue using ftrace, I am observing that it is taking 200us for transition from timer_interrupt_exit(interrupts disabled) to timer_start  in Kworker/-297 thread.
I have tried multiple times and always observed similar trace. Please help me in debugging the issue.

Trace Logs:

cyclicte-3307    0d...2.. 104489596us+: hrtimer_start: hrtimer=dfee9e98 function=hrtimer_wakeup expires=1587365915748 softexpires=1587365865748
cyclicte-3307    0d...3.. 104489599us+: sched_stat_runtime: comm=cyclictest pid=3307 runtime=119552 [ns] vruntime=22040816169 [ns]
cyclicte-3307

Livelock in handle_pte_fault [Was: Re: timerfd read does not return]

Stanislav Meduna — 2013-05-14T08:31:09


The flags in the pagefault handler are 0x28 - if I understand
it correctly, FAULT_FLAG_KILLABLE | FAULT_FLAG_ALLOW_RETRY.
The faulting address is indeed the one from stack that worked
for hours before, is mlockall()-ed and I have (of course)
no swap. I will add some code to print the content of
the offending pte.

The code in handle_pte_fault proceeds through the
  entry = pte_mkyoung(entry);
line and the following
  ptep_set_access_flags
returns zero. This repeats ad nauseum without anything run
in between. I will add some tracing prints to output
the content of the pte.

Adding flush_tlb_page(vma, address) at the beginning of
handle_pte_fault does not change anything.

The length of the hang could correlate with the time until
some SCHED_OTHER process is scheduled after the RT throttler
activates. There is a process running each 2 seconds and the
length of the hang is usually between 1 and 3 seconds. This
is not (yet) verified.

I am starting to think that the virtual memory mapping of the
process got so

Re: [PATCH V2 Resend 4/4] timer: Migrate running timer

Thomas Gleixner — 2013-05-13T10:35:30

Which mechanism is migrating the timer away?


I have no objections to the functionality per se, but the proposed
solution is not going to fly.

Aside of bloating the data structure you're changing the semantics of
__mod_timer(). No __mod_timer() caller can deal with -EBUSY. So you'd
break the world and some more.

Here is a list of questions:

      - Which mechanism migrates timers?

      - How is that mechanism triggered?

      - How does that deal with CPU bound timers?

Thanks,

tglx



--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo< at >vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

3.8.11-rt8 NFS triggered seizures

Mike Galbraith — 2013-05-13T09:31:05

Letting my little Toshiba Satellite download an opensuse install DVD (at
20 KiB/s so I can use the other half of my wonderful bandwidth for
work), after it has been up over night, if I mount my desktop box, and
try to install a kernel to later test, laptop goes comatose.  It might
be workqueue related.  Interrupts are still happening, I can ping and
poke sysrq-c, but box is completely useless.

I got kdump working again (finally), and crashed it this morning.  Now a
few hours later, it fails to repeat of course, _seems_ it requires me
leaving it alone for an extended period to repeat.

I've switched kernels a few times, and it _seems_ to only happen with
3.8-rt, though I can't really be rock solid about that, can only say
3.8-rt has jammed up a few times, and no other kernel has. 

When make modules_install install hung first thing this morning, box was
responsive enough to fire up top, which displayed nada.  Desktop then
froze, so I crashed it.  Seems completion ain't gonna happen.

Trying to repeat, now th

Re: [PATCH V5 1/5] workqueues: Introduce new flag WQ_POWER_EFFICIENT for power oriented workqueues

Viresh Kumar — 2013-05-13T08:29:33

Sorry for the long delay for such a small change. I went on long leaves..

I have added following to make things more clear at places:
(Idle from scheduler's perspective. Which may or may not be physically idle)..

Let me know if it is still unclear..

And this is the new patch: (Attached it too for applying cleanly)

---------x---------------x-------------------

From: Viresh Kumar <viresh.kumar< at >linaro.org>
Date: Mon, 8 Apr 2013 16:45:40 +0530
Subject: [PATCH V5 resent 1/5] workqueues: Introduce new flag
WQ_POWER_EFFICIENT for power oriented workqueues

Workqueues can be performance or power-oriented. Currently, most workqueues are
bound to the CPU they were created on. This gives good performance (due to cache
effects) at the cost of potentially waking up otherwise idle cores (Idle from
scheduler's perspective. Which may or may not be physically idle) just to
process some work. To save power, we can allow the work to be rescheduled on a
core that is already awake.

Workqueues created with the WQ_UNBOUND f

Re: timerfd read does not return - caused by MM fault

Stanislav Meduna — 2013-05-13T08:05:02


Yep, this is a page fault. I added tracing of the
  mm_fault_error
  handle_mm_fault
  handle_pte_fault
  do_page_fault
functions


The result is

0....0 62811.755379: bprint:    timerfd_read: timerfd_read after unlock, res=0
0d...0 62811.755382: function:  do_page_fault
0....0 62811.755386: function:     handle_mm_fault
0....0 62811.755389: function:        handle_pte_fault
0d...0 62811.755394: function:  do_page_fault
0....0 62811.755396: function:     handle_mm_fault
0....0 62811.755398: function:        handle_pte_fault
0d...0 62811.755402: function:  do_page_fault
0....0 62811.755404: function:     handle_mm_fault
0....0 62811.755406: function:        handle_pte_fault



The number of minor faults of the thread jumped to 1433400 and
the process recovered after 2.77 seconds.

The trace file is at
  http://www.meduna.org/tmp/trace.mmfaulthang.dat.gz

Thanks

Re: timerfd read does not return - hangs inside put_user

Stanislav Meduna — 2013-05-12T23:20:32


Hmm... I added some trace_printks into fs/timerfd.c:


static ssize_t timerfd_read(
{
...
trace_printk("timerfd_read before unlock, res=%d", res);
spin_unlock_irq(&ctx->wqh.lock);
trace_printk("timerfd_read after unlock, res=%d", res);
if (ticks)
res = put_user(ticks, (u64 __user *) buf) ? -EFAULT: sizeof(ticks);
trace_printk("timerfd_read return, res=%d, ticks=%llu, buf=%p", res, (unsigned long long) ticks, buf);
return res;
}

The first two are printed. The last one is not.

0....0 timerfd_read: timerfd_read before unlock, res=0
0....0 timerfd_read: timerfd_read after unlock, res=0

The usual

0....0 timerfd_read: timerfd_read return, res=8, ticks=1, buf=0xb7600158

does _not_ happen here.

So it looks like the problem happens inside the put_user,
maybe a pagefault? The buf is an address on the stack:

while(alive) {
u64 exp;

if (read(tmrFd, &exp, sizeof(exp)) != sizeof(exp) || exp < 1)
continue;
...

and the process is running with mlockall(MCL_CURRENT | MCL_FUTURE),
so the whole stack is f