gmane.linux.kernel.aio.general

Re: libaio status

Chris Farey — 2012-05-17T09:16:05

We've got an application that uses aio with O_DIRECT to LVM volumes. Originally 
we had io_submit() and io_getevents() in separate threads, but as part of a 
recent re-write moved to using eventfd() with poll() to determine when 
completions are ready. There are other optimizations in the code, and generally 
the new, non-threaded code is 5-10% faster.

When I profile the code under load, it show 70-80% of the time waiting in 
io_submit() - but I'm suspicious of these numbers, because if I try modifying 
the new code to put io_submit() in a separate thread, it does not make any 
perceptible difference to throughput.

I would like to know, under what circumstances, when using O_DIRECT to a block 
device, can io_submit() block?

Chris Farey

On 16/05/2012 21:07, Raz wrote:

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo< at >kvack.org.  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart< at >kvack.org">aart< at >kvack.org</a>

Re: libaio status

Ted Ts'o — 2012-05-16T22:15:11

As far as I know any allocating write (extending or writing into the
middle of a sprase, non-fallocated region) will block on all file
systems.  The current AIO framework calls the file-system specific
bmap function, and the only thing which is asynchronous is the actual
Data I/O block itself.

I looked into fixing it, and it would have creating an entirely new
AIO framework (which would have initially only have been used for
ext4).  And then it would had a number of interesting problems in a
heavily containerized world (i.e., where you have multiple cgroups
strictly allocating how much memory, CPU, and prop I/O you allow to be
used to each group of processes comprising a job), since if you run
the metadata I/O requests out of a workgroup, you'll be constrained by
whatever cgroup was used to mount the file system, which might not
normally be given huge amounts of CPU and memory to play with.

And you *do* have to run out of some kind of foreground process
context, since the Block I/O layer doesn't support m

Re: libaio status

Jeff Moyer — 2012-05-16T20:53:41


Libaio doesn't really have a website.  That page is a part of the old
linux scalability effort website.


Looks pretty good, but I didn't read too closely.


Well, how do you define "good?"  Libaio will work just fine on regular
files when opened with O_DIRECT.  As mentioned elsewhere, buffered I/O
is not supported in an asynchronous manner, so io_submit will block
until the I/O is complete.

Some caveats:
- if a metadata read is required, that can block io_submit
- if there is memory pressure, memory allocations can block io_submit
- if the file is not preallocated, some file systems will fall back to
  buffered I/O for hole filling (so io_submit will block until the I/O is
  complete).  Since you explicitly mentioned ext4, you don't have this
  problem, so long as you use a sufficiently recent kernel.

If it is at all possible, I would preallocate your files before doing
AIO to them.


If you can't afford *any* blocking, then you'll have to farm out the
io_submit call to another thread, as another person

Re: libaio status

Raz — 2012-05-16T20:07:00

libio is not truely asynchronous -- last time i checked. problem is
that io_submit may take far too long time.
I suggest you profile io_submit yourself. I was forced to move
io_submit to a different thread.


On Tue, May 15, 2012 at 4:46 PM, Leandro Lucarella <luca< at >llucax.com.ar> wrote:

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo< at >kvack.org.  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart< at >kvack.org">aart< at >kvack.org</a>

libaio status

Leandro Lucarella — 2012-05-15T13:46:22

Hello. I'm looking for alternatives to do async I/O in Linux and it
looks like libaio is the only "native" way to do it, but the information
about it is really scarce and incomplete.

libaio website (if [1] is the website) looks outdated and have a lot of
broken links. The manpage footers says Linux 2.4 and I can't find any
mention about the restrictions mentioned in the website, so I don't know
if those restrictions are up to date and accurate.

I found this document [2], which so far looks like the most
comprehensive and up to date resource available, but again, being not an
"official" document, I'm not sure how accurate is it.

My feeling is libaio is only good to do O_DIRECT I/O on raw block
devices, but I needed for regular files in an ext4 filesystem (ideally
I shouldn't  impose a limitation on the filesystem to use, but I guess
I could do that if necessary). I need to do I/O in a server that needs
to have extremely low latency, so I can't afford any type of blocking.
Using threads is not a viable opti

Re: What's the usage of io_setup?

Ryan Wang — 2012-04-19T15:44:18

Thanks, Jeff.

在 2012年4月19日 下午10:51，Jeff Moyer <jmoyer< at >redhat.com>写道：

Re: What's the usage of io_setup?

Jeff Moyer — 2012-04-19T14:51:27


No, you have misread the code.


To submit I/O.  You can try looking through the man pages for a better
understanding.  Or, you could have a look at other code which uses the
aio interface (such as fio, or aio-stress, or any number of other
projects).

Cheers,
Jeff

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo< at >kvack.org.  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart< at >kvack.org">aart< at >kvack.org</a>

What's the usage of io_setup?

Ryan Wang — 2012-04-19T01:58:58

Hi,

I'm new to libaio, and have some question about io_setup.

I walked through the code of io_setup, and found that it frees the ioctx
after obtain ioctx->user_id. So the caller just gets a handle for a freed
ioctx, right?

So what's the usage of io_submit in real word applications?

thanks,
Ryan

Re: Where can I find the git tree for libaio?

Ryan Wang — 2012-04-17T00:08:34

在 2012年4月17日 上午4:53，Jeff Moyer <jmoyer< at >redhat.com>写道：

Re: Where can I find the git tree for libaio?

Jeff Moyer — 2012-04-16T20:53:58


The new location is here:
  http://git.fedorahosted.org/git/?p=libaio.git

Sorry for the delay.

Cheers,
Jeff

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo< at >kvack.org.  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart< at >kvack.org">aart< at >kvack.org</a>

Where can I find the git tree for libaio?

Ryan Wang — 2012-04-16T06:18:04

Hi,

I'm studying libaio recently, and now I want to get the upstream libaio.
I wonder where can I find the git tree for libaio, please?

thanks,
Ryan

Re: [patch] aio: change a stray spin_unlock_bh() to spin_unlock()

Al Viro — 2012-03-20T19:13:23

Nice catch; folded into the offending commit

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo< at >kvack.org.  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart< at >kvack.org">aart< at >kvack.org</a>

[patch] aio: change a stray spin_unlock_bh() to spin_unlock()

Dan Carpenter — 2012-03-20T13:09:19

We missed this spin_unlock_bh() when we removed the _bh from the other
locks in cb22bbe9f7 "aio: aio_nr_lock is taken only synchronously now"

Signed-off-by: Dan Carpenter <dan.carpenter< at >oracle.com>

diff --git a/fs/aio.c b/fs/aio.c
index 7b6b9d5..4f71627 100644
--- a/fs/aio.c
+++ b/fs/aio.c
< at >< at > -280,7 +280,7 < at >< at > static struct kioctx *ioctx_alloc(unsigned nr_events)
 spin_lock(&aio_nr_lock);
 if (aio_nr + nr_events > aio_max_nr ||
     aio_nr + nr_events < aio_nr) {
-spin_unlock_bh(&aio_nr_lock);
+spin_unlock(&aio_nr_lock);
 goto out_cleanup;
 }
 aio_nr += ctx->max_reqs;

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo< at >kvack.org.  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart< at >kvack.org">aart< at >kvack.org</a>

Re: [PATCH] aio: fix the "too late munmap()" race

Jeff Moyer — 2012-03-08T18:15:20


Looks good to me.

Reviewed-by: Jeff Moyer <jmoyer< at >redhat.com>


--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo< at >kvack.org.  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart< at >kvack.org">aart< at >kvack.org</a>

Re: [PATCH] aio: fix io_setup/io_destroy race

Jeff Moyer — 2012-03-07T18:16:18


Al, you certainly are creative.  ;-) I agree with the problem and the
fix.  It would be nice, though if you had added comments.

I ran xfstests ./check -g aio, and there were no problems.

Reviewed-by: Jeff Moyer <jmoyer< at >redhat.com>

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo< at >kvack.org.  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart< at >kvack.org">aart< at >kvack.org</a>

[patch] aio: wake up waiters when freeing unused kiocbs

Jeff Moyer — 2012-02-16T19:56:15

Hi,

Bart Van Assche reported a hung fio process when either hot-removing
storage or when interrupting the fio process itself.  The (pruned) call
trace for the latter looks like so:

fio             D 0000000000000001     0  6849   6848 0x00000004
 ffff880092541b88 0000000000000046 ffff880000000000 ffff88012fa11dc0
 ffff88012404be70 ffff880092541fd8 ffff880092541fd8 ffff880092541fd8
 ffff880128b894d0 ffff88012404be70 ffff880092541b88 000000018106f24d
Call Trace:
 [<ffffffff813b683f>] schedule+0x3f/0x60
 [<ffffffff813b68ef>] io_schedule+0x8f/0xd0
 [<ffffffff81174410>] wait_for_all_aios+0xc0/0x100
 [<ffffffff81175385>] exit_aio+0x55/0xc0
 [<ffffffff810413cd>] mmput+0x2d/0x110
 [<ffffffff81047c1d>] exit_mm+0x10d/0x130
 [<ffffffff810482b1>] do_exit+0x671/0x860
 [<ffffffff81048804>] do_group_exit+0x44/0xb0
 [<ffffffff81058018>] get_signal_to_deliver+0x218/0x5a0
 [<ffffffff81002065>] do_signal+0x65/0x700
 [<ffffffff81002785>] do_notify_resume+0x65/0x80
 [<ffffffff813c0333>] int_signal+0x12/0x17

The problem lies w

[PATCH 27/60] fs: remove the second argument of k[un]map_atomic()

Cong Wang — 2012-02-10T05:39:48

Acked-by: Benjamin LaHaise <bcrl< at >kvack.org>
Signed-off-by: Cong Wang <amwang< at >redhat.com>
---
 fs/aio.c            |   30 +++++++++++++++---------------
 fs/bio-integrity.c  |   10 +++++-----
 fs/exec.c           |    4 ++--
 fs/namei.c          |    4 ++--
 fs/pipe.c           |    8 ++++----
 fs/splice.c         |    7 ++-----
 include/linux/bio.h |    8 ++++----
 7 files changed, 34 insertions(+), 37 deletions(-)

diff --git a/fs/aio.c b/fs/aio.c
index 969beb0..04ae7e2 100644
--- a/fs/aio.c
+++ b/fs/aio.c
< at >< at > -160,7 +160,7 < at >< at > static int aio_setup_ring(struct kioctx *ctx)
 
 info->nr = nr_events;/* trusted copy */
 
-ring = kmap_atomic(info->ring_pages[0], KM_USER0);
+ring = kmap_atomic(info->ring_pages[0]);
 ring->nr = nr_events;/* user copy */
 ring->id = ctx->user_id;
 ring->head = ring->tail = 0;
< at >< at > -168,32 +168,32 < at >< at > static int aio_setup_ring(struct kioctx *ctx)
 ring->compat_features = AIO_RING_COMPAT_FEATURES;
 ring->incompat_features = AIO_RING_INCOMPAT_FEATURES;
 ring->header_length = si

Re: [PATCH] AIO: Don't plug the I/O queue in do_io_submit()

Jeff Moyer — 2011-12-16T14:45:07


I believe the original plugging here was done on a per fd basis.  So, I
concede that the behaviour may have changed a bit since the initial
patch for this was merged.


I have a patch slated for 3.2 that should help that.  It batches the
allocation of the aio requests, which showed a good improvement in
microbenchmarks there.

commit 080d676de095a14ecba14c0b9a91acb5bbb634df
Author: Jeff Moyer <jmoyer< at >redhat.com>
Date:   Wed Nov 2 13:40:10 2011 -0700

    aio: allocate kiocbs in batches

Cheers,
Jeff

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo< at >kvack.org.  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart< at >kvack.org">aart< at >kvack.org</a>

Re: [PATCH] AIO: Don't plug the I/O queue in do_io_submit()

Chris Mason — 2011-12-15T16:40:49

Each io_submit call is sending down about 34K of IO to two different devices.
The latencies were measured just on the process writing the redo
logs, so it is a very specific subset of the overall benchmark.

The patched kernel only does 4x more iops for the redo logs than the
unpatched kernel, so we're talking ~8K ios here.

-chris

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo< at >kvack.org.  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart< at >kvack.org">aart< at >kvack.org</a>

Re: [PATCH] AIO: Don't plug the I/O queue in do_io_submit()

Jens Axboe — 2011-12-15T16:15:26

I think that would indeed be an interesting addition to test on top of
the 3.0 kernel being used.

This is a bit of a sticky situation. We want the plugging and merging on
rotational storage, and on SSDs we want the batch addition to the queue
to avoid hammering on the queue lock. At this level, we have no idea.
But we don't want to introduce longer latencies. So the question is, are
these latencies due to long queues (and hence would be helped with the
auto-replug on 3.1 and newer), or are they due to the submissions
running for too long. If the latter, then we can either look into
reducing the time spent between submitting the individual pieces. Or at
least not holding up too long.

Re: [PATCH] AIO: Don't plug the I/O queue in do_io_submit()

Shaohua Li — 2011-12-15T01:09:04

2011/12/14 Dave Kleikamp <dave.kleikamp< at >oracle.com>:
can you explain why this can help? Note, in 3.1 kernel we now force flush
plug list if the list is too long, which will remove a lot of latency.

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo< at >kvack.org.  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart< at >kvack.org">aart< at >kvack.org</a>