gmane.linux.kernel.aio.general

libaio status

Leandro Lucarella — 2012-05-15T13:46:22

Hello. I'm looking for alternatives to do async I/O in Linux and it
looks like libaio is the only "native" way to do it, but the information
about it is really scarce and incomplete.

libaio website (if [1] is the website) looks outdated and have a lot of
broken links. The manpage footers says Linux 2.4 and I can't find any
mention about the restrictions mentioned in the website, so I don't know
if those restrictions are up to date and accurate.

I found this document [2], which so far looks like the most
comprehensive and up to date resource available, but again, being not an
"official" document, I'm not sure how accurate is it.

My feeling is libaio is only good to do O_DIRECT I/O on raw block
devices, but I needed for regular files in an ext4 filesystem (ideally
I shouldn't  impose a limitation on the filesystem to use, but I guess
I could do that if necessary). I need to do I/O in a server that needs
to have extremely low latency, so I can't afford any type of blocking.
Using threads is not a viable opti

What's the usage of io_setup?

Ryan Wang — 2012-04-19T01:58:58

Hi,

I'm new to libaio, and have some question about io_setup.

I walked through the code of io_setup, and found that it frees the ioctx
after obtain ioctx->user_id. So the caller just gets a handle for a freed
ioctx, right?

So what's the usage of io_submit in real word applications?

thanks,
Ryan

Where can I find the git tree for libaio?

Ryan Wang — 2012-04-16T06:18:04

Hi,

I'm studying libaio recently, and now I want to get the upstream libaio.
I wonder where can I find the git tree for libaio, please?

thanks,
Ryan

[patch] aio: change a stray spin_unlock_bh() to spin_unlock()

Dan Carpenter — 2012-03-20T13:09:19

We missed this spin_unlock_bh() when we removed the _bh from the other
locks in cb22bbe9f7 "aio: aio_nr_lock is taken only synchronously now"

Signed-off-by: Dan Carpenter <dan.carpenter< at >oracle.com>

diff --git a/fs/aio.c b/fs/aio.c
index 7b6b9d5..4f71627 100644
--- a/fs/aio.c
+++ b/fs/aio.c
< at >< at > -280,7 +280,7 < at >< at > static struct kioctx *ioctx_alloc(unsigned nr_events)
 spin_lock(&aio_nr_lock);
 if (aio_nr + nr_events > aio_max_nr ||
     aio_nr + nr_events < aio_nr) {
-spin_unlock_bh(&aio_nr_lock);
+spin_unlock(&aio_nr_lock);
 goto out_cleanup;
 }
 aio_nr += ctx->max_reqs;

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo< at >kvack.org.  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart< at >kvack.org">aart< at >kvack.org</a>

[patch] aio: wake up waiters when freeing unused kiocbs

Jeff Moyer — 2012-02-16T19:56:15

Hi,

Bart Van Assche reported a hung fio process when either hot-removing
storage or when interrupting the fio process itself.  The (pruned) call
trace for the latter looks like so:

fio             D 0000000000000001     0  6849   6848 0x00000004
 ffff880092541b88 0000000000000046 ffff880000000000 ffff88012fa11dc0
 ffff88012404be70 ffff880092541fd8 ffff880092541fd8 ffff880092541fd8
 ffff880128b894d0 ffff88012404be70 ffff880092541b88 000000018106f24d
Call Trace:
 [<ffffffff813b683f>] schedule+0x3f/0x60
 [<ffffffff813b68ef>] io_schedule+0x8f/0xd0
 [<ffffffff81174410>] wait_for_all_aios+0xc0/0x100
 [<ffffffff81175385>] exit_aio+0x55/0xc0
 [<ffffffff810413cd>] mmput+0x2d/0x110
 [<ffffffff81047c1d>] exit_mm+0x10d/0x130
 [<ffffffff810482b1>] do_exit+0x671/0x860
 [<ffffffff81048804>] do_group_exit+0x44/0xb0
 [<ffffffff81058018>] get_signal_to_deliver+0x218/0x5a0
 [<ffffffff81002065>] do_signal+0x65/0x700
 [<ffffffff81002785>] do_notify_resume+0x65/0x80
 [<ffffffff813c0333>] int_signal+0x12/0x17

The problem lies w

[PATCH] AIO: Don't plug the I/O queue in do_io_submit()

Dave Kleikamp — 2011-12-13T21:44:45

Asynchronous I/O latency to a solid-state disk greatly increased
between the 2.6.32 and 3.0 kernels. By removing the plug from
do_io_submit(), we observed a 34% improvement in the I/O latency.

Unfortunately, at this level, we don't know if the request is to
a rotating disk or not.

Signed-off-by: Dave Kleikamp <dave.kleikamp< at >oracle.com>
Cc: linux-aio< at >kvack.org
Cc: Chris Mason <chris.mason< at >oracle.com>
Cc: Jens Axboe <axboe< at >kernel.dk>
Cc: Andi Kleen <ak< at >linux.intel.com>
Cc: Jeff Moyer <jmoyer< at >redhat.com>

diff --git a/fs/aio.c b/fs/aio.c
index 78c514c..d131a2c 100644
--- a/fs/aio.c
+++ b/fs/aio.c
< at >< at > -1696,7 +1696,6 < at >< at > long do_io_submit(aio_context_t ctx_id, long nr,
 struct kioctx *ctx;
 long ret = 0;
 int i = 0;
-struct blk_plug plug;
 struct kiocb_batch batch;
 
 if (unlikely(nr < 0))
< at >< at > -1716,8 +1715,6 < at >< at > long do_io_submit(aio_context_t ctx_id, long nr,
 
 kiocb_batch_init(&batch, nr);
 
-blk_start_plug(&plug);
-
 /*
  * AKPM: should this return a partial result if some of the IOs were
  * succe

AIO kernel.org websites

Dan Cecile — 2011-11-26T03:05:57

Hi,

I'm trying to find the correct websites for the AIO project. I've
tried a few kernel.org addresses, without any luck:

  (main page) http://www.kernel.org/pub/linux/libs/aio/

  (userspace GIT) git://git.kernel.org/pub/scm/libs/libaio/libaio.git

  (userspace FTP) ftp://ftp.kernel.org/pub/linux/libs/aio/

Should these addresses be working after September's kernel.org
infrastructure changes?

Best regards,
Dan Cecile

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo< at >kvack.org.  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart< at >kvack.org">aart< at >kvack.org</a>

[PATCH][RFC] A readahead complete notify approach to implement buffer aio

Zhu Yanhai — 2011-11-01T09:00:24

The current libaio/aio has to be Direct-IO, otherwise it falls back into sync IO.
However, the aio core has already been asychronous naturally. This patch adds a complete
notify mechanism to implement buffer aio, the main idea is to readahead()-like in
io_submit(), counts the non-uptodated pages assocaiated with each iocb, then put each ref
in the bio complete path just before unlock_page(), and hook them on to the aio ring buffer
finally when the ref drops to zero. In io_getevents(), we call vfs_read() as a safe net
since there is still little possibility that the pages had brought in were reclaimed
between io_submit() and io_getevents().

I have tested this patch for a while, for the small size random io request, its
performance is more or less the same with the traditional aio, for the big io request,
the overhead of one extra memory copy arises.

I think so far it has at least below obvious drawbacks,

* mpage_readpage() is a really narrow interface, I have no way to pass down
the new control struct baio

blocking io_submit

2011-09-23T02:25:33

I've just realized the magnitude of the blocking
io_submit().

In my test [1], run on ubuntu 11.04, the majority of
the jobs submitted in io_submit are complete by the
time it returns. i.e. the level of asynchrony is fairly
low (for being an async. API).

Graphing the output [2] shows that the test spends essentially
all its time blocked by io_submit (as opposed to waiting
for completion events).

My attempt as a (user-level) work-around is to spawn a few
threads who spend their time submitting jobs. At least
my main disk thread won't get blocked, and I can react to
completion events much sooner. It also lets me submit jobs
sooner, by submitting from more than one thread.

Are there any caveats with this approach?

Does anyone have a better or different idea of ways to
work around this problem?

thanks,

[patch, v3] aio: allocate kiocbs in batches

Jeff Moyer — 2011-09-22T16:41:51

Hi,

In testing aio on a fast storage device, I found that the context lock
takes up a fair amount of cpu time in the I/O submission path.  The
reason is that we take it for every I/O submitted (see __aio_get_req).
Since we know how many I/Os are passed to io_submit, we can preallocate
the kiocbs in batches, reducing the number of times we take and release
the lock.  In my testing, I was able to reduce the amount of time spent
in _raw_spin_lock_irq by .56% (average of 3 runs).  The command I used
to test this was:
   aio-stress -O -o 2 -o 3 -r 8 -d 128 -b 32 -i 32 -s 16384 <dev>

I also tested the patch with various numbers of events passed to
io_submit, and I ran the xfstests aio group of tests to ensure I didn't
break anything.

Signed-off-by: Jeff Moyer <jmoyer< at >redhat.com>

---
Changes from v2 -> v3:
- got rid of an extraneous structure member in the kiocb_batch.
- fixed up some haphazard variable types
- fixed a build warning

Changes from rfc -> v2:
- folded in akpm's incremental patch which fixes codin

[patch, v2] aio: allocate kiocbs in batches

Jeff Moyer — 2011-09-21T17:16:00

Hi,

In testing aio on a fast storage device, I found that the context lock
takes up a fair amount of cpu time in the I/O submission path.  The
reason is that we take it for every I/O submitted (see __aio_get_req).
Since we know how many I/Os are passed to io_submit, we can preallocate
the kiocbs in batches, reducing the number of times we take and release
the lock.  In my testing, I was able to reduce the amount of time spent
in _raw_spin_lock_irq by .56% (average of 3 runs).  The command I used
to test this was:
   aio-stress -O -o 2 -o 3 -r 8 -d 128 -b 32 -i 32 -s 16384 <dev>

I also tested the patch with various numbers of events passed to
io_submit, and I ran the xfstests aio group of tests to ensure I didn't
break anything.

Signed-off-by: Jeff Moyer <jmoyer< at >redhat.com>

---
Changes from rfc -> v2:
- folded in akpm's incremental patch which fixes coding style and
  variable names
- tried to clarify a comment about a starvation case
- fixed up my breaking of the handling of that starvation case
- moved

io_getevents() segfaults

Cyril Hrubis — 2011-03-23T17:10:23

Hi!
I've been fixing libaio tests in LTP and found that io_getevents() may
segfault on random ctx.

The manual however says that we should get EINVAL in this case.

The cause for this is code in io_getevents() that dereferences ctx to
check for empty queue.

I've been told by Jeff that this code was flawed and not really
implemented anyway so attached patch simply removes it.

[PATCH] aio: Wake all waiters when destroying ctx

Roland Dreier — 2011-03-11T05:55:37

From: Roland Dreier <roland< at >purestorage.com>

The test program below will hang because io_getevents() uses
add_wait_queue_exclusive(), which means the wake_up() in io_destroy()
only wakes up one of the threads.  Fix this by using wake_up_all() in
the aio code paths where we want to make sure no one gets stuck.

// t.c -- compile with gcc -lpthread -laio t.c

#include <libaio.h>
#include <pthread.h>
#include <stdio.h>
#include <unistd.h>

static const int nthr = 2;

void *getev(void *ctx)
{
struct io_event ev;
io_getevents(ctx, 1, 1, &ev, NULL);
printf("io_getevents returned\n");
return NULL;
}

int main(int argc, char *argv[])
{
io_context_t ctx = 0;
pthread_t thread[nthr];
int i;

io_setup(1024, &ctx);

for (i = 0; i < nthr; ++i)
pthread_create(&thread[i], NULL, getev, ctx);

sleep(1);

io_destroy(ctx);

for (i = 0; i < nthr; ++i)
pthread_join(thread[i], NULL);

return 0;
}

Cc: <stable< at >kernel.org>
Signed-off-by: Roland Dreier <roland< at >purestorage.com>
---
 fs/aio.

The program.

Stef Bon — 2011-01-31T09:47:26

Forgot to attach the program.
Please this program could be much better, the checking of errors like
ENOMEM could be better. I know.
But it does what it's written for. Do aio writes to an ordinary file
at a non aligned offset.

Stef

Some questions.

Stef Bon — 2011-01-31T09:34:42

Hello,

as I've already mentioned here, I'm working on a fuse fs, and want to
make it work with aio.

I've some questions:

a. The aiocp example is a very good example. It's much better than the
aiocp in the man file of io, which totallly does not mention the
alignment.
My advice is to replace that. I've been trying to make aio work for
weeks now, and it's very frustating to not find the right
documentation. Just some days ago I first got the idea how important
the alignment is, when doing aio. The manpage should mention that.

About alignment, Jeff Moyer told me that when reading or writing from
a sata block device, the value alignment is the (logical) blocksize,
which you can detect with BLKSSZGET.
Isn't it the same value when doing a stavfs? My fs can do this once
for every underlying fs/blockdevice and can get this value once and
can store this value for later use, in stead of doing this every time
when doing an aio.

b. When looking at the aiocp example, at the start an array of iocb is
initialised :

i

Problem using compile option -D_FILE_OFFSET_BITS=64.

Stef Bon — 2011-01-28T11:57:04

Hi,

I'm trying to make my fuse fs use aio.

I've sent an email about this earlier to this list.

Now I've written a small test program to try to test the aio in combination with
signals and epoll.

This program works very good, at least when it's compiled like:


gcc -Wall -lrt testsignal.c -o testsignal


It's a advanced version of the test program found in the manpage of signalfd.

now test it:

./testsignal
mainloop: adding signalfd 4 to epoll
^CGot SIGINT: start the test aio read.


the progress in logfile:

Jan 28 11:58:08 clfs20091030 testsignal: aio_read_file
Jan 28 11:58:08 clfs20091030 testsignal: aio_read return: 0
Jan 28 11:58:08 clfs20091030 testsignal: mainloop aio read, no error,
nbytes: 14196767
Jan 28 11:58:08 clfs20091030 testsignal: aio_write_file
Jan 28 11:58:08 clfs20091030 testsignal: aio_write return: 0
Jan 28 11:58:08 clfs20091030 testsignal: mainloop aio write, no error,
nbytes: 14196767


This is as expected.

Now when I compile it using the -D_FILE_OFFSET_BITS=64

the same logoutpu

problem using aio in fuse fs.

Stef Bon — 2011-01-26T12:02:12

Hi,

I'm writing a fuse fs, fuse-workspace, which helps to create a
userfriendly, effective and powerfull layer for the average user. See:


http://linux.bononline.nl/wiki/index.php/Changes_and_issues

for changes and recent screenshots.

Mainpage:

http://linux.bononline.nl/wiki/index.php/Mount.md5key


Well the fuse-fs is the cosmetic layer, especially when in "GoboLinux" mode.
Blocking behaviour is very unwanted, cause this will block every app
running in userspace, wanting to do some io.
And then I mean every io.

Read and writes at this moment are blocking.


Now FUSE offers a construction to handle aio very easy.
By using a single thead, and in the mainloop a epoll instance,
watching standard the fuse fd for incoming fs events, like the call
getattr, open and read.
Now the processing of this request is done in a fuse fs specific call.
When ready it will send the results back using fuse_reply, using the
request handle to indentify it.


Now using aio here is very possible by:

a. add an eventfd to the l

[patch] aio: remove unused function, aio_run_iocbs

Jeff Moyer — 2011-01-05T21:19:50

Hi,

aio_run_iocbs is not used at all, so get rid of it.

Signed-off-by: Jeff Moyer <jmoyer< at >redhat.com>

diff --git a/fs/aio.c b/fs/aio.c
index 8c8f6c5..6ca2f96 100644
--- a/fs/aio.c
+++ b/fs/aio.c
< at >< at > -798,30 +798,14 < at >< at > static void aio_queue_work(struct kioctx * ctx)
 queue_delayed_work(aio_wq, &ctx->wq, timeout);
 }
 
-
 /*
- * aio_run_iocbs:
+ * aio_run_all_iocbs:
  * Process all pending retries queued on the ioctx
- * run list.
+ * run list, and keep running them until the list
+ * stays empty.
  * Assumes it is operating within the aio issuer's mm
  * context.
  */
-static inline void aio_run_iocbs(struct kioctx *ctx)
-{
-int requeue;
-
-spin_lock_irq(&ctx->ctx_lock);
-
-requeue = __aio_run_iocbs(ctx);
-spin_unlock_irq(&ctx->ctx_lock);
-if (requeue)
-aio_queue_work(ctx);
-}
-
-/*
- * just like aio_run_iocbs, but keeps running them until
- * the list stays empty
- */
 static inline void aio_run_all_iocbs(struct kioctx *ctx)
 {
 spin_lock_irq(&ctx->ctx_lock);

--
To unsubscribe, send a message

[PATCH] aio: remove unnecessary check

Namhyung Kim — 2010-12-16T09:09:05

'nr >= min_nr >= 0' always satisfies 'nr >= 0' so the check is unnecesary.

Signed-off-by: Namhyung Kim <namhyung< at >gmail.com>
---
 fs/aio.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/fs/aio.c b/fs/aio.c
index c56153f5f73e..45766460fa57 100644
--- a/fs/aio.c
+++ b/fs/aio.c
< at >< at > -1839,7 +1839,7 < at >< at > SYSCALL_DEFINE5(io_getevents, aio_context_t, ctx_id,
 long ret = -EINVAL;
 
 if (likely(ioctx)) {
-if (likely(min_nr <= nr && min_nr >= 0 && nr >= 0))
+if (likely(min_nr <= nr && min_nr >= 0))
 ret = read_events(ioctx, min_nr, nr, events, timeout);
 put_ioctx(ioctx);
 }

[PATCH] aio: using hash table for active requests

Li Yu — 2010-12-15T03:03:05

This patch remove a TODO in fs/aio.c, that is to use hash table for active requests.

I prefer add an iocb at tail of collision chain, so I do not use hlist here.

Signed-off-by: Li Yu <raise.sail< at >gmail.com>
---

 fs/aio.c            |   90 ++++++++++++++++++++++++++++++++++++++--------------
 include/linux/aio.h |    2 -
 2 files changed, 68 insertions(+), 24 deletions(-)

diff --git a/fs/aio.c b/fs/aio.c
index 8c8f6c5..fee2aa3 100644
--- a/fs/aio.c
+++ b/fs/aio.c
< at >< at > -65,6 +65,15 < at >< at > static DECLARE_WORK(fput_work, aio_fput_routine);
 static DEFINE_SPINLOCK(fput_lock);
 static LIST_HEAD(fput_head);
 
+#if BITS_PER_LONG == 64
+#define AIO_ACTREQ_BUCKETS_SHIFT36
+#elif BITS_PER_LONG == 32
+#define AIO_ACTREQ_BUCKETS_SHIFT        24
+#endif
+
+/* AIO_ACTREQ_BUCKETS must be power of 2 */
+#define AIO_ACTREQ_BUCKETS(2*PAGE_SIZE/sizeof(struct list_head))
+
 #define AIO_BATCH_HASH_BITS3 /* allocated on-stack, so don't go crazy */
 #define AIO_BATCH_HASH_SIZE(1 << AIO_BATCH_HASH_BITS)
 struct aio_batch_entry {
< at >< at >

[PATCH] aio: check return value of create_workqueue()

Namhyung Kim — 2010-12-13T15:06:25

Signed-off-by: Namhyung Kim <namhyung< at >gmail.com>
---
 fs/aio.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/fs/aio.c b/fs/aio.c
index 8c8f6c5b6d79..c56153f5f73e 100644
--- a/fs/aio.c
+++ b/fs/aio.c
< at >< at > -87,7 +87,7 < at >< at > static int __init aio_setup(void)
 
 aio_wq = create_workqueue("aio");
 abe_pool = mempool_create_kmalloc_pool(1, sizeof(struct aio_batch_entry));
-BUG_ON(!abe_pool);
+BUG_ON(!aio_wq || !abe_pool);
 
 pr_debug("aio_setup: sizeof(struct page) = %d\n", (int)sizeof(struct page));