gmane.linux.file-systems.cachefs.general

Re: [PATCH] fs: cachefiles: Add support for largefiles in filesystem caching

Suresh Jayaraman — 2012-05-22T05:05:58

Hello David,

This patch doesn't seem to have made it to linux-next (for May 21),
yet. Could you kindly re-check?

Thanks
Suresh



On Tue, May 1, 2012 at 2:43 PM, David Howells <dhowells< at >redhat.com> wrote:

NFS caching on the server-side

Abraham Aldaco — 2012-04-26T16:06:02

Hello,
I need help for configuring a NFS version 4 server for caching what clients read
from it. That is, the server has to cache locally (not the client) what any NFS
client reads from this server. The porpouse of this configuration if to relief
the work load of the server HDD.

Any suggestion or recommendation will be appreciated.
Abraham Aldaco
abraham.aldaco< at >gmail.com

nfs/fscache writeback cache

Sertaç Olgunsoylu — 2012-04-26T03:10:28

Hi,
First of all, I am not sure if this list is appropriate place to ask that
question but I think it is the most relevant one. I saw this project
proposal[1] on the Fedora Project's GSOC 2012 ideas page few weeks ago. It
proposes to add writeback caching support to NFS. I didn't apply to GSOC
since I am not a student but I want to spend my time working on that
project although this is not accepted by GSOC. I want to learn that is
there any ongoing effort on that side? If any, could you point me to that
so that i will not duplicate work and I can contribute to current project.

[1] -
http://fedoraproject.org/wiki/Summer_coding_ideas_for_2012#Linux_kernel_project

Thanks

Re: [PATCH] fs: cachefiles: Add support for largefiles in filesystem caching

Suresh Jayaraman — 2012-04-11T14:57:22

While preparing a similar patch to make cachefiles support files larger
than 2GB, I noticed that you had beaten me to it. Confirming that this
patch allows caching of larger files which would not be cached otherwise.

Tested-by: Suresh Jayaraman <sjayaraman< at >suse.com>


diff --git a/fs/cachefiles/rdwr.c b/fs/cachefiles/rdwr.c

oops when I mount nfs with -o fsc

fanchaoting — 2012-03-28T03:48:38

Hi: in nfs(RHEL6.x), i do as following. 
#cat /etc/exports
    /nfsroot      *(rw,no_root_squash,fsid=0,insecure)
#mount –t nfs4 –o fsc localhost:/   /nfsmnt
#sh create_file.sh

wait a while, i find oops.

PID: 4196   TASK: ffff88012365b580  CPU: 0   COMMAND: "sh"
 #0 [ffff880139435750] machine_kexec at ffffffff810310db
 #1 [ffff8801394357b0] crash_kexec at ffffffff810b63b2
 #2 [ffff880139435880] oops_end at ffffffff814dec50
 #3 [ffff8801394358b0] die at ffffffff8100f2fb
 #4 [ffff8801394358e0] do_trap at ffffffff814de544
 #5 [ffff880139435940] do_invalid_op at ffffffff8100ceb5
 #6 [ffff8801394359e0] invalid_op at ffffffff8100bf5b
    [exception RIP: __nfs_fscache_invalidate_page+121]
 #7 [ffff880139435ab0] nfs_invalidate_page at ffffffffa06fff6e [nfs]
 #8 [ffff880139435ad0] do_invalidatepage at ffffffff811243d5
 #9 [ffff880139435ae0] truncate_inode_page at ffffffff811245f2
#10 [ffff880139435b00] truncate_inode_pages_range at ffffffff811248f0
#11 [ffff880139435bf0] truncate_inode_pages at ffffffff81124c05
#12 [ffff880139435c00] truncate_pagecache at ffffffff81124c57
#13 [ffff880139435c30] nfs_setattr_update_inode at ffffffffa07033ab [nfs]
#14 [ffff880139435c70] nfs4_proc_setattr at ffffffffa071bb45 [nfs]
#15 [ffff880139435cb0] nfs_setattr at ffffffffa07043e0 [nfs]
#16 [ffff880139435cf0] notify_change at ffffffff8118e0c8
#17 [ffff880139435d60] do_truncate at ffffffff81170a84
#18 [ffff880139435dd0] do_filp_open at ffffffff81182ba9
#19 [ffff880139435f20] do_sys_open at ffffffff8116f849
#20 [ffff880139435f70] sys_open at ffffffff8116f960
#21 [ffff880139435f80] tracesys at ffffffff8100b387 (via system_call)
#!/bin/bash
i=0
BEGIN_TIME=$(date +%s)
TIME_OUT=14400 #Set the testTime.
SERVER_FILE="/nfsroot/file"
CLIENT_FILE="/nfsmnt/file"
while :
do
    dd if=/dev/zero of="$SERVER_FILE$i" bs=1b count=1 &>/dev/null
    cat "$CLIENT_FILE$i" &>/dev/null
    echo "good" > "$CLIENT_FILE$i"
    rm -fr "$CLIENT_FILE$i"
    now=$(date +%s)
    if (($now - $BEGIN_TIME > $TIME_OUT))
    then
        break
    fi
        let i=i+1
done

echo "This probleam may be fix.\n"

oops when I mount nfs with -o fsc

fanchaoting — 2012-03-29T01:53:27

Hi: in nfs(RHEL6.x), i do as following. 
#cat /etc/exports
    /nfsroot      *(rw,no_root_squash,fsid=0,insecure)
#mount –t nfs4 –o fsc localhost:/   /nfsmnt
#sh create_file.sh

wait a while, i find oops.

PID: 4196   TASK: ffff88012365b580  CPU: 0   COMMAND: "sh"
 #0 [ffff880139435750] machine_kexec at ffffffff810310db
 #1 [ffff8801394357b0] crash_kexec at ffffffff810b63b2
 #2 [ffff880139435880] oops_end at ffffffff814dec50
 #3 [ffff8801394358b0] die at ffffffff8100f2fb
 #4 [ffff8801394358e0] do_trap at ffffffff814de544
 #5 [ffff880139435940] do_invalid_op at ffffffff8100ceb5
 #6 [ffff8801394359e0] invalid_op at ffffffff8100bf5b
    [exception RIP: __nfs_fscache_invalidate_page+121]
 #7 [ffff880139435ab0] nfs_invalidate_page at ffffffffa06fff6e [nfs]
 #8 [ffff880139435ad0] do_invalidatepage at ffffffff811243d5
 #9 [ffff880139435ae0] truncate_inode_page at ffffffff811245f2
#10 [ffff880139435b00] truncate_inode_pages_range at ffffffff811248f0
#11 [ffff880139435bf0] truncate_inode_pages at ffffffff81124c05
#12 [ffff880139435c00] truncate_pagecache at ffffffff81124c57
#13 [ffff880139435c30] nfs_setattr_update_inode at ffffffffa07033ab [nfs]
#14 [ffff880139435c70] nfs4_proc_setattr at ffffffffa071bb45 [nfs]
#15 [ffff880139435cb0] nfs_setattr at ffffffffa07043e0 [nfs]
#16 [ffff880139435cf0] notify_change at ffffffff8118e0c8
#17 [ffff880139435d60] do_truncate at ffffffff81170a84
#18 [ffff880139435dd0] do_filp_open at ffffffff81182ba9
#19 [ffff880139435f20] do_sys_open at ffffffff8116f849
#20 [ffff880139435f70] sys_open at ffffffff8116f960
#21 [ffff880139435f80] tracesys at ffffffff8100b387 (via system_call)
#!/bin/bash
i=0
BEGIN_TIME=$(date +%s)
TIME_OUT=14400 #Set the testTime.
SERVER_FILE="/nfsroot/file"
CLIENT_FILE="/nfsmnt/file"
while :
do
    dd if=/dev/zero of="$SERVER_FILE$i" bs=1b count=1 &>/dev/null
    cat "$CLIENT_FILE$i" &>/dev/null
    echo "good" > "$CLIENT_FILE$i"
    rm -fr "$CLIENT_FILE$i"
    now=$(date +%s)
    if (($now - $BEGIN_TIME > $TIME_OUT))
    then
        break
    fi
        let i=i+1
done

echo "This probleam may be fix.\n"

Re: oops when I mount nfs with -o fsc

fanchaoting — 2012-03-29T01:50:46

fanchaoting 写道:



--
Linux-cachefs mailing list
Linux-cachefs< at >redhat.com
https://www.redhat.com/mailman/listinfo/linux-cachefs

Rebélate by self-management, first project of free software by which we bet all / Rebélate por la autogestión, primer proyecto de software libre por el que apostamos todas

Orquidea Salt mas — 2012-03-02T19:22:05

Inglés :

Many already we have contributed to the first project of free software
dedicated to self-management in this campaign of collective financing,
it collaborates and it spreads!/


Beginning campaign collective financing

http://www.goteo.org/project/rebelaos-publicacion-por-la-autogestion?lang=en


Login to enter with user of social networks and for would register in Goteo :

http://www.goteo.org/user/login?lang=en


Rebelaos! Publication by self-management A massive publication that
floods the public transport, the work centers, the parks, the
consumption centers, by means of distribution of 500,000 gratuitous
units, acting simultaneously in all sides and nowhere.

We announce the main tool of a vestibule Web for the management of
self-sustaining resources by means of Drupal, in addition in the
publication there will be an article dedicated to free software,
hardware, It is being prepared in inglès,  the machinery You can see
more details in the index of the
publication    https://n-1.cc/pg/file/read/1151902/indexresumen-de-los-contenidos-pdf

 . A computer system that allows us to share resources in all the
scopes of our life so that we do not have to generate means different
for each subject nor for each territory.

A point of contact digitalis to generate projects of life outside
Capitalism and to margin of the State.


A tool to spread and to impel the social transformation through the
resources that will set out in their contents around self-management,
the autoorganización, the disobedience and the collective action.

In which the capitalist system goes to the collapse, in a while
immersed in a deep systemic crisis (ecological, political and
economic, but mainly of values), where individual and collective of
people they are being lacking of his fundamental rights, is necessary
to develop a horizontal collective process where all the human beings
we pruned to interact in equality of conditions and freedom.


To interact means to relate to us (as much human as economically), to
communicate to us, to cover our basic needs, to generate and to
protect communal properties, to know and to provide collective
solutions us problematic that our lives interfere. We want abrir a
breach within normality in the monotonous life state-capitalist, a day
anyone, that finally will not be any day.


By means of this publication we try:

- To drive a horizontal collective process where all and all we pruned
to interact in equality of conditions and freedom.

 - To create communications network between the people it jeopardize
with the change and arranged to act.


 - To find collective solutions to problematic that our lives interfere


- To facilitate the access to resources that make possible self-management.

- To participate in the construction of networks of mutual support,
generated horizontals, asamblearias and from the base.


 - To publish all this information in an attractive format stops to
facilitate the access to all the society.



There are 15 days remaining for the upcoming March 15, the day that
will come Rebelaos!, Magazine for the selfmanagement

Today, we issue the cover of Rebelaos! (Castilian version) that can be
displayed on the following link:
https://n-1.cc/pg/file/read/1200503/portada-15-de-marzo-rebelaos
The contents of the store owners to us by 15 March. Do you? Do you
keep on 15 March?

In addition, we have over 200 distribution nodes, distributed
throughout the Spanish state. Check the map:
https://afinidadrebelde.crowdmap.com/

On the other hand, the funding campaign continues to move and still
have 12 days to collect the remaining 6,000 euros. We can all make a
bit for all the grains of sand become a great beach on March 15. You
can access the co-financing campaign:
http://www.goteo.org/project/rebelaos-publicacion-por-la-autogestion

Rebel Affinity group
www.rebelaos.net


-------------------------------------------------------------------------------
Castellano:

Muchos ya hemos aportado al primer proyecto de software libre dedicado
a la la financiación colectiva, colabora y diffunde !!!!!

Inicio campaña financiación colectiva goteo.org

www.goteo.org/project/rebelaos-publicacion-por-la-autogestion


Link para registrarse en Goteo y acceder a redes sociales para
colaborar en la difusín

http://www.goteo.org/user/login

¡Rebelaos! Publicación por la autogestión

Una publicación masiva que inunde el transporte público, los centros
de trabajo, los parques, los centros de consumo, mediante la
distribución de 500.000 ejemplares gratuitos, actuando simultáneamente
en todos lados y en ninguna parte.


Anunciamos la herramienta principal de un  portal web para la gestión
de recursos autogestionados mediante Drupal, además en  la publicación
habrá un artículo dedicado al software libre, el hardware, la
maquinaria... Puedes ver más detalles en el índice de la publicación
https://n-1.cc/pg/file/read/1151902/indexresumen-de-los-contenidos-pdf

Un sistema infórmatico que nos permita compartir recursos en todos los
ámbitos de nuestra vida de forma que no tengamos que generar un medio
distinto para cada tema ni para cada territorio. Un punto de encuentro
digital para generar proyectos de vida fuera del capitalismo y al
margen del Estado.


Una herramienta para difundir e impulsar la transformación social a
través de los recursos que se propondrán en sus contenidos en torno a
la autogestión, la autoorganización, la desobediencia y la acción
colectiva.

En un momento en que el sistema capitalista se dirige al colapso,
inmerso en una profunda crisis sistémica (ecológica, política y
económica, pero principalmente de valores), donde individuos y
colectivos de personas están siendo desprovistos de sus derechos
fundamentales, es necesario desarrollar un proceso colectivo
horizontal donde todos los seres humanos podamos interactuar en
igualdad de condiciones y en libertad.

Interactuar significa relacionarnos (tanto humana como
económicamente), comunicarnos, cubrir nuestras necesidades básicas,
generar y proteger bienes comunes, conocernos y dar soluciones
colectivas a problemáticas que interfieren nuestras vidas. Queremos
abrir una brecha dentro de la normalidad en la monótona vida
estatal-capitalista, un día cualquiera, que finalmente no será
cualquier día.

Mediante esta publicación pretendemos:

- Impulsar un proceso colectivo horizontal donde todos y todas podamos
interactuar en igualdad de condiciones y en libertad.

- Crear red de comunicaciones entre las personas comprometidas con el
cambio y dispuestas a actuar.

- Encontrar soluciones colectivas a problemáticas que interfieren
nuestras vidas.

- Facilitar el acceso a recursos que posibiliten la autogestión.

- Participar en la construcción de redes de apoyo mutuo, horizontales,
asamblearias y generadas desde la base.

- Publicar toda esta información en un formato atractivo para
facilitar el acceso a toda la sociedad.

Son 15 los días que restan para el próximo 15 de marzo, día en el que
verá la luz ¡Rebelaos!, publicación por la autogestión.

Hoy, hacemos pública la portada de ¡Rebelaos! (versión en castellano)
que podéis visualizar en el siguiente enlace:
https://n-1.cc/pg/file/read/1200503/portada-15-de-marzo-rebelaos
El contenido de los titulares nos los guardamos para el 15 de marzo.
¿Y tú? ¿Te guardas el 15 de marzo?

Además, ya hemos superado los 200 nodos de distribución, repartidos
por todo el estado español. Ver el mapa:
https://afinidadrebelde.crowdmap.com/

Por otro lado, la campaña de financiación continúa avanzando y todavía
quedan 12 días para reunir los 6.000 euros que restan. Todas podemos
aportar un poco para que todos los granitos de arena se conviertan en
una gran playa el 15 de marzo. Podéis acceder a la campaña de
cofinanciación en:
http://www.goteo.org/project/rebelaos-publicacion-por-la-autogestion

Colectivo Afinidad Rebelde
www.rebelaos.net

Re: FS-Cache "Netfs" (nfs) Fedora 15/16/17

David Howells — 2012-02-21T18:14:26


Unless you're mounting an NFS fs with -o fsc immediately thereafter it should
do very little at this point, and even so, it shouldn't do much more unless
you've also loaded the cachefiles module and started up a cache.

Try:

echo 65535 >/sys/module/fscache/parameters/debug

David

FS-Cache "Netfs" (nfs) Fedora 15/16/17

Frank Murphy — 2012-02-21T17:41:39

Hi,

Am trying to figure out why during bootup,
at the following point across real and virt boxes.

eb 19 11:22:07 frank01 kernel: [  482.640618] FS-Cache: Loaded
Feb 19 11:22:07 frank01 kernel: [  482.672487] FS-Cache: Netfs 'nfs' 
registered for caching

It sits there for about 10 seconds, with no indication as to what's 
happening.


nfs mount during bootup can be autofs or fstab depending on box.


Is this still relevant for debugging FS-Cache:
https://www.redhat.com/archives/linux-cachefs/2010-February/msg00022.html

" You can also turn on NFS debugging for FS-Cache to see what it's doing:

echo $((0x800)) >/proc/sys/sunrpc/nfs_debug  "


kernels across boxes: 3.2* 3.3*

[PATCH 17/17] NFS: nfs_migrate_page() does not wait for FS-Cache to finish with a page

David Howells — 2012-02-08T21:19:35

nfs_migrate_page() does not wait for FS-Cache to finish with a page, probably
leading to the following bad-page-state:

 BUG: Bad page state in process python-bin  pfn:17d39b
 page:ffffea00053649e8 flags:004000000000100c count:0 mapcount:0 mapping:(null)
index:38686 (Tainted: G    B      ---------------- )
 Pid: 31053, comm: python-bin Tainted: G    B      ----------------
2.6.32-71.24.1.el6.x86_64 #1
 Call Trace:
 [<ffffffff8111bfe7>] bad_page+0x107/0x160
 [<ffffffff8111ee69>] free_hot_cold_page+0x1c9/0x220
 [<ffffffff8111ef19>] __pagevec_free+0x59/0xb0
 [<ffffffff8104b988>] ? flush_tlb_others_ipi+0x128/0x130
 [<ffffffff8112230c>] release_pages+0x21c/0x250
 [<ffffffff8115b92a>] ? remove_migration_pte+0x28a/0x2b0
 [<ffffffff8115f3f8>] ? mem_cgroup_get_reclaim_stat_from_page+0x18/0x70
 [<ffffffff81122687>] ____pagevec_lru_add+0x167/0x180
 [<ffffffff811226f8>] __lru_cache_add+0x58/0x70
 [<ffffffff81122731>] lru_cache_add_lru+0x21/0x40
 [<ffffffff81123f49>] putback_lru_page+0x69/0x100
 [<ffffffff8115c0bd>] migrate_pages+0x13d/0x5d0
 [<ffffffff81122687>] ? ____pagevec_lru_add+0x167/0x180
 [<ffffffff81152ab0>] ? compaction_alloc+0x0/0x370
 [<ffffffff8115255c>] compact_zone+0x4cc/0x600
 [<ffffffff8111cfac>] ? get_page_from_freelist+0x15c/0x820
 [<ffffffff810672f4>] ? check_preempt_wakeup+0x1c4/0x3c0
 [<ffffffff8115290e>] compact_zone_order+0x7e/0xb0
 [<ffffffff81152a49>] try_to_compact_pages+0x109/0x170
 [<ffffffff8111e94d>] __alloc_pages_nodemask+0x5ed/0x850
 [<ffffffff814c9136>] ? thread_return+0x4e/0x778
 [<ffffffff81150d43>] alloc_pages_vma+0x93/0x150
 [<ffffffff81167ea5>] do_huge_pmd_anonymous_page+0x135/0x340
 [<ffffffff814cb6f6>] ? rwsem_down_read_failed+0x26/0x30
 [<ffffffff81136755>] handle_mm_fault+0x245/0x2b0
 [<ffffffff814ce383>] do_page_fault+0x123/0x3a0
 [<ffffffff814cbdf5>] page_fault+0x25/0x30

nfs_migrate_page() calls nfs_fscache_release_page() which doesn't actually wait
- even if __GFP_WAIT is set.  The reason that doesn't wait is that
fscache_maybe_release_page() might deadlock the allocator as the work threads
writing to the cache may all end up sleeping on memory allocation.

However, I wonder if that is actually a problem.  There are a number of things
I can do to deal with this:

 (1) Make nfs_migrate_page() wait.

 (2) Make fscache_maybe_release_page() honour the __GFP_WAIT flag.

 (3) Set a timeout around the wait.

 (4) Make nfs_migrate_page() return an error if the page is still busy.

For the moment, I'll select (2) and (4).

Signed-off-by: David Howells <dhowells< at >redhat.com>
Acked-by: Jeff Layton <jlayton< at >redhat.com>
---

 fs/fscache/internal.h |    1 +
 fs/fscache/page.c     |   19 ++++++++++++++-----
 fs/fscache/stats.c    |    6 ++++--
 fs/nfs/write.c        |    4 +++-
 4 files changed, 22 insertions(+), 8 deletions(-)

diff --git a/fs/fscache/internal.h b/fs/fscache/internal.h
index dcb3e1d..88a48cc 100644
--- a/fs/fscache/internal.h
+++ b/fs/fscache/internal.h
< at >< at > -200,6 +200,7 < at >< at > extern atomic_t fscache_n_store_vmscan_not_storing;
 extern atomic_t fscache_n_store_vmscan_gone;
 extern atomic_t fscache_n_store_vmscan_busy;
 extern atomic_t fscache_n_store_vmscan_cancelled;
+extern atomic_t fscache_n_store_vmscan_wait;
 
 extern atomic_t fscache_n_marks;
 extern atomic_t fscache_n_uncaches;
diff --git a/fs/fscache/page.c b/fs/fscache/page.c
index 4dbbca1..f9b2fb3 100644
--- a/fs/fscache/page.c
+++ b/fs/fscache/page.c
< at >< at > -56,6 +56,7 < at >< at > bool __fscache_maybe_release_page(struct fscache_cookie *cookie,
 
 _enter("%p,%p,%x", cookie, page, gfp);
 
+try_again:
 rcu_read_lock();
 val = radix_tree_lookup(&cookie->stores, page->index);
 if (!val) {
< at >< at > -104,11 +105,19 < at >< at > bool __fscache_maybe_release_page(struct fscache_cookie *cookie,
 return true;
 
 page_busy:
-/* we might want to wait here, but that could deadlock the allocator as
- * the work threads writing to the cache may all end up sleeping
- * on memory allocation */
-fscache_stat(&fscache_n_store_vmscan_busy);
-return false;
+/* We will wait here if we're allowed to, but that could deadlock the
+ * allocator as the work threads writing to the cache may all end up
+ * sleeping on memory allocation, so we may need to impose a timeout
+ * too. */
+if (!(gfp & __GFP_WAIT)) {
+fscache_stat(&fscache_n_store_vmscan_busy);
+return false;
+}
+
+fscache_stat(&fscache_n_store_vmscan_wait);
+__fscache_wait_on_page_write(cookie, page);
+gfp &= ~__GFP_WAIT;
+goto try_again;
 }
 EXPORT_SYMBOL(__fscache_maybe_release_page);
 
diff --git a/fs/fscache/stats.c b/fs/fscache/stats.c
index 51cdaee..8179e8b 100644
--- a/fs/fscache/stats.c
+++ b/fs/fscache/stats.c
< at >< at > -69,6 +69,7 < at >< at > atomic_t fscache_n_store_vmscan_not_storing;
 atomic_t fscache_n_store_vmscan_gone;
 atomic_t fscache_n_store_vmscan_busy;
 atomic_t fscache_n_store_vmscan_cancelled;
+atomic_t fscache_n_store_vmscan_wait;
 
 atomic_t fscache_n_marks;
 atomic_t fscache_n_uncaches;
< at >< at > -232,11 +233,12 < at >< at > static int fscache_stats_show(struct seq_file *m, void *v)
    atomic_read(&fscache_n_store_radix_deletes),
    atomic_read(&fscache_n_store_pages_over_limit));
 
-seq_printf(m, "VmScan : nos=%u gon=%u bsy=%u can=%u\n",
+seq_printf(m, "VmScan : nos=%u gon=%u bsy=%u can=%u wt=%u\n",
    atomic_read(&fscache_n_store_vmscan_not_storing),
    atomic_read(&fscache_n_store_vmscan_gone),
    atomic_read(&fscache_n_store_vmscan_busy),
-   atomic_read(&fscache_n_store_vmscan_cancelled));
+   atomic_read(&fscache_n_store_vmscan_cancelled),
+   atomic_read(&fscache_n_store_vmscan_wait));
 
 seq_printf(m, "Ops    : pend=%u run=%u enq=%u can=%u rej=%u\n",
    atomic_read(&fscache_n_op_pend),
diff --git a/fs/nfs/write.c b/fs/nfs/write.c
index 834f0fe..5fbee48 100644
--- a/fs/nfs/write.c
+++ b/fs/nfs/write.c
< at >< at > -1701,7 +1701,9 < at >< at > int nfs_migrate_page(struct address_space *mapping, struct page *newpage,
 if (PagePrivate(page))
 return -EBUSY;
 
-nfs_fscache_release_page(page, GFP_KERNEL);
+ret = -EBUSY;
+if (!nfs_fscache_release_page(page, GFP_KERNEL))
+goto out;
 
 return migrate_page(mapping, newpage, page, mode);
 }

[PATCH 16/17] FS-Cache: Exclusive op submission can BUG if there's been an I/O error

David Howells — 2012-02-08T21:19:24

The function to submit an exclusive op (fscache_submit_exclusive_op()) can BUG
if there's been an I/O error because it may see the parent cache object in an
unexpected state.  It should only BUG if there hasn't been an I/O error.

In this case the problem was produced by remounting the cache partition to be
R/O.  The EROFS state was detected and the cache was aborted, but not
everything handled the aborting correctly.

SysRq : Emergency Remount R/O
EXT4-fs (sda6): re-mounted. Opts: (null)
Emergency Remount complete
CacheFiles: I/O Error: Failed to update xattr with error -30
FS-Cache: Cache cachefiles stopped due to I/O error
------------[ cut here ]------------
kernel BUG at fs/fscache/operation.c:128!
invalid opcode: 0000 [#1] SMP 
CPU 0 
Modules linked in: cachefiles nfs fscache auth_rpcgss nfs_acl lockd sunrpc

Pid: 6612, comm: kworker/u:2 Not tainted 3.1.0-rc8-fsdevel+ #1093                  /DG965RY
RIP: 0010:[<ffffffffa00739c0>]  [<ffffffffa00739c0>] fscache_submit_exclusive_op+0x2ad/0x2c2 [fscache]
RSP: 0018:ffff880000853d40  EFLAGS: 00010206
RAX: ffff880038ac72a8 RBX: ffff8800181f2260 RCX: ffffffff81f2b2b0
RDX: 0000000000000001 RSI: ffffffff8179a478 RDI: ffff8800181f2280
RBP: ffff880000853d60 R08: 0000000000000002 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000001 R12: ffff880038ac7268
R13: ffff8800181f2280 R14: ffff88003a359190 R15: 000000010122b162
FS:  0000000000000000(0000) GS:ffff88003bc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000034cc4a77f0 CR3: 0000000010e96000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kworker/u:2 (pid: 6612, threadinfo ffff880000852000, task ffff880014c3c040)
Stack:
 ffff8800181f2260 ffff8800181f2310 ffff880038ac7268 ffff8800181f2260
 ffff880000853dc0 ffffffffa0072375 ffff880037ecfe00 ffff88003a359198
 ffff880000853dc0 0000000000000246 0000000000000000 ffff88000a91d308
Call Trace:
 [<ffffffffa0072375>] fscache_object_work_func+0x792/0xe65 [fscache]
 [<ffffffff81047e44>] process_one_work+0x1eb/0x37f
 [<ffffffff81047de6>] ? process_one_work+0x18d/0x37f
 [<ffffffffa0071be3>] ? fscache_enqueue_dependents+0xd8/0xd8 [fscache]
 [<ffffffff810482e4>] worker_thread+0x15a/0x21a
 [<ffffffff8104818a>] ? rescuer_thread+0x188/0x188
 [<ffffffff8104bf96>] kthread+0x7f/0x87
 [<ffffffff813ad6f4>] kernel_thread_helper+0x4/0x10
 [<ffffffff81026b98>] ? finish_task_switch+0x45/0xc0
 [<ffffffff813abd1d>] ? retint_restore_args+0xe/0xe
 [<ffffffff8104bf17>] ? __init_kthread_worker+0x53/0x53
 [<ffffffff813ad6f0>] ? gs_change+0xb/0xb


Signed-off-by: David Howells <dhowells< at >redhat.com>
---

 fs/fscache/internal.h  |    1 +
 fs/fscache/object.c    |   23 +++++++++++++++++------
 fs/fscache/operation.c |   13 ++++++++++---
 3 files changed, 28 insertions(+), 9 deletions(-)

diff --git a/fs/fscache/internal.h b/fs/fscache/internal.h
index c811793..dcb3e1d 100644
--- a/fs/fscache/internal.h
+++ b/fs/fscache/internal.h
< at >< at > -288,6 +288,7 < at >< at > extern const struct file_operations fscache_stats_fops;
 static inline void fscache_raise_event(struct fscache_object *object,
        unsigned event)
 {
+BUG_ON(event >= NR_FSCACHE_OBJECT_EVENTS);
 if (!test_and_set_bit(event, &object->events) &&
     test_bit(event, &object->event_mask))
 fscache_enqueue_object(object);
diff --git a/fs/fscache/object.c b/fs/fscache/object.c
index 2ef8a08..2c512cb 100644
--- a/fs/fscache/object.c
+++ b/fs/fscache/object.c
< at >< at > -103,6 +103,7 < at >< at > static void fscache_object_state_machine(struct fscache_object *object)
 {
 enum fscache_object_state new_state;
 struct fscache_cookie *cookie;
+int event;
 
 ASSERT(object != NULL);
 
< at >< at > -275,7 +276,8 < at >< at > static void fscache_object_state_machine(struct fscache_object *object)
 
 /* determine the transition from a lookup state */
 lookup_transit:
-switch (fls(object->events & object->event_mask) - 1) {
+event = fls(object->events & object->event_mask) - 1;
+switch (event) {
 case FSCACHE_OBJECT_EV_WITHDRAW:
 case FSCACHE_OBJECT_EV_RETIRE:
 case FSCACHE_OBJECT_EV_RELEASE:
< at >< at > -292,7 +294,8 < at >< at > lookup_transit:
 
 /* determine the transition from an active state */
 active_transit:
-switch (fls(object->events & object->event_mask) - 1) {
+event = fls(object->events & object->event_mask) - 1;
+switch (event) {
 case FSCACHE_OBJECT_EV_WITHDRAW:
 case FSCACHE_OBJECT_EV_RETIRE:
 case FSCACHE_OBJECT_EV_RELEASE:
< at >< at > -314,7 +317,8 < at >< at > active_transit:
 
 /* determine the transition from a terminal state */
 terminal_transit:
-switch (fls(object->events & object->event_mask) - 1) {
+event = fls(object->events & object->event_mask) - 1;
+switch (event) {
 case FSCACHE_OBJECT_EV_WITHDRAW:
 new_state = FSCACHE_OBJECT_WITHDRAWING;
 goto change_state;
< at >< at > -347,8 +351,8 < at >< at > done:
 
 unsupported_event:
 printk(KERN_ERR "FS-Cache:"
-       " Unsupported event %lx [mask %lx] in state %s\n",
-       object->events, object->event_mask,
+       " Unsupported event %d [%lx/%lx] in state %s\n",
+       event, object->events, object->event_mask,
        fscache_object_states[object->state]);
 BUG();
 }
< at >< at > -945,7 +949,7 < at >< at > static void fscache_invalidate_object(struct fscache_object *object)
 
 spin_lock(&cookie->lock);
 if (fscache_submit_exclusive_op(object, op) < 0)
-BUG();
+goto submit_op_failed;
 spin_unlock(&cookie->lock);
 fscache_put_operation(op);
 
< at >< at > -960,4 +964,11 < at >< at > static void fscache_invalidate_object(struct fscache_object *object)
  */
 fscache_invalidation_complete(cookie);
 _leave("");
+return;
+
+submit_op_failed:
+spin_unlock(&cookie->lock);
+kfree(op);
+fscache_raise_event(object, FSCACHE_OBJECT_EV_ERROR);
+_leave(" [EIO]");
 }
diff --git a/fs/fscache/operation.c b/fs/fscache/operation.c
index c58dbe6..9e6b7d2 100644
--- a/fs/fscache/operation.c
+++ b/fs/fscache/operation.c
< at >< at > -84,6 +84,8 < at >< at > static void fscache_run_op(struct fscache_object *object,
 int fscache_submit_exclusive_op(struct fscache_object *object,
 struct fscache_operation *op)
 {
+int ret;
+
 _enter("{OBJ%x OP%x},", object->debug_id, op->debug_id);
 
 ASSERTCMP(op->state, ==, FSCACHE_OP_ST_INITIALISED);
< at >< at > -116,6 +118,7 < at >< at > int fscache_submit_exclusive_op(struct fscache_object *object,
 
 /* need to issue a new write op after this */
 clear_bit(FSCACHE_OBJECT_PENDING_WRITE, &object->flags);
+ret = 0;
 } else if (object->state == FSCACHE_OBJECT_CREATING) {
 op->object = object;
 object->n_ops++;
< at >< at > -123,13 +126,17 < at >< at > int fscache_submit_exclusive_op(struct fscache_object *object,
 atomic_inc(&op->usage);
 list_add_tail(&op->pend_link, &object->pending_ops);
 fscache_stat(&fscache_n_op_pend);
+ret = 0;
 } else {
-/* not allowed to submit ops in any other state */
-BUG();
+/* If we're in any other state, there must have been an I/O
+ * error of some nature.
+ */
+ASSERT(test_bit(FSCACHE_IOERROR, &object->cache->flags));
+ret = -EIO;
 }
 
 spin_unlock(&object->lock);
-return 0;
+return ret;
 }
 
 /*

[PATCH 15/17] FS-Cache: Limit the number of I/O error reports for a cache

David Howells — 2012-02-08T21:19:15

Limit the number of I/O error reports for a cache to 1 to prevent massive
amounts of noise.  After the first I/O error the cache is taken off line
automatically, so must be restarted to resume caching.

Signed-off-by: David Howells <dhowells< at >redhat.com>
---

 fs/fscache/cache.c |    8 ++++----
 1 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/fs/fscache/cache.c b/fs/fscache/cache.c
index 6a3c48a..b52aed1 100644
--- a/fs/fscache/cache.c
+++ b/fs/fscache/cache.c
< at >< at > -314,10 +314,10 < at >< at > EXPORT_SYMBOL(fscache_add_cache);
  */
 void fscache_io_error(struct fscache_cache *cache)
 {
-set_bit(FSCACHE_IOERROR, &cache->flags);
-
-printk(KERN_ERR "FS-Cache: Cache %s stopped due to I/O error\n",
-       cache->ops->name);
+if (!test_and_set_bit(FSCACHE_IOERROR, &cache->flags))
+printk(KERN_ERR "FS-Cache:"
+       " Cache '%s' stopped due to I/O error\n",
+       cache->ops->name);
 }
 EXPORT_SYMBOL(fscache_io_error);

[PATCH 14/17] FS-Cache: Don't mask off the object event mask when printing it

David Howells — 2012-02-08T21:19:03

Don't mask off the object event mask when printing it.  That way it can be seen
if threre are bits set that shouldn't be.

Signed-off-by: David Howells <dhowells< at >redhat.com>
---

 fs/cachefiles/namei.c    |    3 +--
 fs/fscache/object-list.c |    2 +-
 2 files changed, 2 insertions(+), 3 deletions(-)

diff --git a/fs/cachefiles/namei.c b/fs/cachefiles/namei.c
index a0358c2..0fd9b6d 100644
--- a/fs/cachefiles/namei.c
+++ b/fs/cachefiles/namei.c
< at >< at > -40,8 +40,7 < at >< at > void __cachefiles_printk_object(struct cachefiles_object *object,
 printk(KERN_ERR "%sobjstate=%s fl=%lx wbusy=%x ev=%lx[%lx]\n",
        prefix, fscache_object_states[object->fscache.state],
        object->fscache.flags, work_busy(&object->fscache.work),
-       object->fscache.events,
-       object->fscache.event_mask & FSCACHE_OBJECT_EVENTS_MASK);
+       object->fscache.events, object->fscache.event_mask);
 printk(KERN_ERR "%sops=%u inp=%u exc=%u\n",
        prefix, object->fscache.n_ops, object->fscache.n_in_progress,
        object->fscache.n_exclusive);
diff --git a/fs/fscache/object-list.c b/fs/fscache/object-list.c
index ebe29c5..f27c89d 100644
--- a/fs/fscache/object-list.c
+++ b/fs/fscache/object-list.c
< at >< at > -245,7 +245,7 < at >< at > static int fscache_objlist_show(struct seq_file *m, void *v)
    obj->n_in_progress,
    obj->n_exclusive,
    atomic_read(&obj->n_reads),
-   obj->event_mask & FSCACHE_OBJECT_EVENTS_MASK,
+   obj->event_mask,
    obj->events,
    obj->flags,
    work_busy(&obj->work));

[PATCH 13/17] FS-Cache: Initialise the object event mask with the calculated mask

David Howells — 2012-02-08T21:18:53

Initialise the object event mask with the calculated mask rather than unmasking
undefined events also.

Signed-off-by: David Howells <dhowells< at >redhat.com>
---

 fs/fscache/object.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/fs/fscache/object.c b/fs/fscache/object.c
index 80b5491..2ef8a08 100644
--- a/fs/fscache/object.c
+++ b/fs/fscache/object.c
< at >< at > -114,7 +114,8 < at >< at > static void fscache_object_state_machine(struct fscache_object *object)
 /* wait for the parent object to become ready */
 case FSCACHE_OBJECT_INIT:
 object->event_mask =
-ULONG_MAX & ~(1 << FSCACHE_OBJECT_EV_CLEARED);
+FSCACHE_OBJECT_EVENTS_MASK &
+~(1 << FSCACHE_OBJECT_EV_CLEARED);
 fscache_initialise_object(object);
 goto done;

[PATCH 12/17] FS-Cache: Convert the object event ID #defines into an enum

David Howells — 2012-02-08T21:18:44

Convert the fscache_object event IDs from #defines into an enum.  Also add an
extra label to the enum to carry the event count and redefine the event mask
in terms of that.

Signed-off-by: David Howells <dhowells< at >redhat.com>
---

 include/linux/fscache-cache.h |   26 +++++++++++++++++---------
 1 files changed, 17 insertions(+), 9 deletions(-)

diff --git a/include/linux/fscache-cache.h b/include/linux/fscache-cache.h
index 1e454ad..73e68c8 100644
--- a/include/linux/fscache-cache.h
+++ b/include/linux/fscache-cache.h
< at >< at > -338,6 +338,23 < at >< at > struct fscache_cookie {
 extern struct fscache_cookie fscache_fsdef_index;
 
 /*
+ * Event list for fscache_object::{event_mask,events}
+ */
+enum {
+FSCACHE_OBJECT_EV_REQUEUE,/* T if object should be requeued */
+FSCACHE_OBJECT_EV_UPDATE,/* T if object should be updated */
+FSCACHE_OBJECT_EV_INVALIDATE,/* T if cache requested object invalidation */
+FSCACHE_OBJECT_EV_CLEARED,/* T if accessors all gone */
+FSCACHE_OBJECT_EV_ERROR,/* T if fatal error occurred during processing */
+FSCACHE_OBJECT_EV_RELEASE,/* T if netfs requested object release */
+FSCACHE_OBJECT_EV_RETIRE,/* T if netfs requested object retirement */
+FSCACHE_OBJECT_EV_WITHDRAW,/* T if cache requested object withdrawal */
+NR_FSCACHE_OBJECT_EVENTS
+};
+
+#define FSCACHE_OBJECT_EVENTS_MASK ((1UL << NR_FSCACHE_OBJECT_EVENTS) - 1)
+
+/*
  * on-disk cache file or index handle
  */
 struct fscache_object {
< at >< at > -376,15 +393,6 < at >< at > struct fscache_object {
 unsigned longevent_mask;/* events this object is interested in */
 unsigned longevents;/* events to be processed by this object
  * (order is important - using fls) */
-#define FSCACHE_OBJECT_EV_REQUEUE0/* T if object should be requeued */
-#define FSCACHE_OBJECT_EV_UPDATE1/* T if object should be updated */
-#define FSCACHE_OBJECT_EV_CLEARED2/* T if accessors all gone */
-#define FSCACHE_OBJECT_EV_ERROR3/* T if fatal error occurred during processing */
-#define FSCACHE_OBJECT_EV_RELEASE4/* T if netfs requested object release */
-#define FSCACHE_OBJECT_EV_RETIRE5/* T if netfs requested object retirement */
-#define FSCACHE_OBJECT_EV_WITHDRAW6/* T if cache requested object withdrawal */
-#define FSCACHE_OBJECT_EV_INVALIDATE7/* T if cache requested object invalidation */
-#define FSCACHE_OBJECT_EVENTS_MASK0xff/* mask of all events*/
 
 unsigned longflags;
 #define FSCACHE_OBJECT_LOCK0/* T if object is busy being processed */

[PATCH 10/17] NFS: Use FS-Cache invalidation

David Howells — 2012-02-08T21:18:23

Use the new FS-Cache invalidation facility from NFS to deal with foreign
changes being detected on the server rather than attempting to retire the old
cookie and get a new one.

The problem with the old method was that NFS did not wait for all outstanding
storage and retrieval ops on the cache to complete.  There was no automatic
wait between the calls to ->readpages() and calls to invalidate_inode_pages2()
as the latter can only wait on locked pages that have been added to the
pagecache (which they haven't yet on entry to ->readpages()).

This was leading to oopses like the one below when an outstanding read got cut
off from its cookie by a premature release.

BUG: unable to handle kernel NULL pointer dereference at 00000000000000a8
IP: [<ffffffffa0075118>] __fscache_read_or_alloc_pages+0x1dd/0x315 [fscache]
PGD 15889067 PUD 15890067 PMD 0
Oops: 0000 [#1] SMP
CPU 0
Modules linked in: cachefiles nfs fscache auth_rpcgss nfs_acl lockd sunrpc

Pid: 4544, comm: tar Not tainted 3.1.0-rc4-fsdevel+ #1064                  /DG965RY
RIP: 0010:[<ffffffffa0075118>]  [<ffffffffa0075118>] __fscache_read_or_alloc_pages+0x1dd/0x315 [fscache]
RSP: 0018:ffff8800158799e8  EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff8800070d41e0 RCX: ffff8800083dc1b0
RDX: 0000000000000000 RSI: ffff880015879960 RDI: ffff88003e627b90
RBP: ffff880015879a28 R08: 0000000000000002 R09: 0000000000000002
R10: 0000000000000001 R11: ffff880015879950 R12: ffff880015879aa4
R13: 0000000000000000 R14: ffff8800083dc158 R15: ffff880015879be8
FS:  00007f671e9d87c0(0000) GS:ffff88003bc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000000000000a8 CR3: 000000001587f000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process tar (pid: 4544, threadinfo ffff880015878000, task ffff880015875040)
Stack:
 ffffffffa00b1759 ffff8800070dc158 ffff8800000213da ffff88002a286508
 ffff880015879aa4 ffff880015879be8 0000000000000001 ffff88002a2866e8
 ffff880015879a88 ffffffffa00b20be 00000000000200da ffff880015875040
Call Trace:
 [<ffffffffa00b1759>] ? nfs_fscache_wait_bit+0xd/0xd [nfs]
 [<ffffffffa00b20be>] __nfs_readpages_from_fscache+0x7e/0x13f [nfs]
 [<ffffffff81095fe7>] ? __alloc_pages_nodemask+0x156/0x662
 [<ffffffffa0098763>] nfs_readpages+0xee/0x187 [nfs]
 [<ffffffff81098a5e>] __do_page_cache_readahead+0x1be/0x267
 [<ffffffff81098942>] ? __do_page_cache_readahead+0xa2/0x267
 [<ffffffff81098d7b>] ra_submit+0x1c/0x20
 [<ffffffff8109900a>] ondemand_readahead+0x28b/0x29a
 [<ffffffff810990ce>] page_cache_sync_readahead+0x38/0x3a
 [<ffffffff81091d8a>] generic_file_aio_read+0x2ab/0x67e
 [<ffffffffa008cfbe>] nfs_file_read+0xa4/0xc9 [nfs]
 [<ffffffff810c22c4>] do_sync_read+0xba/0xfa
 [<ffffffff810a62c9>] ? might_fault+0x4e/0x9e
 [<ffffffff81177a47>] ? security_file_permission+0x7b/0x84
 [<ffffffff810c25dd>] ? rw_verify_area+0xab/0xc8
 [<ffffffff810c29a4>] vfs_read+0xaa/0x13a
 [<ffffffff810c2a79>] sys_read+0x45/0x6c
 [<ffffffff813ac37b>] system_call_fastpath+0x16/0x1b

Reported-by: Mark Moseley <moseleymark< at >gmail.com>
Signed-off-by: David Howells <dhowells< at >redhat.com>
---

 fs/nfs/fscache.h  |   20 +++++++++++++++++++-
 fs/nfs/inode.c    |   20 ++++++++++++++++----
 fs/nfs/nfs4proc.c |    2 ++
 3 files changed, 37 insertions(+), 5 deletions(-)

diff --git a/fs/nfs/fscache.h b/fs/nfs/fscache.h
index b9c572d..851fee1 100644
--- a/fs/nfs/fscache.h
+++ b/fs/nfs/fscache.h
< at >< at > -155,6 +155,22 < at >< at > static inline void nfs_readpage_to_fscache(struct inode *inode,
 }
 
 /*
+ * Invalidate the contents of fscache for this inode.  This will not sleep.
+ */
+static inline void nfs_fscache_invalidate(struct inode *inode)
+{
+fscache_invalidate(NFS_I(inode)->fscache);
+}
+
+/*
+ * Wait for an object to finish being invalidated.
+ */
+static inline void nfs_fscache_wait_on_invalidate(struct inode *inode)
+{
+fscache_wait_on_invalidate(NFS_I(inode)->fscache);
+}
+
+/*
  * indicate the client caching state as readable text
  */
 static inline const char *nfs_server_fscache_state(struct nfs_server *server)
< at >< at > -164,7 +180,6 < at >< at > static inline const char *nfs_server_fscache_state(struct nfs_server *server)
 return "no ";
 }
 
-
 #else /* CONFIG_NFS_FSCACHE */
 static inline int nfs_fscache_register(void) { return 0; }
 static inline void nfs_fscache_unregister(void) {}
< at >< at > -213,6 +228,9 < at >< at > static inline int nfs_readpages_from_fscache(struct nfs_open_context *ctx,
 static inline void nfs_readpage_to_fscache(struct inode *inode,
    struct page *page, int sync) {}
 
+
+static inline void nfs_fscache_invalidate(struct inode *inode) {}
+
 static inline const char *nfs_server_fscache_state(struct nfs_server *server)
 {
 return "no ";
diff --git a/fs/nfs/inode.c b/fs/nfs/inode.c
index f649fba..54383e9 100644
--- a/fs/nfs/inode.c
+++ b/fs/nfs/inode.c
< at >< at > -152,10 +152,12 < at >< at > static void nfs_zap_caches_locked(struct inode *inode)
 nfsi->attrtimeo_timestamp = jiffies;
 
 memset(NFS_COOKIEVERF(inode), 0, sizeof(NFS_COOKIEVERF(inode)));
-if (S_ISREG(mode) || S_ISDIR(mode) || S_ISLNK(mode))
+if (S_ISREG(mode) || S_ISDIR(mode) || S_ISLNK(mode)) {
 nfsi->cache_validity |= NFS_INO_INVALID_ATTR|NFS_INO_INVALID_DATA|NFS_INO_INVALID_ACCESS|NFS_INO_INVALID_ACL|NFS_INO_REVAL_PAGECACHE;
-else
+nfs_fscache_invalidate(inode);
+} else {
 nfsi->cache_validity |= NFS_INO_INVALID_ATTR|NFS_INO_INVALID_ACCESS|NFS_INO_INVALID_ACL|NFS_INO_REVAL_PAGECACHE;
+}
 }
 
 void nfs_zap_caches(struct inode *inode)
< at >< at > -170,6 +172,7 < at >< at > void nfs_zap_mapping(struct inode *inode, struct address_space *mapping)
 if (mapping->nrpages != 0) {
 spin_lock(&inode->i_lock);
 NFS_I(inode)->cache_validity |= NFS_INO_INVALID_DATA;
+nfs_fscache_invalidate(inode);
 spin_unlock(&inode->i_lock);
 }
 }
< at >< at > -862,7 +865,7 < at >< at > static int nfs_invalidate_mapping(struct inode *inode, struct address_space *map
 memset(nfsi->cookieverf, 0, sizeof(nfsi->cookieverf));
 spin_unlock(&inode->i_lock);
 nfs_inc_stats(inode, NFSIOS_DATAINVALIDATE);
-nfs_fscache_reset_inode_cookie(inode);
+nfs_fscache_wait_on_invalidate(inode);
 dfprintk(PAGECACHE, "NFS: (%s/%Ld) data cache invalidated\n",
 inode->i_sb->s_id, (long long)NFS_FILEID(inode));
 return 0;
< at >< at > -927,6 +930,10 < at >< at > static unsigned long nfs_wcc_update_inode(struct inode *inode, struct nfs_fattr
 i_size_write(inode, nfs_size_to_loff_t(fattr->size));
 ret |= NFS_INO_INVALID_ATTR;
 }
+
+if (nfsi->cache_validity & NFS_INO_INVALID_DATA)
+nfs_fscache_invalidate(inode);
+
 return ret;
 }
 
< at >< at > -1108,8 +1115,10 < at >< at > static int nfs_post_op_update_inode_locked(struct inode *inode, struct nfs_fattr
 struct nfs_inode *nfsi = NFS_I(inode);
 
 nfsi->cache_validity |= NFS_INO_INVALID_ATTR|NFS_INO_REVAL_PAGECACHE;
-if (S_ISDIR(inode->i_mode))
+if (S_ISDIR(inode->i_mode)) {
 nfsi->cache_validity |= NFS_INO_INVALID_DATA;
+nfs_fscache_invalidate(inode);
+}
 if ((fattr->valid & NFS_ATTR_FATTR) == 0)
 return 0;
 return nfs_refresh_inode_locked(inode, fattr);
< at >< at > -1401,6 +1410,9 < at >< at > static int nfs_update_inode(struct inode *inode, struct nfs_fattr *fattr)
 (save_cache_validity & NFS_INO_REVAL_FORCED))
 nfsi->cache_validity |= invalid;
 
+if (invalid & NFS_INO_INVALID_DATA)
+nfs_fscache_invalidate(inode);
+
 return 0;
  out_changed:
 /*
diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index f0c849c..2b98a35 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
< at >< at > -64,6 +64,7 < at >< at >
 #include "iostat.h"
 #include "callback.h"
 #include "pnfs.h"
+#include "fscache.h"
 
 #define NFSDBG_FACILITYNFSDBG_PROC
 
< at >< at > -753,6 +754,7 < at >< at > static void update_changeattr(struct inode *dir, struct nfs4_change_info *cinfo)
 if (!cinfo->atomic || cinfo->before != dir->i_version)
 nfs_force_lookup_revalidate(dir);
 dir->i_version = cinfo->after;
+nfs_fscache_invalidate(dir);
 spin_unlock(&dir->i_lock);
 }

[PATCH 09/17] CacheFiles: Implement invalidation

David Howells — 2012-02-08T21:18:13

Implement invalidation for CacheFiles.  This is in two parts:

 (1) Provide an invalidation method (which just truncates the backing file).

 (2) Abort attempts to copy anything read from the backing file whilst
     invalidation is in progress.

Question: CacheFiles uses truncation in a couple of places.  It has been using
notify_change() rather than sys_truncate() or something similar.  This means
it bypasses a bunch of checks and suchlike that it possibly should be making
(security, file locking, lease breaking, vfsmount write).  Should it be using
vfs_truncate() as added by a preceding patch or should it use notify_write()
and assume that anyone poking around in the cache files on disk gets
everything they deserve?

Signed-off-by: David Howells <dhowells< at >redhat.com>
---

 fs/cachefiles/interface.c |   49 +++++++++++++++++++++++++++++++++++++++++++++
 fs/cachefiles/rdwr.c      |    5 ++++-
 2 files changed, 53 insertions(+), 1 deletions(-)

diff --git a/fs/cachefiles/interface.c b/fs/cachefiles/interface.c
index 9bff0f8..7a9d574 100644
--- a/fs/cachefiles/interface.c
+++ b/fs/cachefiles/interface.c
< at >< at > -441,6 +441,54 < at >< at > truncate_failed:
 }
 
 /*
+ * Invalidate an object
+ */
+static void cachefiles_invalidate_object(struct fscache_operation *op)
+{
+struct cachefiles_object *object;
+struct cachefiles_cache *cache;
+const struct cred *saved_cred;
+struct path path;
+uint64_t ni_size;
+int ret;
+
+object = container_of(op->object, struct cachefiles_object, fscache);
+cache = container_of(object->fscache.cache,
+     struct cachefiles_cache, cache);
+
+op->object->cookie->def->get_attr(op->object->cookie->netfs_data,
+  &ni_size);
+
+_enter("{OBJ%x},[%llu]",
+       op->object->debug_id, (unsigned long long)ni_size);
+
+if (object->backer) {
+ASSERT(S_ISREG(object->backer->d_inode->i_mode));
+
+fscache_set_store_limit(&object->fscache, ni_size);
+
+path.dentry = object->backer;
+path.mnt = cache->mnt;
+
+cachefiles_begin_secure(cache, &saved_cred);
+ret = vfs_truncate(&path, 0);
+if (ret == 0)
+ret = vfs_truncate(&path, ni_size);
+cachefiles_end_secure(cache, saved_cred);
+
+if (ret != 0) {
+fscache_set_store_limit(&object->fscache, 0);
+if (ret == -EIO)
+cachefiles_io_error_obj(object,
+"Invalidate failed");
+}
+}
+
+fscache_op_complete(op);
+_leave("");
+}
+
+/*
  * dissociate a cache from all the pages it was backing
  */
 static void cachefiles_dissociate_pages(struct fscache_cache *cache)
< at >< at > -455,6 +503,7 < at >< at > const struct fscache_cache_ops cachefiles_cache_ops = {
 .lookup_complete= cachefiles_lookup_complete,
 .grab_object= cachefiles_grab_object,
 .update_object= cachefiles_update_object,
+.invalidate_object= cachefiles_invalidate_object,
 .drop_object= cachefiles_drop_object,
 .put_object= cachefiles_put_object,
 .sync_cache= cachefiles_sync_cache,
diff --git a/fs/cachefiles/rdwr.c b/fs/cachefiles/rdwr.c
index 31cda81..f437e97 100644
--- a/fs/cachefiles/rdwr.c
+++ b/fs/cachefiles/rdwr.c
< at >< at > -174,7 +174,10 < at >< at > static void cachefiles_read_copier(struct fscache_operation *_op)
 _debug("- copy {%lu}", monitor->back_page->index);
 
 recheck:
-if (PageUptodate(monitor->back_page)) {
+if (test_bit(FSCACHE_COOKIE_INVALIDATING,
+     &object->fscache.cookie->flags)) {
+error = -ESTALE;
+} else if (PageUptodate(monitor->back_page)) {
 copy_highpage(monitor->netfs_page, monitor->back_page);
 fscache_mark_page_cached(monitor->op,
  monitor->netfs_page);

[PATCH 08/17] VFS: Make more complete truncate operation available to CacheFiles

David Howells — 2012-02-08T21:18:03

Make a more complete truncate operation available to CacheFiles (including
security checks and suchlike) so that it can use this to clear invalidated
cache files.

Signed-off-by: David Howells <dhowells< at >redhat.com>
---

 fs/open.c          |   50 +++++++++++++++++++++++++++-----------------------
 include/linux/fs.h |    1 +
 2 files changed, 28 insertions(+), 23 deletions(-)

diff --git a/fs/open.c b/fs/open.c
index 77becc0..7535620 100644
--- a/fs/open.c
+++ b/fs/open.c
< at >< at > -61,33 +61,22 < at >< at > int do_truncate(struct dentry *dentry, loff_t length, unsigned int time_attrs,
 return ret;
 }
 
-static long do_sys_truncate(const char __user *pathname, loff_t length)
+long vfs_truncate(struct path *path, loff_t length)
 {
-struct path path;
 struct inode *inode;
-int error;
-
-error = -EINVAL;
-if (length < 0)/* sorry, but loff_t says... */
-goto out;
+long error;
 
-error = user_path(pathname, &path);
-if (error)
-goto out;
-inode = path.dentry->d_inode;
+inode = path->dentry->d_inode;
 
 /* For directories it's -EISDIR, for other non-regulars - -EINVAL */
-error = -EISDIR;
 if (S_ISDIR(inode->i_mode))
-goto dput_and_out;
-
-error = -EINVAL;
+return -EISDIR;
 if (!S_ISREG(inode->i_mode))
-goto dput_and_out;
+return -EINVAL;
 
-error = mnt_want_write(path.mnt);
+error = mnt_want_write(path->mnt);
 if (error)
-goto dput_and_out;
+goto out;
 
 error = inode_permission(inode, MAY_WRITE);
 if (error)
< at >< at > -111,19 +100,34 < at >< at > static long do_sys_truncate(const char __user *pathname, loff_t length)
 
 error = locks_verify_truncate(inode, NULL, length);
 if (!error)
-error = security_path_truncate(&path);
+error = security_path_truncate(path);
 if (!error)
-error = do_truncate(path.dentry, length, 0, NULL);
+error = do_truncate(path->dentry, length, 0, NULL);
 
 put_write_and_out:
 put_write_access(inode);
 mnt_drop_write_and_out:
-mnt_drop_write(path.mnt);
-dput_and_out:
-path_put(&path);
+mnt_drop_write(path->mnt);
 out:
 return error;
 }
+EXPORT_SYMBOL_GPL(vfs_truncate);
+
+static long do_sys_truncate(const char __user *pathname, loff_t length)
+{
+struct path path;
+int error;
+
+if (length < 0)/* sorry, but loff_t says... */
+return -EINVAL;
+
+error = user_path(pathname, &path);
+if (!error) {
+error = vfs_truncate(&path, length);
+path_put(&path);
+}
+return error;
+}
 
 SYSCALL_DEFINE2(truncate, const char __user *, path, long, length)
 {
diff --git a/include/linux/fs.h b/include/linux/fs.h
index 386da09..79449e4 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
< at >< at > -2014,6 +2014,7 < at >< at > static inline int break_lease(struct inode *inode, unsigned int mode)
 
 /* fs/open.c */
 
+extern long vfs_truncate(struct path *, loff_t);
 extern int do_truncate(struct dentry *, loff_t start, unsigned int time_attrs,
        struct file *filp);
 extern int do_fallocate(struct file *file, int mode, loff_t offset,

[PATCH 07/17] FS-Cache: Provide proper invalidation

David Howells — 2012-02-08T21:17:52

Provide a proper invalidation method rather than relying on the netfs retiring
the cookie it has and getting a new one.  The problem with this is that isn't
easy for the netfs to make sure that it has completed/cancelled all its
outstanding storage and retrieval operations on the cookie it is retiring.

Instead, have the cache provide an invalidation method that will cancel or wait
for all currently outstanding operations before invalidating the cache, and
will cause new operations to queue up behind that.  Whilst invalidation is in
progress, some requests will be rejected until the cache can stack a barrier on
the operation queue to cause new operations to be deferred behind it.

Signed-off-by: David Howells <dhowells< at >redhat.com>
---

 Documentation/filesystems/caching/backend-api.txt |   12 ++++
 Documentation/filesystems/caching/netfs-api.txt   |   46 +++++++++++--
 Documentation/filesystems/caching/object.txt      |   23 ++++---
 fs/fscache/cookie.c                               |   60 ++++++++++++++++++
 fs/fscache/internal.h                             |   10 +++
 fs/fscache/object.c                               |   72 +++++++++++++++++++++
 fs/fscache/operation.c                            |   32 +++++++++
 fs/fscache/page.c                                 |   51 +++++++++++++++
 fs/fscache/stats.c                                |   11 +++
 include/linux/fscache-cache.h                     |    8 ++
 include/linux/fscache.h                           |   38 +++++++++++
 11 files changed, 345 insertions(+), 18 deletions(-)

diff --git a/Documentation/filesystems/caching/backend-api.txt b/Documentation/filesystems/caching/backend-api.txt
index f4769b9..d78bab9 100644
--- a/Documentation/filesystems/caching/backend-api.txt
+++ b/Documentation/filesystems/caching/backend-api.txt
< at >< at > -308,6 +308,18 < at >< at > performed on the denizens of the cache.  These are held in a structure of type:
      obtained by calling object->cookie->def->get_aux()/get_attr().
 
 
+ (*) Invalidate data object [mandatory]:
+
+int (*invalidate_object)(struct fscache_operation *op)
+
+     This is called to invalidate a data object (as pointed to by op->object).
+     All the data stored for this object should be discarded and an
+     attr_changed operation should be performed.  The caller will follow up
+     with an object update operation.
+
+     fscache_op_complete() must be called on op before returning.
+
+
  (*) Discard object [mandatory]:
 
 void (*drop_object)(struct fscache_object *object)
diff --git a/Documentation/filesystems/caching/netfs-api.txt b/Documentation/filesystems/caching/netfs-api.txt
index 7cc6bf2..97e6c0e 100644
--- a/Documentation/filesystems/caching/netfs-api.txt
+++ b/Documentation/filesystems/caching/netfs-api.txt
< at >< at > -35,8 +35,9 < at >< at > This document contains the following sections:
 (12) Index and data file update
 (13) Miscellaneous cookie operations
 (14) Cookie unregistration
-(15) Index and data file invalidation
-(16) FS-Cache specific page flags.
+(15) Index invalidation
+(16) Data file invalidation
+(17) FS-Cache specific page flags.
 
 
 =============================
< at >< at > -767,13 +768,42 < at >< at > the cookies for "child" indices, objects and pages have been relinquished
 first.
 
 
-================================
-INDEX AND DATA FILE INVALIDATION
-================================
+==================
+INDEX INVALIDATION
+==================
+
+There is no direct way to invalidate an index subtree.  To do this, the caller
+should relinquish and retire the cookie they have, and then acquire a new one.
+
+
+======================
+DATA FILE INVALIDATION
+======================
+
+Sometimes it will be necessary to invalidate an object that contains data.
+Typically this will be necessary when the server tells the netfs of a foreign
+change - at which point the netfs has to throw away all the state it had for an
+inode and reload from the server.
+
+To indicate that a cache object should be invalidated, the following function
+can be called:
+
+void fscache_invalidate(struct fscache_cookie *cookie);
+
+This can be called with spinlocks held as it defers the work to a thread pool.
+All extant storage, retrieval and attribute change ops at this point are
+cancelled and discarded.  Some future operations will be rejected until the
+cache has had a chance to insert a barrier in the operations queue.  After
+that, operations will be queued again behind the invalidation operation.
+
+The invalidation operation will perform an attribute change operation and an
+auxiliary data update operation as it is very likely these will have changed.
+
+Using the following function, the netfs can wait for the invalidation operation
+to have reached a point at which it can start submitting ordinary operations
+once again:
 
-There is no direct way to invalidate an index subtree or a data file.  To do
-this, the caller should relinquish and retire the cookie they have, and then
-acquire a new one.
+void fscache_wait_on_invalidate(struct fscache_cookie *cookie);
 
 
 ===========================
diff --git a/Documentation/filesystems/caching/object.txt b/Documentation/filesystems/caching/object.txt
index 5831334..100ff41 100644
--- a/Documentation/filesystems/caching/object.txt
+++ b/Documentation/filesystems/caching/object.txt
< at >< at > -216,7 +216,14 < at >< at > servicing netfs requests:
      The normal running state.  In this state, requests the netfs makes will be
      passed on to the cache.
 
- (6) State FSCACHE_OBJECT_UPDATING.
+ (6) State FSCACHE_OBJECT_INVALIDATING.
+
+     The object is undergoing invalidation.  When the state comes here, it
+     discards all pending read, write and attribute change operations as it is
+     going to clear out the cache entirely and reinitialise it.  It will then
+     continue to the FSCACHE_OBJECT_UPDATING state.
+
+ (7) State FSCACHE_OBJECT_UPDATING.
 
      The state machine comes here to update the object in the cache from the
      netfs's records.  This involves updating the auxiliary data that is used
< at >< at > -225,13 +232,13 < at >< at > servicing netfs requests:
 And there are terminal states in which an object cleans itself up, deallocates
 memory and potentially deletes stuff from disk:
 
- (7) State FSCACHE_OBJECT_LC_DYING.
+ (8) State FSCACHE_OBJECT_LC_DYING.
 
      The object comes here if it is dying because of a lookup or creation
      error.  This would be due to a disk error or system error of some sort.
      Temporary data is cleaned up, and the parent is released.
 
- (8) State FSCACHE_OBJECT_DYING.
+ (9) State FSCACHE_OBJECT_DYING.
 
      The object comes here if it is dying due to an error, because its parent
      cookie has been relinquished by the netfs or because the cache is being
< at >< at > -241,27 +248,27 < at >< at > memory and potentially deletes stuff from disk:
      can destroy themselves.  This object waits for all its children to go away
      before advancing to the next state.
 
- (9) State FSCACHE_OBJECT_ABORT_INIT.
+(10) State FSCACHE_OBJECT_ABORT_INIT.
 
      The object comes to this state if it was waiting on its parent in
      FSCACHE_OBJECT_INIT, but its parent died.  The object will destroy itself
      so that the parent may proceed from the FSCACHE_OBJECT_DYING state.
 
-(10) State FSCACHE_OBJECT_RELEASING.
-(11) State FSCACHE_OBJECT_RECYCLING.
+(11) State FSCACHE_OBJECT_RELEASING.
+(12) State FSCACHE_OBJECT_RECYCLING.
 
      The object comes to one of these two states when dying once it is rid of
      all its children, if it is dying because the netfs relinquished its
      cookie.  In the first state, the cached data is expected to persist, and
      in the second it will be deleted.
 
-(12) State FSCACHE_OBJECT_WITHDRAWING.
+(13) State FSCACHE_OBJECT_WITHDRAWING.
 
      The object transits to this state if the cache decides it wants to
      withdraw the object from service, perhaps to make space, but also due to
      error or just because the whole cache is being withdrawn.
 
-(13) State FSCACHE_OBJECT_DEAD.
+(14) State FSCACHE_OBJECT_DEAD.
 
      The object transits to this state when the in-memory object record is
      ready to be deleted.  The object processor shouldn't ever see an object in
diff --git a/fs/fscache/cookie.c b/fs/fscache/cookie.c
index 66be9ec..8dcb114 100644
--- a/fs/fscache/cookie.c
+++ b/fs/fscache/cookie.c
< at >< at > -370,6 +370,66 < at >< at > cant_attach_object:
 }
 
 /*
+ * Invalidate an object.  Callable with spinlocks held.
+ */
+void __fscache_invalidate(struct fscache_cookie *cookie)
+{
+struct fscache_object *object;
+
+_enter("{%s}", cookie->def->name);
+
+fscache_stat(&fscache_n_invalidates);
+
+/* Only permit invalidation of data files.  Invalidating an index will
+ * require the caller to release all its attachments to the tree rooted
+ * there, and if it's doing that, it may as well just retire the
+ * cookie.
+ */
+ASSERTCMP(cookie->def->type, ==, FSCACHE_COOKIE_TYPE_DATAFILE);
+
+/* We will be updating the cookie too. */
+BUG_ON(!cookie->def->get_aux);
+
+/* If there's an object, we tell the object state machine to handle the
+ * invalidation on our behalf, otherwise there's nothing to do.
+ */
+if (!hlist_empty(&cookie->backing_objects)) {
+spin_lock(&cookie->lock);
+
+if (!hlist_empty(&cookie->backing_objects) &&
+    !test_and_set_bit(FSCACHE_COOKIE_INVALIDATING,
+      &cookie->flags)) {
+object = hlist_entry(cookie->backing_objects.first,
+     struct fscache_object,
+     cookie_link);
+if (object->state < FSCACHE_OBJECT_DYING)
+fscache_raise_event(
+object, FSCACHE_OBJECT_EV_INVALIDATE);
+}
+
+spin_unlock(&cookie->lock);
+}
+
+_leave("");
+}
+EXPORT_SYMBOL(__fscache_invalidate);
+
+/*
+ * Wait for object invalidation to complete.
+ */
+void __fscache_wait_on_invalidate(struct fscache_cookie *cookie)
+{
+_enter("%p", cookie);
+
+wait_on_bit(&cookie->flags, FSCACHE_COOKIE_INVALIDATING,
+    fscache_wait_bit_interruptible,
+    TASK_UNINTERRUPTIBLE);
+
+_leave("");
+}
+EXPORT_SYMBOL(__fscache_wait_on_invalidate);
+
+/*
  * update the index entries backing a cookie
  */
 void __fscache_update_cookie(struct fscache_cookie *cookie)
diff --git a/fs/fscache/internal.h b/fs/fscache/internal.h
index f6aad48..c811793 100644
--- a/fs/fscache/internal.h
+++ b/fs/fscache/internal.h
< at >< at > -122,11 +122,17 < at >< at > extern int fscache_submit_exclusive_op(struct fscache_object *,
 extern int fscache_submit_op(struct fscache_object *,
      struct fscache_operation *);
 extern int fscache_cancel_op(struct fscache_operation *);
+extern void fscache_cancel_all_ops(struct fscache_object *);
 extern void fscache_abort_object(struct fscache_object *);
 extern void fscache_start_operations(struct fscache_object *);
 extern void fscache_operation_gc(struct work_struct *);
 
 /*
+ * page.c
+ */
+extern void fscache_invalidate_writes(struct fscache_cookie *);
+
+/*
  * proc.c
  */
 #ifdef CONFIG_PROC_FS
< at >< at > -205,6 +211,9 < at >< at > extern atomic_t fscache_n_acquires_ok;
 extern atomic_t fscache_n_acquires_nobufs;
 extern atomic_t fscache_n_acquires_oom;
 
+extern atomic_t fscache_n_invalidates;
+extern atomic_t fscache_n_invalidates_run;
+
 extern atomic_t fscache_n_updates;
 extern atomic_t fscache_n_updates_null;
 extern atomic_t fscache_n_updates_run;
< at >< at > -237,6 +246,7 < at >< at > extern atomic_t fscache_n_cop_alloc_object;
 extern atomic_t fscache_n_cop_lookup_object;
 extern atomic_t fscache_n_cop_lookup_complete;
 extern atomic_t fscache_n_cop_grab_object;
+extern atomic_t fscache_n_cop_invalidate_object;
 extern atomic_t fscache_n_cop_update_object;
 extern atomic_t fscache_n_cop_drop_object;
 extern atomic_t fscache_n_cop_put_object;
diff --git a/fs/fscache/object.c b/fs/fscache/object.c
index 773bc79..80b5491 100644
--- a/fs/fscache/object.c
+++ b/fs/fscache/object.c
< at >< at > -14,6 +14,7 < at >< at >
 
 #define FSCACHE_DEBUG_LEVEL COOKIE
 #include <linux/module.h>
+#include <linux/slab.h>
 #include "internal.h"
 
 const char *fscache_object_states[FSCACHE_OBJECT__NSTATES] = {
< at >< at > -22,6 +23,7 < at >< at > const char *fscache_object_states[FSCACHE_OBJECT__NSTATES] = {
 [FSCACHE_OBJECT_CREATING]= "OBJECT_CREATING",
 [FSCACHE_OBJECT_AVAILABLE]= "OBJECT_AVAILABLE",
 [FSCACHE_OBJECT_ACTIVE]= "OBJECT_ACTIVE",
+[FSCACHE_OBJECT_INVALIDATING]= "OBJECT_INVALIDATING",
 [FSCACHE_OBJECT_UPDATING]= "OBJECT_UPDATING",
 [FSCACHE_OBJECT_DYING]= "OBJECT_DYING",
 [FSCACHE_OBJECT_LC_DYING]= "OBJECT_LC_DYING",
< at >< at > -39,6 +41,7 < at >< at > const char fscache_object_states_short[FSCACHE_OBJECT__NSTATES][5] = {
 [FSCACHE_OBJECT_CREATING]= "CRTN",
 [FSCACHE_OBJECT_AVAILABLE]= "AVBL",
 [FSCACHE_OBJECT_ACTIVE]= "ACTV",
+[FSCACHE_OBJECT_INVALIDATING]= "INVL",
 [FSCACHE_OBJECT_UPDATING]= "UPDT",
 [FSCACHE_OBJECT_DYING]= "DYNG",
 [FSCACHE_OBJECT_LC_DYING]= "LCDY",
< at >< at > -54,6 +57,7 < at >< at > static void fscache_put_object(struct fscache_object *);
 static void fscache_initialise_object(struct fscache_object *);
 static void fscache_lookup_object(struct fscache_object *);
 static void fscache_object_available(struct fscache_object *);
+static void fscache_invalidate_object(struct fscache_object *);
 static void fscache_release_object(struct fscache_object *);
 static void fscache_withdraw_object(struct fscache_object *);
 static void fscache_enqueue_dependents(struct fscache_object *);
< at >< at > -79,6 +83,15 < at >< at > static inline void fscache_done_parent_op(struct fscache_object *object)
 }
 
 /*
+ * Notify netfs of invalidation completion.
+ */
+static inline void fscache_invalidation_complete(struct fscache_cookie *cookie)
+{
+if (test_and_clear_bit(FSCACHE_COOKIE_INVALIDATING, &cookie->flags))
+wake_up_bit(&cookie->flags, FSCACHE_COOKIE_INVALIDATING);
+}
+
+/*
  * process events that have been sent to an object's state machine
  * - initiates parent lookup
  * - does object lookup
< at >< at > -125,6 +138,16 < at >< at > static void fscache_object_state_machine(struct fscache_object *object)
 case FSCACHE_OBJECT_ACTIVE:
 goto active_transit;
 
+/* Invalidate an object on disk */
+case FSCACHE_OBJECT_INVALIDATING:
+clear_bit(FSCACHE_OBJECT_EV_INVALIDATE, &object->events);
+fscache_stat(&fscache_n_invalidates_run);
+fscache_stat(&fscache_n_cop_invalidate_object);
+fscache_invalidate_object(object);
+fscache_stat_d(&fscache_n_cop_invalidate_object);
+fscache_raise_event(object, FSCACHE_OBJECT_EV_UPDATE);
+goto active_transit;
+
 /* update the object metadata on disk */
 case FSCACHE_OBJECT_UPDATING:
 clear_bit(FSCACHE_OBJECT_EV_UPDATE, &object->events);
< at >< at > -275,6 +298,9 < at >< at > active_transit:
 case FSCACHE_OBJECT_EV_ERROR:
 new_state = FSCACHE_OBJECT_DYING;
 goto change_state;
+case FSCACHE_OBJECT_EV_INVALIDATE:
+new_state = FSCACHE_OBJECT_INVALIDATING;
+goto change_state;
 case FSCACHE_OBJECT_EV_UPDATE:
 new_state = FSCACHE_OBJECT_UPDATING;
 goto change_state;
< at >< at > -679,6 +705,7 < at >< at > static void fscache_withdraw_object(struct fscache_object *object)
 if (object->cookie == cookie) {
 hlist_del_init(&object->cookie_link);
 object->cookie = NULL;
+fscache_invalidation_complete(cookie);
 detached = true;
 }
 spin_unlock(&cookie->lock);
< at >< at > -888,3 +915,48 < at >< at > enum fscache_checkaux fscache_check_aux(struct fscache_object *object,
 return result;
 }
 EXPORT_SYMBOL(fscache_check_aux);
+
+/*
+ * Asynchronously invalidate an object.
+ */
+static void fscache_invalidate_object(struct fscache_object *object)
+{
+struct fscache_operation *op;
+struct fscache_cookie *cookie = object->cookie;
+
+_enter("{OBJ%x}", object->debug_id);
+
+/* Reject any new read/write ops and abort any that are pending. */
+fscache_invalidate_writes(cookie);
+clear_bit(FSCACHE_OBJECT_PENDING_WRITE, &object->flags);
+fscache_cancel_all_ops(object);
+
+/* Now we have to wait for in-progress reads and writes */
+op = kzalloc(sizeof(*op), GFP_KERNEL);
+if (!op) {
+fscache_raise_event(object, FSCACHE_OBJECT_EV_ERROR);
+_leave(" [ENOMEM]");
+return;
+}
+
+fscache_operation_init(op, object->cache->ops->invalidate_object, NULL);
+op->flags = FSCACHE_OP_ASYNC | (1 << FSCACHE_OP_EXCLUSIVE);
+
+spin_lock(&cookie->lock);
+if (fscache_submit_exclusive_op(object, op) < 0)
+BUG();
+spin_unlock(&cookie->lock);
+fscache_put_operation(op);
+
+/* Once we've completed the invalidation, we know there will be no data
+ * stored in the cache and thus we can reinstate the data-check-skip
+ * optimisation.
+ */
+set_bit(FSCACHE_COOKIE_NO_DATA_YET, &cookie->flags);
+
+/* We can allow read and write requests to come in once again.  They'll
+ * queue up behind our exclusive invalidation operation.
+ */
+fscache_invalidation_complete(cookie);
+_leave("");
+}
diff --git a/fs/fscache/operation.c b/fs/fscache/operation.c
index 748f955..c58dbe6 100644
--- a/fs/fscache/operation.c
+++ b/fs/fscache/operation.c
< at >< at > -324,6 +324,38 < at >< at > int fscache_cancel_op(struct fscache_operation *op)
 }
 
 /*
+ * Cancel all pending operations on an object
+ */
+void fscache_cancel_all_ops(struct fscache_object *object)
+{
+struct fscache_operation *op;
+
+_enter("OBJ%x", object->debug_id);
+
+spin_lock(&object->lock);
+
+while (!list_empty(&object->pending_ops)) {
+op = list_entry(object->pending_ops.next,
+struct fscache_operation, pend_link);
+fscache_stat(&fscache_n_op_cancelled);
+list_del_init(&op->pend_link);
+
+ASSERTCMP(op->state, ==, FSCACHE_OP_ST_PENDING);
+op->state = FSCACHE_OP_ST_CANCELLED;
+
+if (test_bit(FSCACHE_OP_EXCLUSIVE, &op->flags))
+object->n_exclusive--;
+if (test_and_clear_bit(FSCACHE_OP_WAITING, &op->flags))
+wake_up_bit(&op->flags, FSCACHE_OP_WAITING);
+fscache_put_operation(op);
+cond_resched_lock(&object->lock);
+}
+
+spin_unlock(&object->lock);
+_leave("");
+}
+
+/*
  * Record the completion of an in-progress operation.
  */
 void fscache_op_complete(struct fscache_operation *op)
diff --git a/fs/fscache/page.c b/fs/fscache/page.c
index b38b13d..7bf9d25 100644
--- a/fs/fscache/page.c
+++ b/fs/fscache/page.c
< at >< at > -361,6 +361,11 < at >< at > int __fscache_read_or_alloc_page(struct fscache_cookie *cookie,
 if (hlist_empty(&cookie->backing_objects))
 goto nobufs;
 
+if (test_bit(FSCACHE_COOKIE_INVALIDATING, &cookie->flags)) {
+_leave(" = -ENOBUFS [invalidating]");
+return -ENOBUFS;
+}
+
 ASSERTCMP(cookie->def->type, !=, FSCACHE_COOKIE_TYPE_INDEX);
 ASSERTCMP(page, !=, NULL);
 
< at >< at > -483,6 +488,11 < at >< at > int __fscache_read_or_alloc_pages(struct fscache_cookie *cookie,
 if (hlist_empty(&cookie->backing_objects))
 goto nobufs;
 
+if (test_bit(FSCACHE_COOKIE_INVALIDATING, &cookie->flags)) {
+_leave(" = -ENOBUFS [invalidating]");
+return -ENOBUFS;
+}
+
 ASSERTCMP(cookie->def->type, !=, FSCACHE_COOKIE_TYPE_INDEX);
 ASSERTCMP(*nr_pages, >, 0);
 ASSERT(!list_empty(pages));
< at >< at > -591,6 +601,11 < at >< at > int __fscache_alloc_page(struct fscache_cookie *cookie,
 ASSERTCMP(cookie->def->type, !=, FSCACHE_COOKIE_TYPE_INDEX);
 ASSERTCMP(page, !=, NULL);
 
+if (test_bit(FSCACHE_COOKIE_INVALIDATING, &cookie->flags)) {
+_leave(" = -ENOBUFS [invalidating]");
+return -ENOBUFS;
+}
+
 if (fscache_wait_for_deferred_lookup(cookie) < 0)
 return -ERESTARTSYS;
 
< at >< at > -731,6 +746,37 < at >< at > superseded:
 }
 
 /*
+ * Clear the pages pending writing for invalidation
+ */
+void fscache_invalidate_writes(struct fscache_cookie *cookie)
+{
+struct page *page;
+void *results[16];
+int n, i;
+
+_enter("");
+
+while (spin_lock(&cookie->stores_lock),
+       n = radix_tree_gang_lookup_tag(&cookie->stores, results, 0,
+      ARRAY_SIZE(results),
+      FSCACHE_COOKIE_PENDING_TAG),
+       n > 0) {
+for (i = n - 1; i >= 0; i--) {
+page = results[i];
+radix_tree_delete(&cookie->stores, page->index);
+}
+
+spin_unlock(&cookie->stores_lock);
+
+for (i = n - 1; i >= 0; i--)
+page_cache_release(results[i]);
+}
+
+spin_unlock(&cookie->stores_lock);
+_leave("");
+}
+
+/*
  * request a page be stored in the cache
  * - returns:
  *   -ENOMEM- out of memory, nothing done
< at >< at > -776,6 +822,11 < at >< at > int __fscache_write_page(struct fscache_cookie *cookie,
 
 fscache_stat(&fscache_n_stores);
 
+if (test_bit(FSCACHE_COOKIE_INVALIDATING, &cookie->flags)) {
+_leave(" = -ENOBUFS [invalidating]");
+return -ENOBUFS;
+}
+
 op = kzalloc(sizeof(*op), GFP_NOIO | __GFP_NOMEMALLOC | __GFP_NORETRY);
 if (!op)
 goto nomem;
diff --git a/fs/fscache/stats.c b/fs/fscache/stats.c
index 4765190..51cdaee 100644
--- a/fs/fscache/stats.c
+++ b/fs/fscache/stats.c
< at >< at > -80,6 +80,9 < at >< at > atomic_t fscache_n_acquires_ok;
 atomic_t fscache_n_acquires_nobufs;
 atomic_t fscache_n_acquires_oom;
 
+atomic_t fscache_n_invalidates;
+atomic_t fscache_n_invalidates_run;
+
 atomic_t fscache_n_updates;
 atomic_t fscache_n_updates_null;
 atomic_t fscache_n_updates_run;
< at >< at > -112,6 +115,7 < at >< at > atomic_t fscache_n_cop_alloc_object;
 atomic_t fscache_n_cop_lookup_object;
 atomic_t fscache_n_cop_lookup_complete;
 atomic_t fscache_n_cop_grab_object;
+atomic_t fscache_n_cop_invalidate_object;
 atomic_t fscache_n_cop_update_object;
 atomic_t fscache_n_cop_drop_object;
 atomic_t fscache_n_cop_put_object;
< at >< at > -168,6 +172,10 < at >< at > static int fscache_stats_show(struct seq_file *m, void *v)
    atomic_read(&fscache_n_object_created),
    atomic_read(&fscache_n_object_lookups_timed_out));
 
+seq_printf(m, "Invals : n=%u run=%u\n",
+   atomic_read(&fscache_n_invalidates),
+   atomic_read(&fscache_n_invalidates_run));
+
 seq_printf(m, "Updates: n=%u nul=%u run=%u\n",
    atomic_read(&fscache_n_updates),
    atomic_read(&fscache_n_updates_null),
< at >< at > -246,7 +254,8 < at >< at > static int fscache_stats_show(struct seq_file *m, void *v)
    atomic_read(&fscache_n_cop_lookup_object),
    atomic_read(&fscache_n_cop_lookup_complete),
    atomic_read(&fscache_n_cop_grab_object));
-seq_printf(m, "CacheOp: upo=%d dro=%d pto=%d atc=%d syn=%d\n",
+seq_printf(m, "CacheOp: inv=%d upo=%d dro=%d pto=%d atc=%d syn=%d\n",
+   atomic_read(&fscache_n_cop_invalidate_object),
    atomic_read(&fscache_n_cop_update_object),
    atomic_read(&fscache_n_cop_drop_object),
    atomic_read(&fscache_n_cop_put_object),
diff --git a/include/linux/fscache-cache.h b/include/linux/fscache-cache.h
index f5facd1..1e454ad 100644
--- a/include/linux/fscache-cache.h
+++ b/include/linux/fscache-cache.h
< at >< at > -254,6 +254,9 < at >< at > struct fscache_cache_ops {
 /* store the updated auxiliary data on an object */
 void (*update_object)(struct fscache_object *object);
 
+/* Invalidate an object */
+void (*invalidate_object)(struct fscache_operation *op);
+
 /* discard the resources pinned by an object and effect retirement if
  * necessary */
 void (*drop_object)(struct fscache_object *object);
< at >< at > -329,6 +332,7 < at >< at > struct fscache_cookie {
 #define FSCACHE_COOKIE_FILLING4/* T if filling object incrementally */
 #define FSCACHE_COOKIE_UNAVAILABLE5/* T if cookie is unavailable (error, etc) */
 #define FSCACHE_COOKIE_WAITING_ON_READS6/* T if cookie is waiting on reads */
+#define FSCACHE_COOKIE_INVALIDATING7/* T if cookie is being invalidated */
 };
 
 extern struct fscache_cookie fscache_fsdef_index;
< at >< at > -345,6 +349,7 < at >< at > struct fscache_object {
 /* active states */
 FSCACHE_OBJECT_AVAILABLE,/* cleaning up object after creation */
 FSCACHE_OBJECT_ACTIVE,/* object is usable */
+FSCACHE_OBJECT_INVALIDATING,/* object is invalidating */
 FSCACHE_OBJECT_UPDATING,/* object is updating */
 
 /* terminal states */
< at >< at > -378,7 +383,8 < at >< at > struct fscache_object {
 #define FSCACHE_OBJECT_EV_RELEASE4/* T if netfs requested object release */
 #define FSCACHE_OBJECT_EV_RETIRE5/* T if netfs requested object retirement */
 #define FSCACHE_OBJECT_EV_WITHDRAW6/* T if cache requested object withdrawal */
-#define FSCACHE_OBJECT_EVENTS_MASK0x7f/* mask of all events*/
+#define FSCACHE_OBJECT_EV_INVALIDATE7/* T if cache requested object invalidation */
+#define FSCACHE_OBJECT_EVENTS_MASK0xff/* mask of all events*/
 
 unsigned longflags;
 #define FSCACHE_OBJECT_LOCK0/* T if object is busy being processed */
diff --git a/include/linux/fscache.h b/include/linux/fscache.h
index f4b6353..7a08623 100644
--- a/include/linux/fscache.h
+++ b/include/linux/fscache.h
< at >< at > -185,6 +185,8 < at >< at > extern struct fscache_cookie *__fscache_acquire_cookie(
 extern void __fscache_relinquish_cookie(struct fscache_cookie *, int);
 extern void __fscache_update_cookie(struct fscache_cookie *);
 extern int __fscache_attr_changed(struct fscache_cookie *);
+extern void __fscache_invalidate(struct fscache_cookie *);
+extern void __fscache_wait_on_invalidate(struct fscache_cookie *);
 extern int __fscache_read_or_alloc_page(struct fscache_cookie *,
 struct page *,
 fscache_rw_complete_t,
< at >< at > -390,6 +392,42 < at >< at > int fscache_attr_changed(struct fscache_cookie *cookie)
 }
 
 /**
+ * fscache_invalidate - Notify cache that an object needs invalidation
+ * < at >cookie: The cookie representing the cache object
+ *
+ * Notify the cache that an object is needs to be invalidated and that it
+ * should abort any retrievals or stores it is doing on the cache.  The object
+ * is then marked non-caching until such time as the invalidation is complete.
+ *
+ * This can be called with spinlocks held.
+ *
+ * See Documentation/filesystems/caching/netfs-api.txt for a complete
+ * description.
+ */
+static inline
+void fscache_invalidate(struct fscache_cookie *cookie)
+{
+if (fscache_cookie_valid(cookie))
+__fscache_invalidate(cookie);
+}
+
+/**
+ * fscache_wait_on_invalidate - Wait for invalidation to complete
+ * < at >cookie: The cookie representing the cache object
+ *
+ * Wait for the invalidation of an object to complete.
+ *
+ * See Documentation/filesystems/caching/netfs-api.txt for a complete
+ * description.
+ */
+static inline
+void fscache_wait_on_invalidate(struct fscache_cookie *cookie)
+{
+if (fscache_cookie_valid(cookie))
+__fscache_wait_on_invalidate(cookie);
+}
+
+/**
  * fscache_reserve_space - Reserve data space for a cached object
  * < at >cookie: The cookie representing the cache object
  * < at >i_size: The amount of space to be reserved

[PATCH 05/17] FS-Cache: Make cookie relinquishment wait for outstanding reads

David Howells — 2012-02-08T21:17:31

Make fscache_relinquish_cookie() log a warning and wait if there are any
outstanding reads left on the cookie it was given.

Signed-off-by: David Howells <dhowells< at >redhat.com>
---

 fs/fscache/cookie.c           |   18 ++++++++++++++----
 fs/fscache/operation.c        |   10 ++++++++--
 include/linux/fscache-cache.h |    1 +
 3 files changed, 23 insertions(+), 6 deletions(-)

diff --git a/fs/fscache/cookie.c b/fs/fscache/cookie.c
index 0666996..66be9ec 100644
--- a/fs/fscache/cookie.c
+++ b/fs/fscache/cookie.c
< at >< at > -442,22 +442,32 < at >< at > void __fscache_relinquish_cookie(struct fscache_cookie *cookie, int retire)
 
 event = retire ? FSCACHE_OBJECT_EV_RETIRE : FSCACHE_OBJECT_EV_RELEASE;
 
+try_again:
 spin_lock(&cookie->lock);
 
 /* break links with all the active objects */
 while (!hlist_empty(&cookie->backing_objects)) {
+int n_reads;
 object = hlist_entry(cookie->backing_objects.first,
      struct fscache_object,
      cookie_link);
 
 _debug("RELEASE OBJ%x", object->debug_id);
 
-if (atomic_read(&object->n_reads)) {
+set_bit(FSCACHE_COOKIE_WAITING_ON_READS, &cookie->flags);
+n_reads = atomic_read(&object->n_reads);
+if (n_reads) {
+int n_ops = object->n_ops;
+int n_in_progress = object->n_in_progress;
 spin_unlock(&cookie->lock);
 printk(KERN_ERR "FS-Cache:"
-       " Cookie '%s' still has %d outstanding reads\n",
-       cookie->def->name, atomic_read(&object->n_reads));
-BUG();
+       " Cookie '%s' still has %d outstanding reads (%d,%d)\n",
+       cookie->def->name,
+       n_reads, n_ops, n_in_progress);
+wait_on_bit(&cookie->flags, FSCACHE_COOKIE_WAITING_ON_READS,
+    fscache_wait_bit, TASK_UNINTERRUPTIBLE);
+printk("Wait finished\n");
+goto try_again;
 }
 
 /* detach each cache object from the object cookie */
diff --git a/fs/fscache/operation.c b/fs/fscache/operation.c
index 30afdfa..c857ab8 100644
--- a/fs/fscache/operation.c
+++ b/fs/fscache/operation.c
< at >< at > -340,8 +340,14 < at >< at > void fscache_put_operation(struct fscache_operation *op)
 
 object = op->object;
 
-if (test_bit(FSCACHE_OP_DEC_READ_CNT, &op->flags))
-atomic_dec(&object->n_reads);
+if (test_bit(FSCACHE_OP_DEC_READ_CNT, &op->flags)) {
+if (atomic_dec_and_test(&object->n_reads)) {
+clear_bit(FSCACHE_COOKIE_WAITING_ON_READS,
+  &object->cookie->flags);
+wake_up_bit(&object->cookie->flags,
+    FSCACHE_COOKIE_WAITING_ON_READS);
+}
+}
 
 /* now... we may get called with the object spinlock held, so we
  * complete the cleanup here only if we can immediately acquire the
diff --git a/include/linux/fscache-cache.h b/include/linux/fscache-cache.h
index 9879183..e3d6d93 100644
--- a/include/linux/fscache-cache.h
+++ b/include/linux/fscache-cache.h
< at >< at > -301,6 +301,7 < at >< at > struct fscache_cookie {
 #define FSCACHE_COOKIE_PENDING_FILL3/* T if pending initial fill on object */
 #define FSCACHE_COOKIE_FILLING4/* T if filling object incrementally */
 #define FSCACHE_COOKIE_UNAVAILABLE5/* T if cookie is unavailable (error, etc) */
+#define FSCACHE_COOKIE_WAITING_ON_READS6/* T if cookie is waiting on reads */
 };
 
 extern struct fscache_cookie fscache_fsdef_index;