gmane.linux.file-systems.cachefs.general http://blog.gmane.org/gmane.linux.file-systems.cachefs.general hourly 1 1901-01-01T00:00+00:00 Gmane http://gmane.org/img/gmane-25t.png http://gmane.org NFS caching on the server-side http://comments.gmane.org/gmane.linux.file-systems.cachefs.general/3100 <pre>Hello, I need help for configuring a NFS version 4 server for caching what clients read from it. That is, the server has to cache locally (not the client) what any NFS client reads from this server. The porpouse of this configuration if to relief the work load of the server HDD. Any suggestion or recommendation will be appreciated. Abraham Aldaco abraham.aldaco&lt; at &gt;gmail.com </pre> Abraham Aldaco 2012-04-26T16:06:02 nfs/fscache writeback cache http://comments.gmane.org/gmane.linux.file-systems.cachefs.general/3099 <pre>Hi, First of all, I am not sure if this list is appropriate place to ask that question but I think it is the most relevant one. I saw this project proposal[1] on the Fedora Project's GSOC 2012 ideas page few weeks ago. It proposes to add writeback caching support to NFS. I didn't apply to GSOC since I am not a student but I want to spend my time working on that project although this is not accepted by GSOC. I want to learn that is there any ongoing effort on that side? If any, could you point me to that so that i will not duplicate work and I can contribute to current project. [1] - http://fedoraproject.org/wiki/Summer_coding_ideas_for_2012#Linux_kernel_project Thanks </pre> Sertaç Olgunsoylu 2012-04-26T03:10:28 oops when I mount nfs with -o fsc http://comments.gmane.org/gmane.linux.file-systems.cachefs.general/3097 <pre>Hi: in nfs(RHEL6.x), i do as following. #cat /etc/exports /nfsroot *(rw,no_root_squash,fsid=0,insecure) #mount –t nfs4 –o fsc localhost:/ /nfsmnt #sh create_file.sh wait a while, i find oops. PID: 4196 TASK: ffff88012365b580 CPU: 0 COMMAND: "sh" #0 [ffff880139435750] machine_kexec at ffffffff810310db #1 [ffff8801394357b0] crash_kexec at ffffffff810b63b2 #2 [ffff880139435880] oops_end at ffffffff814dec50 #3 [ffff8801394358b0] die at ffffffff8100f2fb #4 [ffff8801394358e0] do_trap at ffffffff814de544 #5 [ffff880139435940] do_invalid_op at ffffffff8100ceb5 #6 [ffff8801394359e0] invalid_op at ffffffff8100bf5b [exception RIP: __nfs_fscache_invalidate_page+121] #7 [ffff880139435ab0] nfs_invalidate_page at ffffffffa06fff6e [nfs] #8 [ffff880139435ad0] do_invalidatepage at ffffffff811243d5 #9 [ffff880139435ae0] truncate_inode_page at ffffffff811245f2 #10 [ffff880139435b00] truncate_inode_pages_range at ffffffff811248f0 #11 [ffff880139435bf0] truncate_inode_pages at ffffffff81124c05 #12 [ffff880139435c00] truncate_pagecache at ffffffff81124c57 #13 [ffff880139435c30] nfs_setattr_update_inode at ffffffffa07033ab [nfs] #14 [ffff880139435c70] nfs4_proc_setattr at ffffffffa071bb45 [nfs] #15 [ffff880139435cb0] nfs_setattr at ffffffffa07043e0 [nfs] #16 [ffff880139435cf0] notify_change at ffffffff8118e0c8 #17 [ffff880139435d60] do_truncate at ffffffff81170a84 #18 [ffff880139435dd0] do_filp_open at ffffffff81182ba9 #19 [ffff880139435f20] do_sys_open at ffffffff8116f849 #20 [ffff880139435f70] sys_open at ffffffff8116f960 #21 [ffff880139435f80] tracesys at ffffffff8100b387 (via system_call) #!/bin/bash i=0 BEGIN_TIME=$(date +%s) TIME_OUT=14400 #Set the testTime. SERVER_FILE="/nfsroot/file" CLIENT_FILE="/nfsmnt/file" while : do dd if=/dev/zero of="$SERVER_FILE$i" bs=1b count=1 &amp;&gt;/dev/null cat "$CLIENT_FILE$i" &amp;&gt;/dev/null echo "good" &gt; "$CLIENT_FILE$i" rm -fr "$CLIENT_FILE$i" now=$(date +%s) if (($now - $BEGIN_TIME &gt; $TIME_OUT)) then break fi let i=i+1 done echo "This probleam may be fix.\n" </pre> fanchaoting 2012-03-28T03:48:38 oops when I mount nfs with -o fsc http://comments.gmane.org/gmane.linux.file-systems.cachefs.general/3095 <pre>Hi: in nfs(RHEL6.x), i do as following. #cat /etc/exports /nfsroot *(rw,no_root_squash,fsid=0,insecure) #mount –t nfs4 –o fsc localhost:/ /nfsmnt #sh create_file.sh wait a while, i find oops. PID: 4196 TASK: ffff88012365b580 CPU: 0 COMMAND: "sh" #0 [ffff880139435750] machine_kexec at ffffffff810310db #1 [ffff8801394357b0] crash_kexec at ffffffff810b63b2 #2 [ffff880139435880] oops_end at ffffffff814dec50 #3 [ffff8801394358b0] die at ffffffff8100f2fb #4 [ffff8801394358e0] do_trap at ffffffff814de544 #5 [ffff880139435940] do_invalid_op at ffffffff8100ceb5 #6 [ffff8801394359e0] invalid_op at ffffffff8100bf5b [exception RIP: __nfs_fscache_invalidate_page+121] #7 [ffff880139435ab0] nfs_invalidate_page at ffffffffa06fff6e [nfs] #8 [ffff880139435ad0] do_invalidatepage at ffffffff811243d5 #9 [ffff880139435ae0] truncate_inode_page at ffffffff811245f2 #10 [ffff880139435b00] truncate_inode_pages_range at ffffffff811248f0 #11 [ffff880139435bf0] truncate_inode_pages at ffffffff81124c05 #12 [ffff880139435c00] truncate_pagecache at ffffffff81124c57 #13 [ffff880139435c30] nfs_setattr_update_inode at ffffffffa07033ab [nfs] #14 [ffff880139435c70] nfs4_proc_setattr at ffffffffa071bb45 [nfs] #15 [ffff880139435cb0] nfs_setattr at ffffffffa07043e0 [nfs] #16 [ffff880139435cf0] notify_change at ffffffff8118e0c8 #17 [ffff880139435d60] do_truncate at ffffffff81170a84 #18 [ffff880139435dd0] do_filp_open at ffffffff81182ba9 #19 [ffff880139435f20] do_sys_open at ffffffff8116f849 #20 [ffff880139435f70] sys_open at ffffffff8116f960 #21 [ffff880139435f80] tracesys at ffffffff8100b387 (via system_call) #!/bin/bash i=0 BEGIN_TIME=$(date +%s) TIME_OUT=14400 #Set the testTime. SERVER_FILE="/nfsroot/file" CLIENT_FILE="/nfsmnt/file" while : do dd if=/dev/zero of="$SERVER_FILE$i" bs=1b count=1 &amp;&gt;/dev/null cat "$CLIENT_FILE$i" &amp;&gt;/dev/null echo "good" &gt; "$CLIENT_FILE$i" rm -fr "$CLIENT_FILE$i" now=$(date +%s) if (($now - $BEGIN_TIME &gt; $TIME_OUT)) then break fi let i=i+1 done echo "This probleam may be fix.\n" </pre> fanchaoting 2012-03-29T01:53:27 Rebélate by self-management, first project of free software by which we bet all / Rebélate por la autogestión, primer proyecto de software libre por el que apostamos todas http://comments.gmane.org/gmane.linux.file-systems.cachefs.general/3092 <pre>Inglés : Many already we have contributed to the first project of free software dedicated to self-management in this campaign of collective financing, it collaborates and it spreads!/ Beginning campaign collective financing http://www.goteo.org/project/rebelaos-publicacion-por-la-autogestion?lang=en Login to enter with user of social networks and for would register in Goteo : http://www.goteo.org/user/login?lang=en Rebelaos! Publication by self-management A massive publication that floods the public transport, the work centers, the parks, the consumption centers, by means of distribution of 500,000 gratuitous units, acting simultaneously in all sides and nowhere. We announce the main tool of a vestibule Web for the management of self-sustaining resources by means of Drupal, in addition in the publication there will be an article dedicated to free software, hardware, It is being prepared in inglès, the machinery You can see more details in the index of the publication https://n-1.cc/pg/file/read/1151902/indexresumen-de-los-contenidos-pdf . A computer system that allows us to share resources in all the scopes of our life so that we do not have to generate means different for each subject nor for each territory. A point of contact digitalis to generate projects of life outside Capitalism and to margin of the State. A tool to spread and to impel the social transformation through the resources that will set out in their contents around self-management, the autoorganización, the disobedience and the collective action. In which the capitalist system goes to the collapse, in a while immersed in a deep systemic crisis (ecological, political and economic, but mainly of values), where individual and collective of people they are being lacking of his fundamental rights, is necessary to develop a horizontal collective process where all the human beings we pruned to interact in equality of conditions and freedom. To interact means to relate to us (as much human as economically), to communicate to us, to cover our basic needs, to generate and to protect communal properties, to know and to provide collective solutions us problematic that our lives interfere. We want abrir a breach within normality in the monotonous life state-capitalist, a day anyone, that finally will not be any day. By means of this publication we try: - To drive a horizontal collective process where all and all we pruned to interact in equality of conditions and freedom. - To create communications network between the people it jeopardize with the change and arranged to act. - To find collective solutions to problematic that our lives interfere - To facilitate the access to resources that make possible self-management. - To participate in the construction of networks of mutual support, generated horizontals, asamblearias and from the base. - To publish all this information in an attractive format stops to facilitate the access to all the society. There are 15 days remaining for the upcoming March 15, the day that will come Rebelaos!, Magazine for the selfmanagement Today, we issue the cover of Rebelaos! (Castilian version) that can be displayed on the following link: https://n-1.cc/pg/file/read/1200503/portada-15-de-marzo-rebelaos The contents of the store owners to us by 15 March. Do you? Do you keep on 15 March? In addition, we have over 200 distribution nodes, distributed throughout the Spanish state. Check the map: https://afinidadrebelde.crowdmap.com/ On the other hand, the funding campaign continues to move and still have 12 days to collect the remaining 6,000 euros. We can all make a bit for all the grains of sand become a great beach on March 15. You can access the co-financing campaign: http://www.goteo.org/project/rebelaos-publicacion-por-la-autogestion Rebel Affinity group www.rebelaos.net ------------------------------------------------------------------------------- Castellano: Muchos ya hemos aportado al primer proyecto de software libre dedicado a la la financiación colectiva, colabora y diffunde !!!!! Inicio campaña financiación colectiva goteo.org www.goteo.org/project/rebelaos-publicacion-por-la-autogestion Link para registrarse en Goteo y acceder a redes sociales para colaborar en la difusín http://www.goteo.org/user/login ¡Rebelaos! Publicación por la autogestión Una publicación masiva que inunde el transporte público, los centros de trabajo, los parques, los centros de consumo, mediante la distribución de 500.000 ejemplares gratuitos, actuando simultáneamente en todos lados y en ninguna parte. Anunciamos la herramienta principal de un portal web para la gestión de recursos autogestionados mediante Drupal, además en la publicación habrá un artículo dedicado al software libre, el hardware, la maquinaria... Puedes ver más detalles en el índice de la publicación https://n-1.cc/pg/file/read/1151902/indexresumen-de-los-contenidos-pdf Un sistema infórmatico que nos permita compartir recursos en todos los ámbitos de nuestra vida de forma que no tengamos que generar un medio distinto para cada tema ni para cada territorio. Un punto de encuentro digital para generar proyectos de vida fuera del capitalismo y al margen del Estado. Una herramienta para difundir e impulsar la transformación social a través de los recursos que se propondrán en sus contenidos en torno a la autogestión, la autoorganización, la desobediencia y la acción colectiva. En un momento en que el sistema capitalista se dirige al colapso, inmerso en una profunda crisis sistémica (ecológica, política y económica, pero principalmente de valores), donde individuos y colectivos de personas están siendo desprovistos de sus derechos fundamentales, es necesario desarrollar un proceso colectivo horizontal donde todos los seres humanos podamos interactuar en igualdad de condiciones y en libertad. Interactuar significa relacionarnos (tanto humana como económicamente), comunicarnos, cubrir nuestras necesidades básicas, generar y proteger bienes comunes, conocernos y dar soluciones colectivas a problemáticas que interfieren nuestras vidas. Queremos abrir una brecha dentro de la normalidad en la monótona vida estatal-capitalista, un día cualquiera, que finalmente no será cualquier día. Mediante esta publicación pretendemos: - Impulsar un proceso colectivo horizontal donde todos y todas podamos interactuar en igualdad de condiciones y en libertad. - Crear red de comunicaciones entre las personas comprometidas con el cambio y dispuestas a actuar. - Encontrar soluciones colectivas a problemáticas que interfieren nuestras vidas. - Facilitar el acceso a recursos que posibiliten la autogestión. - Participar en la construcción de redes de apoyo mutuo, horizontales, asamblearias y generadas desde la base. - Publicar toda esta información en un formato atractivo para facilitar el acceso a toda la sociedad. Son 15 los días que restan para el próximo 15 de marzo, día en el que verá la luz ¡Rebelaos!, publicación por la autogestión. Hoy, hacemos pública la portada de ¡Rebelaos! (versión en castellano) que podéis visualizar en el siguiente enlace: https://n-1.cc/pg/file/read/1200503/portada-15-de-marzo-rebelaos El contenido de los titulares nos los guardamos para el 15 de marzo. ¿Y tú? ¿Te guardas el 15 de marzo? Además, ya hemos superado los 200 nodos de distribución, repartidos por todo el estado español. Ver el mapa: https://afinidadrebelde.crowdmap.com/ Por otro lado, la campaña de financiación continúa avanzando y todavía quedan 12 días para reunir los 6.000 euros que restan. Todas podemos aportar un poco para que todos los granitos de arena se conviertan en una gran playa el 15 de marzo. Podéis acceder a la campaña de cofinanciación en: http://www.goteo.org/project/rebelaos-publicacion-por-la-autogestion Colectivo Afinidad Rebelde www.rebelaos.net </pre> Orquidea Salt mas 2012-03-02T19:22:05 FS-Cache "Netfs" (nfs) Fedora 15/16/17 http://comments.gmane.org/gmane.linux.file-systems.cachefs.general/3090 <pre>Hi, Am trying to figure out why during bootup, at the following point across real and virt boxes. eb 19 11:22:07 frank01 kernel: [ 482.640618] FS-Cache: Loaded Feb 19 11:22:07 frank01 kernel: [ 482.672487] FS-Cache: Netfs 'nfs' registered for caching It sits there for about 10 seconds, with no indication as to what's happening. nfs mount during bootup can be autofs or fstab depending on box. Is this still relevant for debugging FS-Cache: https://www.redhat.com/archives/linux-cachefs/2010-February/msg00022.html " You can also turn on NFS debugging for FS-Cache to see what it's doing: echo $((0x800)) &gt;/proc/sys/sunrpc/nfs_debug " kernels across boxes: 3.2* 3.3* </pre> Frank Murphy 2012-02-21T17:41:39 hang on __fscache_wait_on_page_write and __nfs_fscache_invalidate_page http://comments.gmane.org/gmane.linux.file-systems.cachefs.general/3070 <pre>I thought I'd sent this in December, but I didn't see it in the archives, so I thought I'd take another crack at it. The problem seems to be spreading to clients mounting Linux server hosted NFS exports as well. Any help would be appreciated. Thanks, Brian ----- Forwarded message from Brian Kroth &lt;bpkroth&lt; at &gt;gmail.com&gt; ----- Date: Fri, 16 Dec 2011 10:36:45 -0600 From: Brian Kroth &lt;bpkroth&lt; at &gt;gmail.com&gt; To: linux-cachefs&lt; at &gt;redhat.com Subject: hang on __fscache_wait_on_page_write and __nfs_fscache_invalidate_page Reply-To: Brian Kroth &lt;bpkroth&lt; at &gt;gmail.com&gt; User-Agent: Mutt/1.5.20 (2009-06-14) X-Operating-System: Linux 2.6.32-bpo.3-amd64 x86_64 Hello all. I hope this is the right place for this question. If not please let me know where else it should go. I've got some machines that from time to time will hang waiting on an umount.nfs command and eventually spit out a call trace referencing __fscache_wait_on_page_write and __nfs_fscache_invalidate_page. The situation is as follows: Machines are configure to nfs automount some set of filesystems. Some are provided by an EMC Celerra NAS, others by Linux hosts, others by Solaris. Most of the mounts are configured to use the fsc option so that data which is read is cached for faster access the next time around. On one of these mounts I've seen on a couple client nodes about a half dozen times now, a situation where 1) The automounter will attempt to unmount a particular volume due to inactivity. I've used lsof to confirm that nothing was using the mount and this should in theory be a safe operation to do. 2) The umount.nfs process will block with the following call trace: [ 722.136057] INFO: task umount.nfs:926 blocked for more than 120 seconds. [ 722.136061] "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 722.136065] umount.nfs D ffff88012bb807a0 0 926 1 0x00000000 [ 722.136071] ffff88012bb807a0 0000000000000082 ffffea0002179998 ffff8800c15ad1c0 [ 722.136077] 0000000000013b40 ffff880016019fd8 ffff880016019fd8 0000000000013b40 [ 722.136083] ffff88012bb807a0 ffff880016018010 0000000000000246 0000000000000001 [ 722.136088] Call Trace: [ 722.136100] [&lt;ffffffffa0ef8c3a&gt;] ? __fscache_wait_on_page_write+0x83/0x9d [fscache] [ 722.136106] [&lt;ffffffff8105f3b1&gt;] ? wake_up_bit+0x20/0x20 [ 722.136120] [&lt;ffffffffa0f444c1&gt;] ? __nfs_fscache_invalidate_page+0x45/0x7a [nfs] [ 722.136126] [&lt;ffffffff810bf556&gt;] ? truncate_inode_page+0x45/0x78 [ 722.136130] [&lt;ffffffff810bfa53&gt;] ? truncate_inode_pages_range+0xca/0x2f2 [ 722.136140] [&lt;ffffffffa0f1f3bd&gt;] ? nfs_access_free_entry+0x15/0x1f [nfs] [ 722.136150] [&lt;ffffffffa0f21477&gt;] ? nfs_access_zap_cache+0x116/0x12a [nfs] [ 722.136161] [&lt;ffffffffa0f2380c&gt;] ? nfs_evict_inode+0x12/0x23 [nfs] [ 722.136166] [&lt;ffffffff8110e5b9&gt;] ? evict+0x79/0x116 [ 722.136169] [&lt;ffffffff8110e682&gt;] ? dispose_list+0x2c/0x36 [ 722.136173] [&lt;ffffffff8110e865&gt;] ? evict_inodes+0xc7/0xea [ 722.136179] [&lt;ffffffff810fd872&gt;] ? generic_shutdown_super+0x4e/0xd4 [ 722.136183] [&lt;ffffffff810fd960&gt;] ? kill_anon_super+0x9/0x40 [ 722.136194] [&lt;ffffffffa0f27996&gt;] ? nfs_kill_super+0x15/0x28 [nfs] [ 722.136199] [&lt;ffffffff810fd9d0&gt;] ? deactivate_locked_super+0x1e/0x44 [ 722.136203] [&lt;ffffffff81112c9c&gt;] ? sys_umount+0x2ea/0x315 [ 722.136207] [&lt;ffffffff81112039&gt;] ? mntput_no_expire+0x8f/0x14f [ 722.136212] [&lt;ffffffff81339392&gt;] ? system_call_fastpath+0x16/0x1b (surprisingly early after boot don't you think?) 3) At some point later, the volume will attempt to be accessed again, causing automount to spawn a mount process, which will hang since there's already a umount process running. Since the processes are blocked in the kernel. There's not much I can do to resolve the issue but kick the machine. The clients (~200 of them) are all running debian squeeze based machines with a 2.6.39 kernel (from debian-backports). They run cachefilesd 0.9-3 with the following conf: dir /var/cache/fscache tag mycache brun 15% bcull 12% bstop 10% frun 15% fcull 12% fstop 10% culltable 16 The backing store for the cache is mounted as follows: # mount | grep fscache /dev/mapper/vg-fscache on /var/cache/fscache type ext4 (rw,relatime,user_xattr) The automount that seems to be having the problems looks like this: cnerg-hard,vers=3,nosuid,fscfilespace:/groups_cnerg It's served by the EMC Celerra. However, none of the other 35 automounts that that server serves, nor any of the ones that are served by the Linux/Solaris hosts, seem to have exhibited this problem. Not necessarily a causal relationship, but suspicious. It might just be the way the group is using the volume, though I haven't been able to pinpoint anything along those lines either. More telling, I think, is that since I've removed the fsc mount option from that automount I haven't seen this problem again. However, I would like to keep that option on. The trouble in debugging is that I don't know how to reproduce the problem. If I did, I'd capture some NFS packet traces to verify nothing is happening there. Due to the call trace however, it seems to me like the fscache is failing for some reason. Which is what brings me here. So, now I'm wondering if anyone has an insight as to any known problems with a setup like this or other things I could look into? Let me know if you need any more details. I have a handful of ps, lsof, and dmesg dumps I can share. Thanks, Brian ----- End forwarded message ----- </pre> Brian Kroth 2012-01-24T17:06:17 Bad Page states, kernel oopses http://comments.gmane.org/gmane.linux.file-systems.cachefs.general/3069 <pre>Hi, We are getting a number of 'bad page state' and kernel oopses which (I suspect) are related to fscache or cachefilesd. At the time that these occur, the process that was running will go in to an uniterruptable state Sample bad page state: ----------------------------------------------------8&lt;---------------------------------------------------- Jan 22 19:10:30 rb031 kernel: BUG: Bad page state in process nuke pfn:9c509 Jan 22 19:10:30 rb031 kernel: page:ffffea0002714240 count:0 mapcount:0 mapping: (null) index:0x6e63 Jan 22 19:10:30 rb031 kernel: page flags: 0x4000000000001000(private_2) Jan 22 19:10:30 rb031 kernel: Pid: 26646, comm: nuke Tainted: G B D 3.1.7 #2 Jan 22 19:10:30 rb031 kernel: Call Trace: Jan 22 19:10:30 rb031 kernel: [&lt;ffffffff810a99c5&gt;] bad_page+0xe5/0xfb Jan 22 19:10:30 rb031 kernel: [&lt;ffffffff810aaf3c&gt;] get_page_from_freelist+0x2ff/0x433 Jan 22 19:10:30 rb031 kernel: [&lt;ffffffff810ab745&gt;] __alloc_pages_nodemask+0x32a/0x6ac Jan 22 19:10:30 rb031 kernel: [&lt;ffffffffa009caf3&gt;] cachefiles_read_backing_file+0x12d/0x4b0 [cachefiles] Jan 22 19:10:30 rb031 kernel: [&lt;ffffffffa009d108&gt;] cachefiles_read_or_alloc_pages+0x292/0x2d1 [cachefiles] Jan 22 19:10:30 rb031 kernel: [&lt;ffffffff81054da8&gt;] ? bit_waitqueue+0x17/0x97 Jan 22 19:10:30 rb031 kernel: [&lt;ffffffff81054e80&gt;] ? wake_up_bit+0x25/0x2a Jan 22 19:10:30 rb031 kernel: [&lt;ffffffffa007947e&gt;] ? fscache_run_op+0x2e/0x48 [fscache] Jan 22 19:10:30 rb031 kernel: [&lt;ffffffffa0079e7a&gt;] ? fscache_submit_op+0x202/0x422 [fscache] Jan 22 19:10:30 rb031 kernel: [&lt;ffffffffa007b450&gt;] __fscache_read_or_alloc_pages+0x21f/0x301 [fscache] Jan 22 19:10:30 rb031 kernel: [&lt;ffffffffa0224fc1&gt;] __nfs_readpages_from_fscache+0x80/0x14e [nfs] Jan 22 19:10:30 rb031 kernel: [&lt;ffffffff810ab745&gt;] ? __alloc_pages_nodemask+0x32a/0x6ac Jan 22 19:10:30 rb031 kernel: [&lt;ffffffffa0202593&gt;] nfs_readpages+0xf2/0x16f [nfs] Jan 22 19:10:30 rb031 kernel: [&lt;ffffffff810ad879&gt;] __do_page_cache_readahead+0x142/0x1c1 Jan 22 19:10:30 rb031 kernel: [&lt;ffffffff810ad919&gt;] ra_submit+0x21/0x25 Jan 22 19:10:30 rb031 kernel: [&lt;ffffffff810adc8b&gt;] ondemand_readahead+0x18e/0x1a1 Jan 22 19:10:30 rb031 kernel: [&lt;ffffffff81054e00&gt;] ? bit_waitqueue+0x6f/0x97 Jan 22 19:10:30 rb031 kernel: [&lt;ffffffff810add5d&gt;] page_cache_sync_readahead+0x3d/0x40 Jan 22 19:10:30 rb031 kernel: [&lt;ffffffff810a7574&gt;] generic_file_aio_read+0x299/0x611 Jan 22 19:10:30 rb031 kernel: [&lt;ffffffff8101b582&gt;] ? flat_send_IPI_mask+0x11/0x13 Jan 22 19:10:30 rb031 kernel: [&lt;ffffffffa01f835f&gt;] nfs_file_read+0xae/0xd3 [nfs] Jan 22 19:10:30 rb031 kernel: [&lt;ffffffff810e3bee&gt;] do_sync_read+0xcb/0x108 Jan 22 19:10:30 rb031 kernel: [&lt;ffffffff811b868d&gt;] ? fsnotify_perm+0x66/0x72 Jan 22 19:10:30 rb031 kernel: [&lt;ffffffff811b86fa&gt;] ? security_file_permission+0x2e/0x33 Jan 22 19:10:30 rb031 kernel: [&lt;ffffffff810e45b5&gt;] vfs_read+0xa7/0xe3 Jan 22 19:10:30 rb031 kernel: [&lt;ffffffff810e46b4&gt;] sys_read+0x4a/0x71 Jan 22 19:10:30 rb031 kernel: [&lt;ffffffff813ba7ab&gt;] system_call_fastpath+0x16/0x1b Jan 22 19:10:30 rb031 kernel: [&lt;ffffffff810ad7f8&gt;] __do_page_cache_readahead+0xc1/0x1c1 Jan 22 19:10:30 rb031 kernel: [&lt;ffffffff813b8867&gt;] ? schedule+0x5a/0x5c Jan 22 19:10:30 rb031 kernel: [&lt;ffffffff813b88bf&gt;] ? io_schedule+0x56/0x6b Jan 22 19:10:30 rb031 kernel: [&lt;ffffffff810ad919&gt;] ra_submit+0x21/0x25 Jan 22 19:10:30 rb031 kernel: [&lt;ffffffff810adc8b&gt;] ondemand_readahead+0x18e/0x1a1 Jan 22 19:10:30 rb031 kernel: [&lt;ffffffff810add17&gt;] page_cache_async_readahead+0x79/0x82 Jan 22 19:10:30 rb031 kernel: [&lt;ffffffff810a5c24&gt;] ? find_get_page+0x48/0x6a Jan 22 19:10:30 rb031 kernel: [&lt;ffffffff810a75b4&gt;] generic_file_aio_read+0x2d9/0x611 Jan 22 19:10:30 rb031 kernel: [&lt;ffffffffa01f835f&gt;] nfs_file_read+0xae/0xd3 [nfs] Jan 22 19:10:30 rb031 kernel: [&lt;ffffffff810e3bee&gt;] do_sync_read+0xcb/0x108 Jan 22 19:10:30 rb031 kernel: [&lt;ffffffff811b868d&gt;] ? fsnotify_perm+0x66/0x72 Jan 22 19:10:30 rb031 kernel: [&lt;ffffffff811b86fa&gt;] ? security_file_permission+0x2e/0x33 Jan 22 19:10:30 rb031 kernel: [&lt;ffffffff810e45b5&gt;] vfs_read+0xa7/0xe3 Jan 22 19:10:30 rb031 kernel: [&lt;ffffffff810e46b4&gt;] sys_read+0x4a/0x71 Jan 22 19:10:30 rb031 kernel: [&lt;ffffffff813ba7ab&gt;] system_call_fastpath+0x16/0x1b Jan 22 19:10:52 rb031 abrt: Kerneloops: Reported 12 kernel oopses to Abrt sample Kernel oops: ----------------------------------------------------8&lt;---------------------------------------------------- Jan 24 12:55:12 ra020 kernel: ------------[ cut here ]------------ Jan 24 12:55:12 ra020 kernel: kernel BUG at fs/nfs/fscache.c:362! Jan 24 12:55:12 ra020 kernel: invalid opcode: 0000 [#3] SMP Jan 24 12:55:12 ra020 kernel: last sysfs file: /sys/devices/system/cpu/cpu7/cache/index2/shared_cpu_map Jan 24 12:55:12 ra020 kernel: CPU 4 Jan 24 12:55:12 ra020 kernel: Modules linked in: ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat bridge stp llc autofs4 nfs lockd nfs_acl auth_rpcgss sunrpc cachefiles fscache ipv6 kvm uinput serio_raw bnx2 iTCO_wdt iTCO_vendor_support dcdbas microcode power_meter [last unloaded: speedstep_lib] Jan 24 12:55:12 ra020 kernel: Jan 24 12:55:12 ra020 kernel: Pid: 81, comm: kswapd0 Tainted: G D 2.6.34.9-69.fc13.x86_64 #1 01V648/PowerEdge R410 Jan 24 12:55:12 ra020 kernel: RIP: 0010:[&lt;ffffffffa01a2185&gt;] [&lt;ffffffffa01a2185&gt;] nfs_fscache_release_page+0x3c/0xa6 [nfs] Jan 24 12:55:12 ra020 kernel: RSP: 0018:ffff88042607f990 EFLAGS: 00010246 Jan 24 12:55:12 ra020 kernel: RAX: ffff88041e8aea88 RBX: ffffea000df278f8 RCX: ffff88041e8ae8d0 Jan 24 12:55:12 ra020 kernel: RDX: 0040000000001009 RSI: 00000000000000d0 RDI: ffffea000df278f8 Jan 24 12:55:12 ra020 kernel: RBP: ffff88042607f9b0 R08: ffffea000df27920 R09: 00000000ffffffc8 Jan 24 12:55:12 ra020 kernel: R10: ffff88042607faf0 R11: ffffea000a23b540 R12: 0000000000000000 Jan 24 12:55:12 ra020 kernel: R13: 00000000000000d0 R14: ffff88042607fc30 R15: 0000000000000001 Jan 24 12:55:12 ra020 kernel: FS: 0000000000000000(0000) GS:ffff880237440000(0000) knlGS:0000000000000000 Jan 24 12:55:12 ra020 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Jan 24 12:55:12 ra020 kernel: CR2: 00007fe1d3b2e000 CR3: 0000000001a42000 CR4: 00000000000006e0 Jan 24 12:55:12 ra020 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jan 24 12:55:12 ra020 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Jan 24 12:55:12 ra020 kernel: Process kswapd0 (pid: 81, threadinfo ffff88042607e000, task ffff880426928000) Jan 24 12:55:12 ra020 kernel: Stack: Jan 24 12:55:12 ra020 kernel: 00000000000000d0 ffffea000df278f8 00000000000000d0 ffff88041e8aeba0 Jan 24 12:55:12 ra020 kernel: &lt;0&gt; ffff88042607f9e0 ffffffffa017a1dc 000000000a23b540 ffffea000df27920 Jan 24 12:55:12 ra020 kernel: &lt;0&gt; ffffea000df278f8 ffff88042607fda0 ffff88042607f9f0 ffffffff810ca39b Jan 24 12:55:12 ra020 kernel: Call Trace: Jan 24 12:55:12 ra020 kernel: [&lt;ffffffffa017a1dc&gt;] nfs_release_page+0x7d/0x85 [nfs] Jan 24 12:55:12 ra020 kernel: [&lt;ffffffff810ca39b&gt;] try_to_release_page+0x34/0x3d Jan 24 12:55:12 ra020 kernel: [&lt;ffffffff810d6287&gt;] shrink_page_list+0x300/0x477 Jan 24 12:55:12 ra020 kernel: [&lt;ffffffff81107e93&gt;] ? mem_cgroup_del_lru+0x39/0x3b Jan 24 12:55:12 ra020 kernel: [&lt;ffffffff810d531c&gt;] ? isolate_pages_global+0xb7/0x1f7 Jan 24 12:55:12 ra020 kernel: [&lt;ffffffff810d6790&gt;] shrink_inactive_list+0x392/0x68e Jan 24 12:55:12 ra020 kernel: [&lt;ffffffff810d6de9&gt;] shrink_zone+0x35d/0x411 Jan 24 12:55:12 ra020 kernel: [&lt;ffffffff810d7b6b&gt;] balance_pgdat+0x337/0x5b6 Jan 24 12:55:12 ra020 kernel: [&lt;ffffffff810d5265&gt;] ? isolate_pages_global+0x0/0x1f7 Jan 24 12:55:12 ra020 kernel: [&lt;ffffffff810d8025&gt;] kswapd+0x23b/0x266 Jan 24 12:55:12 ra020 kernel: [&lt;ffffffff810662ff&gt;] ? autoremove_wake_function+0x0/0x39 Jan 24 12:55:12 ra020 kernel: [&lt;ffffffff810d7dea&gt;] ? kswapd+0x0/0x266 Jan 24 12:55:12 ra020 kernel: [&lt;ffffffff81065e85&gt;] kthread+0x7f/0x87 Jan 24 12:55:12 ra020 kernel: [&lt;ffffffff8100aa64&gt;] kernel_thread_helper+0x4/0x10 Jan 24 12:55:12 ra020 kernel: [&lt;ffffffff81065e06&gt;] ? kthread+0x0/0x87 Jan 24 12:55:12 ra020 kernel: [&lt;ffffffff8100aa60&gt;] ? kernel_thread_helper+0x0/0x10 Jan 24 12:55:12 ra020 kernel: Code: 00 48 8b 17 b8 01 00 00 00 48 89 fb 41 89 f5 80 e6 10 74 79 48 8b 47 18 48 8b 00 4c 8b 60 f8 48 8d 88 48 fe ff ff 4d 85 e4 75 04 &lt;0f&gt; 0b eb fe f6 05 3d 29 f8 ff 08 74 14 48 89 fa 4c 89 e6 48 c7 Jan 24 12:55:12 ra020 kernel: RIP [&lt;ffffffffa01a2185&gt;] nfs_fscache_release_page+0x3c/0xa6 [nfs] Jan 24 12:55:12 ra020 kernel: RSP &lt;ffff88042607f990&gt; Jan 24 12:55:12 ra020 kernel: ---[ end trace 37b1b21bd4ae7177 ]--- Jan 24 12:56:18 ra020 abrt: Kerneloops: Reported 1 kernel oopses to Abrt ----------------------------------------------------8&lt;---------------------------------------------------- We also get unreadable PIDs for processes which appear to be related to these bad page states, i.e., trying to run a 'ps', 'w' or anything else that needs to read the PID's directory of the process that has a bad page state in /proc hangs (is this a result of page state corruption?) A bit about our environment: The NFS server is an Isilon running OneFS. We have two clients accessing the NFS server, both running FS-Cache. One set of clients submits graphics rendering jobs on an NFS filesystem that are rendered by the other clients (dedicated render boxes). All clients are running Fedora 13 x86_64 FS-cache + cachefiles (cachefilesd-0.10.1-1.fc13.x86_64) over NFS on Fedora 13 x86_64 (most run kernel 2.6.34.9-69.fc13.x86_64, although quite a few run other variants of 2.6.34 such as 2.6.34.7-56.fc13.x86_64, we have also tried a backported 2.6.35 fedora 14 kernel, and 2.6.39 kernel, as well as a vanilla 3.1.7 kernel, with no decrease in errors). We have also tried switching off cachefilesd user space process, but that does not stop the bad page states or oopses. After a while, maybe due to page state problems, the load begins to grow due to uninterruptible processes and unreadable PIDs and the boxes need to be rebooted. Any idea what is going wrong here? Any help greatly appreciated! Kind regards, Campbell --------------------------------------------------------------------- PS: probably unrelated, but we also get a lot of these errors (sometimes these appear, sometimes not): Jan 22 03:51:41 ra042 kernel: CacheFiles: Error: Overlong wait for old active object to go away Jan 22 03:51:41 ra042 kernel: object: OBJ640b Jan 22 03:51:41 ra042 kernel: objstate=OBJECT_LOOKING_UP fl=0 swfl=a ev=0[7b] Jan 22 03:51:41 ra042 kernel: ops=0 inp=0 exc=0 Jan 22 03:51:41 ra042 kernel: parent=ffff880420814900 Jan 22 03:51:41 ra042 kernel: cookie=ffff8802252110a0 [pr=ffff88042508b230 nd=ffff88014d7e2850 fl=7] Jan 22 03:51:41 ra042 kernel: key=[32] '60a6fe830c0000001862a15901000000ffffffff000000000c00420001000000' Jan 22 03:51:41 ra042 kernel: xobject: OBJ640a Jan 22 03:51:41 ra042 kernel: xobjstate=OBJECT_DYING fl=0 swfl=8 ev=20[5] Jan 22 03:51:41 ra042 kernel: xops=1 inp=1 exc=0 Jan 22 03:51:41 ra042 kernel: xparent=ffff880420814900 Jan 22 03:51:41 ra042 kernel: xcookie=NULL Jan 22 03:52:41 ra042 kernel: </pre> Cam Mac 2012-01-24T16:19:58 Problem reading cached files http://comments.gmane.org/gmane.linux.file-systems.cachefs.general/3065 <pre>Hello, I have been trying to get local caching to work on a host running SuSE SLES11 SP1 (cachefilesd is still unsupported). I had no problems compiling and inserting a cachefiles module in the kernel or starting the cachefsd daemon (cachefilesd-0.10.1.tar.bz2). I also see files being cached to /var/cache/fscache (ext3 partition mounted with user_xattr) Nov 14 19:38:17 cachefilesd[6586]: About to bind cache Nov 14 19:38:17 cachefilesd[6586]: Bound cache Nov 14 19:38:17 cachefilesd[6589]: Daemon Started Nov 14 19:38:17 kernel: [ 9667.331384] FS-Cache: Cache "mycache" added (type cachefiles) Nov 14 19:38:17 kernel: [ 9667.331394] CacheFiles: File cache on cciss/c0d0p11 registered Nov 14 19:38:17 cachefilesd[6589]: Scan complete Nov 14 19:38:17 cachefilesd[6589]: Decant (all 213) However when i try to cat a previously cached file ( after flushing the kernel page cache using echo 3 &gt;/proc/sys/vm/drop_caches) the process hangs indefinitely. . It shows up with a status sync_p in process listing and a trace on the PID shows that the process is stuck on a read for device /dev/cachefiles. I earlier though this might be because of the SELinux function but it appears that SELinux functionality is not enabled on this host. Any help with debugging the hang issue is gretaly appreciated. I </pre> Raghuram Bondalapati 2011-11-16T04:00:47 Possible page cache related bug http://comments.gmane.org/gmane.linux.file-systems.cachefs.general/3064 <pre>Hi, I have encountered an issue when using fscache where the Linux page cache consumes memory on the system that then cannot be cleared. This eventually leaves the system in a crippled swapping state. To outline what I am doing: [With a data set of ~35GB in ~2000 files] 1. Start four processes that each read the entire contents of a file then move to the next etc. 2. After the first read from NFS we are constantly reading ~250MB/s from the cache. 3. Very high system CPU utilisation is observed on (kswapd0 &amp; flush-0). 4. After ~30min we find that part of the page cache can't be cleared (by echo 3 &gt; /proc/sys/vm/drop_caches) 5. Eventually all RAM becomes consumed and the system starts swapping heavily. 6. The four reading processes are terminated. 7. The page cache still will not clear, even after unmounting the NFS filesystem and restarting cachefilesd. Many messages of this form are seen in /var/log/messages, but not sure if this is connected: FS-Cache: Assertion failed 1 == 0 is false FS-Cache: Assertion failed 0 &gt;= 1 is false When performing the same test without fscache this does not occur, however the test without fscache never read anything like the quantity of data from the real NFS server as from the cache when testing with fscache. Sorry I don't have much data to provide, but I hope this will still be of some use. thanks, Stef Kernel: 2.6.39.4 (x86_64) RAM: 4GB CPUs: 4 cachefilesd: bdec8cc (Bump the revision to 0.10.2) Kernel / fscache: 4dc2606 3.0.3 64-bit Crash running fscache/cachefilesd 3b27dea NFS: Use FS-Cache invalidation 7438164 CacheFiles: Implement invalidation 9f27574 VFS: Make more complete truncate operation available to CacheFiles 7b4919b FS-Cache: Provide proper invalidation 5dacc0f FS-Cache: Fix operation state management and accounting b83629c fscache: remove dead code under CONFIG_WORKQUEUE_DEBUGFS e3249f6 FS-Cache: Make cookie relinquishment wait for outstanding reads 0e2f15f CacheFiles: Make some debugging statements conditional c2934b8 FS-Cache: Check cookie is still correct in __fscache_read_or_alloc_pages() 94ed160 FS-Cache: Check that there are no read ops when cookie relinquished 7f29d79 CacheFiles: Downgrade the requirements passed to the allocator f6fb3c4 FS-Cache: Validate page mapping pointer value 1999e47 CacheFiles: Fix the marking of cached pages c540312 Noisefs: A predictable noise producing fs for testing things e0b2ac6 ALSA: snd_usb_caiaq: track submitted output urbs fa01632 ALSA: snd-usb-caiaq: Correct offset fields of outbound iso_frame_desc 78e0e10 ALSA: snd-usb-caiaq: Fix keymap for RigKontrol3 ea0dc0d Linux 2.6.39.4 </pre> Stefan Blanke 2011-11-04T16:10:59 NFS performance not as expected with FSCache http://comments.gmane.org/gmane.linux.file-systems.cachefs.general/3063 <pre>We have had a very heavily loaded web server which delivers mp3 files through apache with the files residing on an NFS file system. The server receives around 4-10 requests a second, and the files are each around 600KB in size. The server delivers a throughput of between 20Mb/s - 60Mb/s depending on the time of day. The NFS file system being used is read only. The load on this server was increasing to over 100 at busy times during the day, I thought, due to high wait times from the NFS server As the load was so high, the server was becoming unresponsive so we have implemented fscache with local ssd drives to alleviate this on a new server. The fs cache implementation has been somewhat successful, load figures on the new server are now down in the rang of 0-30 which is much better, and looking at throughput on the network and the fscache stats, I can see that around 75% of requests are now satisfied by fscache. See fscache stats below, which I hope I am interpreting correctly. Despite this, files which are definitely cached by fscache still can take over 10 seconds to deliver to our monitoring server connected via a 100Mbit switch, but sometime take less than a second, and I am not sure why. Our monitoring server every minute asks for the same file to be delivered to it, hence I know its cached by fscache. I have spent a lot of time checking out articles on NFS client performance and have made a number of changes to kernel networking and sunrpc parameters (see below also) in an attempt to resolve the issue which have definitely helped. Can anyone shed any light on why a file cached by fscache might take so long to be delivered. A limitation of nfs perhaps or a problem with fscache? I am not sure where best to start to debug the problem. Regards Ben FS-Cache statistics Cookies: idx=6 dat=145221 spc=0 Objects: alc=145226 nal=0 avl=145226 ded=143437 ChkAux : non=0 ok=102608 upd=0 obs=2 Pages : mrk=31558074 unc=31364899 Acquire: n=145227 nul=0 noc=0 ok=145227 nbf=0 oom=0 Lookups: n=145226 neg=42614 pos=102612 crt=42614 tmo=0 Updates: n=0 nul=0 run=0 Relinqs: n=143439 nul=0 wcr=0 rtr=0 AttrChg: n=0 ok=0 nbf=0 oom=0 run=0 Allocs : n=0 ok=0 wt=0 nbf=0 int=0 Allocs : ops=0 owt=0 abt=0 Retrvls: n=200055 ok=157332 wt=28965 nod=42723 nbf=0 int=0 oom=0 Retrvls: ops=200055 owt=13453 abt=0 Stores : n=7141275 ok=7141275 agn=0 nbf=0 oom=0 Stores : ops=903546 run=8044821 pgs=7141275 rxd=7141275 olm=0 VmScan : nos=31171518 gon=0 bsy=0 can=0 Ops : pend=13454 run=1103601 enq=39680550 can=0 rej=0 Ops : dfr=242 rel=1103601 gc=242 CacheOp: alo=0 luo=0 luc=0 gro=0 CacheOp: upo=0 dro=0 pto=0 atc=0 syn=0 CacheOp: rap=0 ras=0 alp=0 als=0 wrp=0 ucp=0 dsp=0 Mountstats: Stats for 10.0.20.192:/live_clips mounted on /mnt/clips: NFS mount options: ro,vers=3,rsize=32768,wsize=32768,namlen=255,acregmin=3,acregmax=60,acdirmin =30,acdirmax=60,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.0.20 .192,mountvers=3,mountport=1234,mountproto=tcp,fsc,local_lock=none NFS server capabilities: caps=0x3fc7,wtmult=512,dtsize=8192,bsize=0,namlen=255 NFS security flavor: 1 pseudoflavor: 0 NFS byte counts: applications read 2517191085 bytes via read(2) applications wrote 0 bytes via write(2) applications read 0 bytes via O_DIRECT read(2) applications wrote 0 bytes via O_DIRECT write(2) client read 28169649739 bytes via NFS READ client wrote 0 bytes via NFS WRITE RPC statistics: 1795475 RPC requests sent, 1795474 RPC replies received (0 XIDs not found) average backlog queue length: 0 GETATTR: 582165 ops (32%) 4 retrans (0%) 0 major timeouts avg bytes sent per op: 124 avg bytes received per op: 112 backlog wait: 0.006519 RTT: 0.351234 total execute time: 0.378106 (milliseconds) LOOKUP: 154984 ops (8%) 0 retrans (0%) 0 major timeouts avg bytes sent per op: 151 avg bytes received per op: 238 backlog wait: 0.006775 RTT: 262.135511 total execute time: 262.158558 (milliseconds) ACCESS: 176336 ops (9%) 1 retrans (0%) 0 major timeouts avg bytes sent per op: 128 avg bytes received per op: 120 backlog wait: 0.004304 RTT: 1.463734 total execute time: 1.482204 (milliseconds) READ: 881948 ops (49%) 0 retrans (0%) 0 major timeouts avg bytes sent per op: 136 avg bytes received per op: 32068 backlog wait: 0.175209 RTT: 40.025127 total execute time: 40.232458 (milliseconds) FSINFO: 2 ops (0%) 0 retrans (0%) 0 major timeouts avg bytes sent per op: 136 avg bytes received per op: 164 backlog wait: 0.000000 RTT: 0.000000 total execute time: 0.000000 (milliseconds) PATHCONF: 1 ops (0%) 0 retrans (0%) 0 major timeouts avg bytes sent per op: 136 avg bytes received per op: 140 backlog wait: 0.000000 RTT: 0.000000 total execute time: 0.000000 (milliseconds) [root&lt; at &gt;jrclips ~]# yum list cachefilesd Installed Packages cachefilesd.x86_64 0.10.1-2.el6 [root&lt; at &gt;jrclips ~]# uname -a Linux jrclips 2.6.32-131.17.1.el6.x86_64 #1 SMP Wed Oct 5 17:19:54 CDT 2011 x86_64 x86_64 x86_64 GNU/Linux Changes to sysctl.conf #Mkae sure this is used before any nfs file systems are mounted sunrpc.tcp_slot_table_entries = 128 # These ensure that TIME_WAIT ports either get reused or closed fast. net.ipv4.tcp_fin_timeout = 1 net.ipv4.tcp_tw_recycle = 1 # TCP memory net.core.rmem_max = 16777216 net.core.rmem_default = 16777216 net.core.wmem_max = 16777216 net.core.wmem_default = 16777216 net.core.netdev_max_backlog = 262144 net.core.somaxconn = 262144 net.ipv4.tcp_syncookies = 1 net.ipv4.tcp_max_orphans = 262144 net.ipv4.tcp_max_syn_backlog = 262144 net.ipv4.tcp_synack_retries = 2 net.ipv4.tcp_syn_retries = 2 net.ipv4.tcp_rmem = 4096 262144 16777216 net.ipv4.tcp_wmem = 4096 262144 16777216 [root&lt; at &gt;jrclips ~]# free total used free shared buffers cached Mem: 2053992 1989756 64236 0 124620 1677612 -/+ buffers/cache: 187524 1866468 Swap: 4128760 0 4128760 </pre> Ben Yarwood 2011-10-31T12:02:47 cachefilsd continuously logging: Scan complete http://comments.gmane.org/gmane.linux.file-systems.cachefs.general/3034 <pre>Every two seconds, cachefilesd logs a Can complete message. Oct 7 06:23:15 node010 cachefilesd[6297]: Scan complete Oct 7 06:23:17 node010 cachefilesd[6297]: Scan complete Oct 7 06:23:19 node010 cachefilesd[6297]: Scan complete Oct 7 06:23:21 node010 cachefilesd[6297]: Scan complete Oct 7 06:23:23 node010 cachefilesd[6297]: Scan complete Oct 7 06:23:25 node010 cachefilesd[6297]: Scan complete Oct 7 06:23:27 node010 cachefilesd[6297]: Scan complete Oct 7 06:23:29 node010 cachefilesd[6297]: Scan complete Why is it looping so fast? /etc/cachefilesd.conf: dir /tmp/fscache tag mycache brun 50% bcull 40% bstop 30% frun 50% fcull 40% fstop 30% # mount /dev/sda1 on / type ext4 (rw,noatime) tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755) proc on /proc type proc (rw,noexec,nosuid,nodev) sysfs on /sys type sysfs (rw,noexec,nosuid,nodev) udev on /dev type tmpfs (rw,mode=0755) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620) /dev/sda2 on /var type ext4 (rw,noatime) /dev/sda4 on /tmp type ext4 (rw,noatime,user_xattr) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) master.cluster:/home on /home type nfs4 (rw,remount,noatime,fsc,addr=192.168.100.254,clientaddr=192.168.100.10) master.cluster:/shared on /cluster/shared type nfs4 (rw,fsc,addr=192.168.100.254,clientaddr=192.168.100.10) # df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 9.2G 1.5G 7.7G 16% / tmpfs 4.0G 0 4.0G 0% /lib/init/rw udev 4.0G 96K 4.0G 1% /dev tmpfs 4.0G 4.0K 4.0G 1% /dev/shm /dev/sda2 4.6G 1.2G 3.4G 26% /var /dev/sda4 123G 188M 122G 1% /tmp master.cluster:/home 7.0T 4.3T 2.4T 65% /home master.cluster:/shared 45G 3.1G 40G 8% /cluster/shared I'm using cachefilesd 0.9 on Debian </pre> Frederik Himpe 2011-10-07T10:00:34 Page cache ignored on second read http://comments.gmane.org/gmane.linux.file-systems.cachefs.general/3028 <pre>Hi David, I tested your recent patches, with much better results. Many thanks for these. I didn't do formal testing, but I've not yet seen the "Overlong wait" message in an hour or so of use, which is excellent. But now that I can reasonably use fscache with NFS, I noticed that the page cache doesn't seem to be used? * When cachefilesd is running, reads do not come from the page cache * The page cache fills up with something, but the local cache filesystem continues to be read * If the page cache was populated before cachefilesd was launched (ie. with NFS pages), then these are successfully used I tested using a set of large image files which fit in RAM. In the first cases, I'm not sure if the page cache memory contains NFS pages, or local disk pages. Is there an easy way to check? This test is on v2.6.39.4, with the recent patches applied. I didn't need to modify any code, as long as I included one other fscache patch since that version. It seems that v3.0 onward does not boot on my hardware, which is something I need to look into separately. Thanks </pre> Mark Hills 2011-10-03T14:53:01 [RFC][PATCH 00/13] Fix FS-Cache problems http://comments.gmane.org/gmane.linux.file-systems.cachefs.general/3003 <pre> Here are some patches to debug and fix FS-Cache problems. (1) A debugging patch to add a predictable pattern generator with a filesystem interface. Can be used to generate regular apparent changes on the NFS server. This was very useful in breaking FS-Cache. (2) A patch to correctly mark cached netfs pages. (3) A debugging patch to validate page-&gt;mapping on a page for which retrieval was requested. (4) A patch to downgrade memory allocation levels in the cache to not try so hard and to be more willing to abort with ENOMEM. It's a cache - it doesn't matter if we can't store something. (5) A debugging patch to check that there are no read operations outstanding on a cookie when it is relinquished. (6) A debugging patch to check that the object cookie pointer doesn't get cleared whilst we're trying to read for it. (7) A patch to make conditional some debugging prints. (8) A patch to make cookie relinquishment log a warning and wait for any outstanding reads. (9) A patch to fix up operation state handling and accounting. (10) A patch to provide proper invalidation facilities. (11) A patch to make a vfs_truncate() for cachefiles's invalidation to call. Question: CacheFiles uses truncation in a couple of places. It has been using notify_change() rather than sys_truncate() or something similar. This means it bypasses a bunch of checks and suchlike that it possibly should be making (security, file locking, lease breaking, vfsmount write). Should it be using vfs_truncate() as added by a preceding patch or should it use notify_write() and assume that anyone poking around in the cache files on disk gets everything they deserve? (12) A patch to provide invalidation for cachefiles. (13) A patch to make NFS use the invalidation call. David --- David Howells (13): NFS: Use FS-Cache invalidation CacheFiles: Implement invalidation VFS: Make more complete truncate operation available to CacheFiles FS-Cache: Provide proper invalidation FS-Cache: Fix operation state management and accounting FS-Cache: Make cookie relinquishment wait for outstanding reads CacheFiles: Make some debugging statements conditional FS-Cache: Check cookie is still correct in __fscache_read_or_alloc_pages() FS-Cache: Check that there are no read ops when cookie relinquished CacheFiles: Downgrade the requirements passed to the allocator FS-Cache: Validate page mapping pointer value CacheFiles: Fix the marking of cached pages Noisefs: A predictable noise producing fs for testing things Documentation/filesystems/caching/backend-api.txt | 38 ++ Documentation/filesystems/caching/netfs-api.txt | 46 ++- Documentation/filesystems/caching/object.txt | 23 + Documentation/filesystems/caching/operations.txt | 2 fs/Kconfig | 1 fs/Makefile | 1 fs/cachefiles/interface.c | 57 +++- fs/cachefiles/internal.h | 2 fs/cachefiles/key.c | 2 fs/cachefiles/rdwr.c | 118 ++++--- fs/cachefiles/xattr.c | 2 fs/fscache/cookie.c | 78 +++++ fs/fscache/internal.h | 10 + fs/fscache/object.c | 74 +++++ fs/fscache/operation.c | 135 ++++++--- fs/fscache/page.c | 188 ++++++++++-- fs/fscache/stats.c | 11 + fs/nfs/fscache.h | 20 + fs/nfs/inode.c | 20 + fs/nfs/nfs4proc.c | 2 fs/noisefs/Kconfig | 20 + fs/noisefs/Makefile | 7 fs/noisefs/file.c | 331 +++++++++++++++++++++ fs/noisefs/inode.c | 171 +++++++++++ fs/noisefs/internal.h | 59 ++++ fs/noisefs/super.c | 302 +++++++++++++++++++ fs/open.c | 50 ++- include/linux/fs.h | 1 include/linux/fscache-cache.h | 53 +++ include/linux/fscache.h | 50 +++ mm/page-writeback.c | 1 31 files changed, 1696 insertions(+), 179 deletions(-) create mode 100644 fs/noisefs/Kconfig create mode 100644 fs/noisefs/Makefile create mode 100644 fs/noisefs/file.c create mode 100644 fs/noisefs/inode.c create mode 100644 fs/noisefs/internal.h create mode 100644 fs/noisefs/super.c </pre> David Howells 2011-09-29T14:45:36 question about write-back / write-behind http://comments.gmane.org/gmane.linux.file-systems.cachefs.general/2988 <pre>Hello to all, I am looking for a possibility to cache reads and writes from and to a nfs mount. Read requests will be cached, that works fine, but I also want to cache writes. The situation is the following: I've got some people working in office A and some in office B. The offices are combined by glusterfs. Write-back or write-behind caching doesn't work in glusterfs, so I am looking for a workaround. The method of storing cached files locally on hard disk is what I'm looking for. So the client applications gets an "finished" and the cachefs / glusterfs is syncing the content in background. Is there a way to use cachefs as a write-back / write-behind cache? thanks a lot, Christian </pre> Christian 2011-08-30T13:19:58 CacheFiles: Error: Overlong wait for old activeobject to go away http://comments.gmane.org/gmane.linux.file-systems.cachefs.general/2984 <pre>Hello. I have same problem as Mark Hills http://www.spinics.net/lists/linux-cachefs/msg02369.html [416779.040059] CacheFiles: Error: Overlong wait for old active object to go away [416779.040104] object: OBJ43981 [416779.040127] objstate=OBJECT_LOOKING_UP fl=0 wbusy=2 ev=0[7b] [416779.040154] ops=0 inp=0 exc=0 [416779.040174] parent=ffff8800cca3c3c0 [416779.040197] cookie=ffff880100029870 [pr=ffff8801131520f0 nd=ffff88000662b180 fl=7] [416779.040241] key=[36] '010007010100a00800000000e6c715cb23cd4aa38338eea581f3d1045c2e680a1b56d03d' [416779.040325] xobject: OBJ431fe [416779.040348] xobjstate=OBJECT_RECYCLING fl=0 wbusy=2 ev=20[3] [416779.040373] xops=0 inp=0 exc=0 [416779.040394] xparent=ffff8800cca3c3c0 [416779.040416] xcookie=NULL kernel: 2.6.38-8-server from natty Sometimes processes hang, sometimes even kernel panic. Googling gave no answer. Could you give at least some clues how to debug it? </pre> Дмитрий Ильин 2011-08-23T07:30:38 Overlong wait for old active object to go away http://comments.gmane.org/gmane.linux.file-systems.cachefs.general/2983 <pre>I find CacheFiles gives a valuable performance benefit, but unfortunately is not usable due to hangs. In dmesg "Overlong wait for old active object to go away" (see full message below), and an application will hang or lock, eg. blocking on read(). Sometimes it unblocks, sometimes it seems to hang indefinitely. Is there a higher level explanation for what is going on here? It seems logical that the message is a symptom, rather than the cause itself. What could hold the lock in such a way? The timeout in question appears to be "60 * HZ" in fs/cachefiles/namei.c. I am using the default configuration of cachefilesd, from Git, and kernel 2.6.39.4. Many thanks, </pre> Mark Hills 2011-08-16T15:33:12 Limiting case of a big cache http://comments.gmane.org/gmane.linux.file-systems.cachefs.general/2982 <pre>Short version: What happens with FS-Cache/cachefilesd as the cache grows to the size of the NFS filesystem it's caching? Where I work we've used a network filesystem to achieve Single System Image for years (actually decades) now. It's been nice, but in the past few years it seems that network speeds haven't kept up. Add to this the fact that disk space is pretty darned cheap, and it makes me wonder about a slightly different operating regime for a network filesystem. We're not using NFS/FS-Cache at work, but I do for my home cluster, prompting this question. I suspect it's a generally applicable question, as well. I'd like to just have enough disk space on my user-facing system(s) to hold all of their data. In this situation, the network filesystem becomes a real-time backup, a way to propagate data to different systems, a way to keep those systems in sync, and since I mentioned backup, a central point to run some sort of "time-machine"-like backup of old files. Are you aware of limitations with the existing FS-Cache/cachefilesd work in this type of very-large-cache situation? Do writes to a cached filesystem write to the cache as well as to the NFS server? Does the write operation release when the NFS client has the data, or after the NFS server tells the client that it has it? Does the validation time for cached data start to dwarf any cache improvements as cache size increases? On the side, what's the status of the in-kernel AFS? Last I knew it was don't-use, just use OpenAFS, but I thought I heard about GSoC work being done with the in-kernel module. Thanks, Dale Pontius </pre> Dale Pontius 2011-08-16T15:24:18 CacheFS over NFS http://comments.gmane.org/gmane.linux.file-systems.cachefs.general/2981 <pre>Hi, I am trying out CacheFS over NFS in RHEL 6.1. If you have any experiences in RHEL6.1 please do share. </pre> Janardhan Molumuri 2011-08-02T01:16:32 [PATCH] FS-Cache: Fix__fscache_uncache_all_inode_pages()'s outer loop http://comments.gmane.org/gmane.linux.file-systems.cachefs.general/2980 <pre>From: Jan Beulich &lt;JBeulich&lt; at &gt;novell.com&gt; The compiler, at least for ix86 and m68k, validly warns that the comparison: next &lt;= (loff_t)-1 is always true (and it's always true also for x86-64 and probably all other arches - as long as pgoff_t isn't wider than loff_t). The intention appears to be to avoid wrapping of "next", so rather than eliminating the pointless comparison, fix the loop to indeed get exited when "next" would otherwise wrap. On m68k the following warning is observed: fs/fscache/page.c: In function '__fscache_uncache_all_inode_pages': fs/fscache/page.c:979: warning: comparison is always false due to limited range of data type Reported-by: Geert Uytterhoeven &lt;geert&lt; at &gt;linux-m68k.org&gt; Reported-by: Jan Beulich &lt;jbeulich&lt; at &gt;novell.com&gt; Signed-off-by: Jan Beulich &lt;jbeulich&lt; at &gt;novell.com&gt; Signed-off-by: David Howells &lt;dhowells&lt; at &gt;redhat.com&gt; Cc: Suresh Jayaraman &lt;sjayaraman&lt; at &gt;suse.de&gt; Cc: Geert Uytterhoeven &lt;geert&lt; at &gt;linux-m68k.org&gt; Cc: stable&lt; at &gt;kernel.org --- fs/fscache/page.c | 14 +++++--------- 1 files changed, 5 insertions(+), 9 deletions(-) diff --git a/fs/fscache/page.c b/fs/fscache/page.c index 60315b3..ff1e68f 100644 --- a/fs/fscache/page.c +++ b/fs/fscache/page.c &lt; at &gt;&lt; at &gt; -993,16 +993,12 &lt; at &gt;&lt; at &gt; void __fscache_uncache_all_inode_pages(struct fscache_cookie *cookie, pagevec_init(&amp;pvec, 0); next = 0; -while (next &lt;= (loff_t)-1 &amp;&amp; - pagevec_lookup(&amp;pvec, mapping, next, PAGEVEC_SIZE) - ) { +do { +if (!pagevec_lookup(&amp;pvec, mapping, next, PAGEVEC_SIZE)) +break; for (i = 0; i &lt; pagevec_count(&amp;pvec); i++) { struct page *page = pvec.pages[i]; -pgoff_t page_index = page-&gt;index; - -ASSERTCMP(page_index, &gt;=, next); -next = page_index + 1; - +next = page-&gt;index; if (PageFsCache(page)) { __fscache_wait_on_page_write(cookie, page); __fscache_uncache_page(cookie, page); &lt; at &gt;&lt; at &gt; -1010,7 +1006,7 &lt; at &gt;&lt; at &gt; void __fscache_uncache_all_inode_pages(struct fscache_cookie *cookie, } pagevec_release(&amp;pvec); cond_resched(); -} +} while (++next); _leave(""); } </pre> David Howells 2011-07-21T14:02:43 Use cachefs to Cache only small files,but on all types of Storage http://comments.gmane.org/gmane.linux.file-systems.cachefs.general/2979 <pre>Hello, I am working on a Linux-Media Center. Here, I have some large (video) files, and some small descriptive files. I need a lot of storage, which is either on USB or Network (NFS, Samba). The files are in many sub-directories. Thus, the listing of the recordings, which needs a) Folder Names b) Some info Files with the descriptions is slow and wakes the HDDs from suspend. Thus, I'd like to cache the directory structure and the info-files. Is cachefs the right thing for me? If not, what's an alternative? Regards, Hendrik </pre> Hendrik Friedel 2011-07-09T17:13:09 Search Engine Search the mailing list at Gmane query http://search.gmane.org/?group=$group=gmane.linux.file-systems.cachefs.general