http://permalink.gmane.org/gmane.linux.openvz.user hourly 1 1901-01-01T00:00+00:00 http://gmane.org/img/gmane-25t.png http://gmane.org http://permalink.gmane.org/gmane.linux.openvz.user/4690 <pre>Hi, ?? 22.5.2012 ?. 13:27 ?., Rene C. ??????: It's not very practical to access the containers from the VZ/private mount point, as it breaks for example the quota stats of the container. If you still want to do things there better go for the VZ/root mount point. (Advice given to me by one of the now-a-days developer of Viruozzo) And as you already mentioned ploop, as far as I know the ploop-container will be mounted to VZ/root of the CT and you'll still have access to the info in there. _______________________________________________ Users mailing list Users-GEFAQzZX7r8dnm+yROfE0A&lt; at &gt;public.gmane.org https://openvz.org/mailman/listinfo/users </pre> Martin Dobrev 2012-05-23T07:14:21 http://permalink.gmane.org/gmane.linux.openvz.user/4689 <pre> Apparently no problems with the file: # vzcfgvalidate -v yes 1407.conf Validation completed: success Thank you to everyone who provided suggestions, ideas and insight. I've added the user_beancounters to my loadmonitoring script. Next time there is a problem I'll check if any values are hitting the limit and see if increasing them may fix the problem. Best, Rene _______________________________________________ Users mailing list Users-GEFAQzZX7r8dnm+yROfE0A&lt; at &gt;public.gmane.org https://openvz.org/mailman/listinfo/users </pre> Rene C. 2012-05-22T17:09:05 http://permalink.gmane.org/gmane.linux.openvz.user/4688 <pre>You could check your &lt;VEID&gt;.conf with vzcfgvalidate. But I think there is quite a risk when giving one of your CT’s unlimited resources. If you want to read-out the UBC’s from the node and see when one fails I could recommend you a very good script I’m using myself; http://wiki.openvz.org/UBC_failcnt_reset There is no need to reset the value’s inside your CT. Van: users-bounces-GEFAQzZX7r8dnm+yROfE0A&lt; at &gt;public.gmane.org [mailto:users-bounces-GEFAQzZX7r8dnm+yROfE0A&lt; at &gt;public.gmane.org] Namens Rene C. Verzonden: dinsdag 22 mei 2012 14:17 Aan: users-GEFAQzZX7r8dnm+yROfE0A&lt; at &gt;public.gmane.org Onderwerp: Re: [Users] occasional high loadavg without any noticeable cpu/memory/io load On Tue, May 22, 2012 at 6:59 PM, Esmé de Wolf &lt;esme-cs4as8ShrIaEVqv0pETR8A&lt; at &gt;public.gmane.org&gt; wrote: I also think that these UBC settings are not consistent. Especially when you have all containers configured with these same UBC settings you will have soon or later problems. See: http://wiki.openvz.org/UBC_consistency_chec</pre> Esmé de Wolf 2012-05-22T12:54:09 http://permalink.gmane.org/gmane.linux.openvz.user/4687 <pre> On May 22, 2012, at 16:17 , Rene C. wrote: On Tue, May 22, 2012 at 6:59 PM, Esmé de Wolf &lt;esme-cs4as8ShrIaEVqv0pETR8A&lt; at &gt;public.gmane.org&lt;mailto:esme-cs4as8ShrIaEVqv0pETR8A&lt; at &gt;public.gmane.org&gt;&gt; wrote: I also think that these UBC settings are not consistent. Especially when you have all containers configured with these same UBC settings you will have soon or later problems. See: http://wiki.openvz.org/UBC_consistency_check and other pages on the WIKI. Kind Regards, Esme I read that UBC page already and used it to set these values. No, all my containers do not have the same UBC settings, they were set depending on how much resources each container should have. Please let me know where any of the values in my conf file conflicts with the UBC recommendations. I do understand that they may need to be fine tuned in each case, but that's basically what this question is about :) So basically at this time I have two questions I don't understand: 1) how is it possible to have physpages hit the limit when top n</pre> Kirill Korotaev 2012-05-22T12:35:33 http://permalink.gmane.org/gmane.linux.openvz.user/4686 <pre> I read that UBC page already and used it to set these values. No, all my containers do not have the same UBC settings, they were set depending on how much resources each container should have. Please let me know where any of the values in my conf file conflicts with the UBC recommendations. I do understand that they may need to be fine tuned in each case, but that's basically what this question is about :) So basically at this time I have two questions I don't understand: 1) how is it possible to have physpages hit the limit when top never shows more than about 75-80% of the memory used? 2) how did dcachesize hit limit when both df -i and df -h shows plenty of resources - and haven't been close to limits? Could the values in the beancounter file be old? Is there a way to reset them (without restarting the CT) ? Best, Rene _______________________________________________ Users mailing list Users-GEFAQzZX7r8dnm+yROfE0A&lt; at &gt;public.gmane.org https://openvz.org/mailman/listinfo/users </pre> Rene C. 2012-05-22T12:17:29 http://permalink.gmane.org/gmane.linux.openvz.user/4685 <pre>2012/5/22 Rene C. &lt;openvz-ujNoZmtUEYbQT0dZR+AlfA&lt; at &gt;public.gmane.org&gt;: Only partially correct :-) You can enter the filesystem of a CT when it's mounted. Meaning - you can enter the root directory when the CT is running. If the CT is shut down you always have the possibility to mount the ploop file to any directory you desire. </pre> Sirk Johannsen 2012-05-22T12:01:27 http://permalink.gmane.org/gmane.linux.openvz.user/4684 <pre>I also think that these UBC settings are not consistent. Especially when you have all containers configured with these same UBC settings you will have soon or later problems. See: http://wiki.openvz.org/UBC_consistency_check and other pages on the WIKI. Kind Regards, Esme Van: users-bounces-GEFAQzZX7r8dnm+yROfE0A&lt; at &gt;public.gmane.org [mailto:users-bounces-GEFAQzZX7r8dnm+yROfE0A&lt; at &gt;public.gmane.org] Namens Kirill Korotaev Verzonden: dinsdag 22 mei 2012 13:05 Aan: users-GEFAQzZX7r8dnm+yROfE0A&lt; at &gt;public.gmane.org users-GEFAQzZX7r8dnm+yROfE0A&lt; at &gt;public.gmane.org; Rene C. Onderwerp: Re: [Users] occasional high loadavg without any noticeable cpu/memory/io load Looks like in your case you've hit physpages limit. In such situations VPS behaves as a standalone machine - it starts to swap out (though "virtually") and process stuck in D state (swap in / swap out), which contributes to loadavg. So either increase memory limits for your VPS or kill/tune the memory hungry workload. Note: loadavg can also </pre> Esmé de Wolf 2012-05-22T11:59:09 http://permalink.gmane.org/gmane.linux.openvz.user/4683 <pre>Looks like in your case you've hit physpages limit. In such situations VPS behaves as a standalone machine - it starts to swap out (though "virtually") and process stuck in D state (swap in / swap out), which contributes to loadavg. So either increase memory limits for your VPS or kill/tune the memory hungry workload. Note: loadavg can also increase due to CPU limits as processes are delayed when overuse their CPU. Thanks, Kirill On May 22, 2012, at 14:49 , Rene C. wrote: Hi Esme, Thanks for the suggestion, much appreciated! I didn't think of checking at the time I'm afraid. I suppose since the container has not been rebooted since, the beancounters should still show any problems encountered at the time right? Below is the user_beancounters of the problem CT. I notice physpages and dcachesize have maxheld values very close to limits (even if failcnt is zero) could that have been the cause? uid resource held maxheld barrier lim</pre> Kirill Korotaev 2012-05-22T11:05:16 http://permalink.gmane.org/gmane.linux.openvz.user/4682 <pre>Actually at the time I cat'ed these beans the memory used according to top was around 2.5GB so that seems right enough. Still doesn't explain how maxheld is so close to limit when top at the time of the trouble showed just around 1.5G memory used. _______________________________________________ Users mailing list Users-GEFAQzZX7r8dnm+yROfE0A&lt; at &gt;public.gmane.org https://openvz.org/mailman/listinfo/users </pre> Rene C. 2012-05-22T10:59:33 http://permalink.gmane.org/gmane.linux.openvz.user/4681 <pre>Hi Esme, load could be caused by buffers that are full. Thanks for the suggestion, much appreciated! I didn't think of checking at the time I'm afraid. I suppose since the container has not been rebooted since, the beancounters should still show any problems encountered at the time right? Below is the user_beancounters of the problem CT. I notice physpages and dcachesize have maxheld values very close to limits (even if failcnt is zero) could that have been the cause? uid resource held maxheld barrier limit failcnt 1407: kmemsize 252703307 1124626432 1932525568 2147483648 0 lockedpages 0 15 524288 524288 0 privvmpages 893372 5683554 9223372036854775807 9223372036854775807 0 shmpages 23 </pre> Rene C. 2012-05-22T10:49:20 http://permalink.gmane.org/gmane.linux.openvz.user/4680 <pre>Hi Sirk, Thanks for the info, much appreciated! Maybe a little off topic, but I am curious to know: At the moment I find it very convenient to go directly into a containers filesystem from the hardware node - i.e. something like /vz/private/xxx/var/log/... etc - Would I be correct in presuming that by using ploop this will no longer be possible? I know I could just setup a test system and try it out but if you know already it would save me some time ;) Indeed, this is the only container on that filesystem and that physical drive. This time there were no "spill over" but previous times when load hit 50 or more the load certainly did spill into other containers. Best, Rene _______________________________________________ Users mailing list Users-GEFAQzZX7r8dnm+yROfE0A&lt; at &gt;public.gmane.org https://openvz.org/mailman/listinfo/users </pre> Rene C. 2012-05-22T10:27:20 http://permalink.gmane.org/gmane.linux.openvz.user/4679 <pre>Hi Rene, Did you check the /proc/user_beancounters of that VPS? Sometime’s a high load could be caused by buffers that are full. Hope it helpes you, Kind Regards, Esme de Wolf Van: users-bounces-GEFAQzZX7r8dnm+yROfE0A&lt; at &gt;public.gmane.org [mailto:users-bounces-GEFAQzZX7r8dnm+yROfE0A&lt; at &gt;public.gmane.org] Namens Rene Dokbua Verzonden: maandag 21 mei 2012 20:07 Aan: users-GEFAQzZX7r8dnm+yROfE0A&lt; at &gt;public.gmane.org Onderwerp: [Users] occasional high loadavg without any noticeable cpu/memory/io load Hello, I occasionally get this extreme load on one of our VPS servers. It is quite large, 4 full E31230 cores, 4 GB RAM and hosting ca. 400 websites + parked/addon/subdomains. The hardware node has 12 active VPS servers and most of the time things are chugging along just fine, something like this. 1401: 0.00 0.00 0.00 1/23 4561 1402: 0.02 0.05 0.05 1/57 16991 1404: 0.01 0.02 0.00 1/73 18863 1406: 0.07 0.13 0.06 1/39 31189 1407: 0.86 1.03 1.14 1/113 31460 1408: 0.17 0.17 0.18 1/79 32579 </pre> Esmé de Wolf 2012-05-22T10:00:25 http://permalink.gmane.org/gmane.linux.openvz.user/4678 <pre>2012/5/22 Rene C. &lt;openvz-ujNoZmtUEYbQT0dZR+AlfA&lt; at &gt;public.gmane.org&gt;: Hi Rene, True, but this list has helped me a lot as well :-) If you want some practical information on ploop: We are using it in a highly productive environment. It was either, try ploop and hope it works, or have the systems fail every 2nd day. So we decided to use ploop and are more than happy. It even solves a lot of issues we had with the private areas directly on the nfs share. But of course, thats totally up to you. I started with only a few "unimportant" CTs and then merged everything after a while (42 CTs). To be honest, I don't know. iostat ist not working because you do not really have a device. This ist handled the way with ploop sadly but could be modified I guess. For ploop you have the ploop-stat command but that dosen't work as expected for me :-) So you have a different FileSystem for the "problem"-Container that is even on a different disk ? If that is the case, this CT should not affect the others at all in terms o</pre> Sirk Johannsen 2012-05-22T09:50:40 http://permalink.gmane.org/gmane.linux.openvz.user/4677 <pre>Hi Sirk, Thanks for your reply. I'm so pleased having found this mailing list after having tried the forum, which seem to have very little activity! Ploop is a great idea technically, but I'm a little concerned about the " Warning: This is a new feature, not yet ready for production systems. Use with caution." on the OpenVZ Wiki page, so I'm kinda waiting for the green-light that it's ready for production environments. It did occur to me that disk-IO could be the cause of the problem, but iostat on the hardware node did not suggest any particular IO problems. I still haven't found a way to see the IO activity within a container - iostat just comes up blank when it's run within a container. Is there a way? We're not using any network storage with this server so that is not the reason. The server has 4 SATA-3 drives, with the root partition being on one drive, the problem container alone on a second drive, and the remaining containers on a third. Best, Rene On Tue, May 22, 2012 at 3:06 PM, Sirk Johann</pre> Rene C. 2012-05-22T09:16:00 http://permalink.gmane.org/gmane.linux.openvz.user/4676 <pre>Actually I made a small shell script that loops through the list of active containers and outputs the content of each containers /proc/loadavg. It started out as a bit more elaborate script that was intended to provide some of the functionality of a script vzstat, that I used to use with Virtuozzo. You can download both scripts from https://www.ourhelpdesk.net/downloads/z.tgz On Tue, May 22, 2012 at 3:15 PM, Steffan &lt;general-nv30jWqA7Mk&lt; at &gt;public.gmane.org&gt; wrote: _______________________________________________ Users mailing list Users-GEFAQzZX7r8dnm+yROfE0A&lt; at &gt;public.gmane.org https://openvz.org/mailman/listinfo/users </pre> Rene C. 2012-05-22T09:06:45 http://permalink.gmane.org/gmane.linux.openvz.user/4675 <pre>Sorry dont have the answer for you But can you tell me what command you used to see all loads on your node ? Thanxs Steffan Van: users-bounces-GEFAQzZX7r8dnm+yROfE0A&lt; at &gt;public.gmane.org [mailto:users-bounces-GEFAQzZX7r8dnm+yROfE0A&lt; at &gt;public.gmane.org] Namens Rene Dokbua Verzonden: maandag 21 mei 2012 20:07 Aan: users-GEFAQzZX7r8dnm+yROfE0A&lt; at &gt;public.gmane.org Onderwerp: [Users] occasional high loadavg without any noticeable cpu/memory/io load Hello, I occasionally get this extreme load on one of our VPS servers. It is quite large, 4 full E31230 cores, 4 GB RAM and hosting ca. 400 websites + parked/addon/subdomains. The hardware node has 12 active VPS servers and most of the time things are chugging along just fine, something like this. 1401: 0.00 0.00 0.00 1/23 4561 1402: 0.02 0.05 0.05 1/57 16991 1404: 0.01 0.02 0.00 1/73 18863 1406: 0.07 0.13 0.06 1/39 31189 1407: 0.86 1.03 1.14 1/113 31460 1408: 0.17 0.17 0.18 1/79 32579 1409: 0.00 0.00 0.02 1/77 21784 1410: 0.01 0.02 0.00 1/60 7454</pre> Steffan 2012-05-22T08:15:40 http://permalink.gmane.org/gmane.linux.openvz.user/4674 <pre>Hi Rene, Since CPU and MEM are fine it's most likely to be Disk-IO. I have similar Problems with a Cluster Setup based on OpenVZ. The problem is that our Storage is way to slow. We have been accessing the storage via NFS and put all our CTs private areas on it. I noticed many times that one CT was doing a lot of disk IO and all other were suffering from that... that even lead to total system failures. This has been solved by converting everything to ploop. Since then our system is at least in a stable state. IO Performance is still an issue but does not bring our system down. You should give ploop a try :-) I am very happy with it. best regards, Sirk 2012/5/21 Rene Dokbua &lt;openvz-ujNoZmtUEYbQT0dZR+AlfA&lt; at &gt;public.gmane.org&gt;: </pre> Sirk Johannsen 2012-05-22T08:06:15 http://permalink.gmane.org/gmane.linux.openvz.user/4673 <pre>Hello, I occasionally get this extreme load on one of our VPS servers. It is quite large, 4 full E31230 cores, 4 GB RAM and hosting ca. 400 websites + parked/addon/subdomains. The hardware node has 12 active VPS servers and most of the time things are chugging along just fine, something like this. 1401: 0.00 0.00 0.00 1/23 4561 1402: 0.02 0.05 0.05 1/57 16991 1404: 0.01 0.02 0.00 1/73 18863 1406: 0.07 0.13 0.06 1/39 31189 1407: 0.86 1.03 1.14 1/113 31460 1408: 0.17 0.17 0.18 1/79 32579 1409: 0.00 0.00 0.02 1/77 21784 1410: 0.01 0.02 0.00 1/60 7454 1413: 0.00 0.00 0.00 1/46 18579 1414: 0.00 0.00 0.00 1/41 23812 1415: 0.00 0.00 0.00 1/45 9831 1416: 0.05 0.02 0.00 1/59 11332 12 active The problem VPS is 1407. As you can see below it only uses a bit of the cpu and memory. top - 17:34:12 up 32 days, 12:21, 0 users, load average: 0.78, 0.95, 1.09 Tasks: 102 total, 4 running, 90 sleeping, 0 stopped, 8 zombie Cpu(s): 16.3%us, 2.9%sy, 0.4%ni, 78.5%id, 1.8%wa, 0.0%hi, 0.0%si, 0.1%st Mem: 4194304k</pre> Rene Dokbua 2012-05-21T18:06:53 http://permalink.gmane.org/gmane.linux.openvz.user/4672 <pre>http://bugzilla.openvz.org/show_bug.cgi?id=2249 On Sat, May 19, 2012 at 10:01 AM, Peter Schultze &lt;petersch-764C0pRuGfqVc3sceRu5cw&lt; at &gt;public.gmane.org&gt; wrote: </pre> helpaz 2012-05-19T07:56:20 http://permalink.gmane.org/gmane.linux.openvz.user/4671 <pre> I am seeing exactly the same problem on a 32-bit CentOS 6.2 machine with vzkernel-2.6.32-042stab055.10.i686 ploop-lib-1.2-1.i386 vzctl-lib-3.2-1.i386 OS installation into a newly created CT fails because nothing can be written to the newly created ploop storage. When the same machine is reverted to 042stab054.3.i686 the ploop container creation is working working fine again. The problem seems to occur under all newer kernels beginning with 042stab054.4, including 042stab055.11 There is no problem on a 64-bit CentOS 6.2 machine with vzkernel-2.6.32-042stab055.10.x86_64 Could it be that this is only happening with the recent 32-bit kernels? </pre> Peter Schultze 2012-05-19T07:01:19 http://permalink.gmane.org/gmane.linux.openvz.user/4670 <pre> Blimey, that's an old bit of kit - I bet it costs more in electricity to run that server for 1 year (probably more than 1000 Euros) than the whole machine is now worth... Still... I assume it doesn't have 64 bit processors installed? If it does, you might want to switch to the 64 bit kernel, as it's better tested. What are your symptoms of the "stall", let's assume it's deadlocked processes - you should find out what the processes are blocked on (e.g. using the top 'WCHAN' column). You can get some state info out of the kernel with: echo 'T' &gt; /proc/sysrq-trigger echo 'W' &gt; /proc/sysrq-trigger etc. Probably best to open a Debian kernel bug, as the OpenVZ team don't support the Debian kernels. You might want to try the official OpenVZ kernels, and see if the problem still occurs. Tim. </pre> Tim Small 2012-05-11T10:43:25 Search the mailing list at Gmane query http://search.gmane.org/?group=$group=gmane.linux.openvz.user