http://blog.gmane.org/gmane.comp.programming.garbage-collection.boehmgc hourly 1 1901-01-01T00:00+00:00 http://gmane.org/img/gmane-25t.png http://gmane.org http://permalink.gmane.org/gmane.comp.programming.garbage-collection.boehmgc/2239 That should work, yes. This is normally supported by the GC library by default, at least if your platform has the proper support. You can turn it on with GC_enable_incremental(). The problem is that this is usually not entirely transparent, since it protects memory and catches the resulting faults. (On some platforms, this isn't necessary, and it should just work. On Linux, the standard kernels don't have the hooks to do this.) --enable-threads=posix That will slow things down a bit. I think the best you can do is to turn off incremental GC, and use it as is. There is some discussion of this in gcconfig.h, around line 2000. Basically malloc redirection with threads is still a work in progress. To really do better probably requires some hooks in NPTL, and possibly the dynamic loader, that we don't currently have. The core problem is that the guts of both of those potentially allocate memory through malloc, but hide pointers in places the collector doesn't expect to find pointers, e.g. currently unused thread stacks. Currently the collector tries fairly hard to work around all of this, using a variety of hacks invented by several people. But it really wants some callbacks to identify roots in weird places. Hans Boehm, Hans 2008-07-18T21:45:06 http://permalink.gmane.org/gmane.comp.programming.garbage-collection.boehmgc/2238 I'm trying to figure out a policy for how to use GC in the real world, with libraries. Sorry if this is too many posts. The prospect of having a GC for C just gets me so excited and hopeful. Looking through the code at the various comments on thread-related matters I'm not sure if it will quite work for me though, or if there might exist some discipline I could follow that would let it mostly work. I think others probably have some of the same uncertainties I do on this one, and if anyone can help I'll prepare some sort of document explaining as best I can how things work. The dream of course it to be able to use whatever precompiled libraries one wants, with as many threads as the heart desires and never worry about garbage collection, nor have the collector cause unseemly hiccups while doing full collection cycles. How achievable is this in general? I think my situation presents an interesting coverage of the possible library issues one might have: 1. glib -- has the set_vtable fctn, and also these days --enable-gc-friendly, --enable-debug=yes, and two significant environment vars that control run-time behavior: G_DEBUG=gc-friendly and G_SLICE=always-malloc For this one, the gc seems to work (i.e. doesn't seg fault in a GC_malloc) for the program I'm working on if either: a. G_DEBUG=gc-friendly and G_SLICE=always-malloc are set at run time (set_vtable alone doesn't do it). or b. gc is compiled with --enable-redirect-malloc (but then the GC does make things many times slower when threads are also enabled, as the docs warn. 2. the GNU Scientific Library (GSL). This library doesn't have support for set_vtable like functionality, but it appears to use plain old malloc internally and no fancy slice style stuff like glib. For a library like this, is --enable-malloc-redirects the only way? I tried just putting my own malloc free etc. that called GC_* in my program, and when the GSL called malloc it always seemed to get my routines, so maybe this is a legitimate way to use a precompiled GC with an existing library that uses simple memory management, like GSL? I asked how best to use GC with GSL on GSL mailing list, and they advised to use the --wrap linker option. I'm not convinced this works though: I still seem to end up in system malloc from gsl_* fctns according to debugger. It seems like it only affects the program being linked, and not the fctns that already linked shared libs use. The implication is that some library authors (even of very high-quality libs like GSL) aren't sure how to use a thing like GC themselves. 3. Gtk. I thought this used threads internally, but now I'm not sure. So I don't know if I myself need any threaded libraries, but its still an interesting question: what are the implications of a threaded library for how gc must be used? As I said at the start, if there is an approach to which type of gc build to use, how to link, and what has to be looked at and done when adding a new library to the list of dependencies for a GC-enabled program or if people who have the experience to do these things without following a specific policy could somehow capture their knowledge, I think it would really help to let more people use the GC. Of course, if all of the above is too easy for you guys, there always the question of how GC_enable_incremental fits in the whole threaded picture... :) Thanks, Britton Britton Kerin 2008-07-18T01:46:41 http://permalink.gmane.org/gmane.comp.programming.garbage-collection.boehmgc/2237 It just doesn't. It is not difficult to do 'make clean' by hand before reconfiguring if that is what you want. -- Henning Makholm OctoShape ApS Henning Makholm 2008-07-17T22:12:59 http://permalink.gmane.org/gmane.comp.programming.garbage-collection.boehmgc/2236 I'm trying to figure out how to build the gc (7.1) for the following requirements: 1. Malloc interposition. I think this just needs --enable-redirect-malloc now? 2. incremental collection 3. Best odds of working with threads. I think its just --enable-thread=posix 4. I also put in --enable-gc-assertions The trouble is when I run programs linked to this library I get the following messages: ./test_ec_lock_test_binary; GC Warning: USE_PROC_FOR_LIBRARIES + GC_LINUX_THREADS performs poorly. GC Warning: Incremental GC incompatible with /proc roots GC Warning: Incremental GC incompatible with /proc roots etc. Am I missing some configure option or is it necessary to hand-modify some Makefiles to get this combination working? Thanks, Britton Britton Kerin 2008-07-17T21:12:33 http://permalink.gmane.org/gmane.comp.programming.garbage-collection.boehmgc/2235 I just noticed the following: ./configure --prefix=$HOME/local --enable-redirect-malloc --enable- threads=posix --enable-gc-assertions make [output snipped] ./configure --prefix=$HOME/local --enable-redirect-malloc --enable-threads=posix make make[1]: Entering directory `/home/bkerin/tmp/gc_stuff/gc-7.1' make[1]: Nothing to be done for `all-am'. make[1]: Leaving directory `/home/bkerin/tmp/gc_stuff/gc-7.1' It seems like rerunning configure should cause everything to get rebuild, but I guess maybe autoconf just doesn't work like that. Britton Britton Kerin 2008-07-17T21:16:41 http://permalink.gmane.org/gmane.comp.programming.garbage-collection.boehmgc/2234 The html docs in gcinterface.html starts like this: On many platforms, a single-threaded garbage collector library can be built to act as a plug-in malloc replacement. (Build with -DREDIRECT_MALLOC=GC_malloc -DIGNORE_FREE.) This is often the best way to deal with third-party libraries which leak or prematurely free objects. -DREDIRECT_MALLOC is intended primarily as an easy way to adapt old code, not for new development. I think that gc is misleading potential users/selling itself a bit short with this description of things. If I understand correctly, if one wants to both a. use GC and b. use outside libraries not compiled to use GC, it is necessary to use malloc interposition and I believe redirect-malloc is the way to do this? It seems like this sort of thing is likely to be the most common case, at least for people who just want a GC for their own quick-and-dirty programs. Britton Britton Kerin 2008-07-17T21:23:28 http://permalink.gmane.org/gmane.comp.programming.garbage-collection.boehmgc/2233 MIPS Technologies provides a number of specifications on its website www.mips.com, which is somewhat vague, and company internal documentation which is clearer. (I was a MIPS employee until a few months ago.) Then it should work for N32 but not for N64. The Debian-packaged standalone version of libatomic-ops had such a problem at some point. Maybe the archived bug reports carry some useful information WRT that, and the m68k patch: http://bugs.debian.org/cgi-bin/pkgreport.cgi?ordering=normal;archive=1;dist=unstable;package=libatomic-ops;repeatmerged=1 Thiemo Thiemo Seufer 2008-07-04T02:21:19 http://permalink.gmane.org/gmane.comp.programming.garbage-collection.boehmgc/2232 Thanks for testing. Is there now an official specification? If so, I fully agree that we should make only assumptions stated there. Last I looked, things seemed unclear, which is why I suggested considering multiple options. AO_t is defined to match the size of pointers. I think that, at least for this package, they were always intended to be automatically generated, though there may have been issues there. Thanks. If nobody objects soon, I'll rip that one out, since I think it actually breaks something. Admittedly, I don't know how many multiprocessor M68K systems are in existence. Hans Boehm, Hans 2008-07-03T22:55:30 http://permalink.gmane.org/gmane.comp.programming.garbage-collection.boehmgc/2231 I can confirm it builds and passes the testsuite on (big-endian) Linux/MIPS. I notice you added a comment: /* * FIXME: This should probably make finer distinctions. SGI MIPS is * much more strongly ordered, and in fact closer to sequentially * consistent. This is really aimed at modern embedded implementations. * It looks to me like this assumes a 32-bit ABI. -HB */ The MIPS architecture specification allows for weaker ordering than what SGI used to implement. Since most modern MIPS SMP implementations use a weak ordering variant, with "sync" being the only common barrier operation, I implemented the variant which works on all MIPS systems. As long as AO_t is a 32-bit quantity the code should also work on N32/N64 ABIs, although I haven't tested this. IIRC I carried those forward from an earlier version of libatomic_ops, where they weren't automatically generated. If that changed now then the versions in mips.h are redundant. AFAICS this was also written for an earlier libatomic_ops, I figure the same rationale applies as for MIPS. That said, I don't know m68k enough to do more than guesswork. :-) Thiemo Thiemo Seufer 2008-07-03T22:41:57 http://permalink.gmane.org/gmane.comp.programming.garbage-collection.boehmgc/2230 My apologies. There was apparently a merge accident related to the mips.h file. I fixed that, and checked in the other two small patches. Things should be OK now. I noticed that in a couple of places, the atomic_ops patches don't seem to take advantage of the structure of the package. If you omit operations, it tries to synthesize them from the exisiting ones. Mips.h seems to supply implementations for things like AO_compare_and_swap_acquire that look like they should be equivalent to the automatically generated ones. Is there a reason not to take those out? The m68k patch seems to supply a non-thread-safe or operation, where I think the synthesized one should correctly use compare-and-swap. I would really like to take that one out. Hans Boehm, Hans 2008-07-03T21:50:32 http://permalink.gmane.org/gmane.comp.programming.garbage-collection.boehmgc/2229 Ping? Those patches are still needed, the current state of CVS doesn't even build on Linux/MIPS. Thiemo Thiemo Seufer 2008-07-03T20:10:49 http://permalink.gmane.org/gmane.comp.programming.garbage-collection.boehmgc/2228 This was addressed more than a year ago. So if you base on current sources you will not have this issue. Andreas Andreas Tobler 2008-07-02T17:45:42 http://permalink.gmane.org/gmane.comp.programming.garbage-collection.boehmgc/2227 Hi, I'm mailing this upstream from the dotgnu project. I've found a problem in libgc that prevents it from compiling on OSX 10.5.3 (and most likely other versions), due to missing underscores in the thread structure. Each register name in OSX is prefixed with two underscores. I don't know if this is necessary in other Darwin variants, though, so please be warned. Thanks! -Kyle H Index: darwin_stop_world.c =================================================================== RCS file: /sources/dotgnu-pnet/pnet/libgc/darwin_stop_world.c,v retrieving revision 1.4 diff -c -r1.4 darwin_stop_world.c *** darwin_stop_world.c5 Nov 2006 18:49:52 -00001.4 --- darwin_stop_world.c2 Jul 2008 06:55:33 -0000 *************** *** 102,116 **** if(r != KERN_SUCCESS) ABORT("thread_get_state failed"); #if defined(I386) ! lo = state.esp; ! GC_push_one(state.eax); ! GC_push_one(state.ebx); ! GC_push_one(state.ecx); ! GC_push_one(state.edx); ! GC_push_one(state.edi); ! GC_push_one(state.esi); ! GC_push_one(state.ebp); #elif defined(POWERPC) lo = (void*)(state.r1 - PPC_RED_ZONE_SIZE); --- 102,116 ---- if(r != KERN_SUCCESS) ABORT("thread_get_state failed"); #if defined(I386) ! lo = state.__esp; ! GC_push_one(state.__eax); ! GC_push_one(state.__ebx); ! GC_push_one(state.__ecx); ! GC_push_one(state.__edx); ! GC_push_one(state.__edi); ! GC_push_one(state.__esi); ! GC_push_one(state.__ebp); #elif defined(POWERPC) lo = (void*)(state.r1 - PPC_RED_ZONE_SIZE); Kyle Hamilton 2008-07-02T07:00:58 http://permalink.gmane.org/gmane.comp.programming.garbage-collection.boehmgc/2226 On Tue, 1 Jul 2008, DrPizza wrote: I agree that new peoject files would be good. nmake has the big advantage that the Makefiles are editable on another platform. The syntax at least is more or less compatible with other make implementations, which are ubiquitous. The commandline options aren't, but that's easier to live with. I think even just allocating small objects (e.g. 32 bytes) in a tight loop would give you some idea. I haven't measured this carefully on Windows. On Linux, the lock acquisition per object is a big deal. My impression is that Windows CriticalSections are no cheaper. It's probably more of an issue on P4s than it is on more modern processors. That works. Hans Hans Boehm 2008-07-01T04:10:52 http://permalink.gmane.org/gmane.comp.programming.garbage-collection.boehmgc/2225 I think all of that is true of the nmake Makefiles too, isn't it? And nmake has the disadvantage that no-one really uses it anyway these days. My feeling was that at least by having a project file, it would be somewhat more approachable. OK. No, at least not according to the documentation. __STDC__ is only defined by VC++ when the MS extensions are disabled; however, I believe the MS extensions are mandatory if one wants to parse the Windows headers. I wasn't sure about this one. It felt safer to me to use DllMain-based registration, just because it feels more "automatic"; you link against the library and it does (more or less) the Right Thing. Is there any kind of a benchmark that would make a case one way or the other? Yes, I saw that; once I can figure out why it's crashing I think updating that to work on x64 should be easy to do. OK. IS the best way to proceed to figure out how to generate a patch and then send it to the mailing list? Peter DrPizza 2008-07-01T00:54:34 http://permalink.gmane.org/gmane.comp.programming.garbage-collection.boehmgc/2224 The current Windows Makefiles are not in great shape. The problem with project files is that they're a single platform, completely nonportable, solution. They tend to be hard to edit for anyone not on a Windows platform. And we've been short a volunteer to maintain them. Certainly updated versions would be appreciated, but they tend to deteriorate to a basis for a do-it-yourself project in the absence of a maintainer. These first two are redundant, at least for recent versions. One should be enough. Already defined by compiler? This really needs tobe defined in client code to get the debug functionality. Otherwise I think it mostly affects testing. That's an interesting and unfortunate tradeoff. I would expect THREAD_LOCAL_ALLOC to be appreciably faster. But it doesn't work with DllMain-based implicit thread registration. If you know that all clients will be using the normal portable GC interface, I'd recommend turning on THREAD_LOCAL_ALLOC for DLL builds, though that may not have been well tested. Me neither. Note that some of the related code is in msvc_dbg.c. Sounds good to me. Hans Boehm, Hans 2008-07-01T00:31:22 http://permalink.gmane.org/gmane.comp.programming.garbage-collection.boehmgc/2223 Hello. I have a question (down below) and some minor modifications and project files that might be nice to share. It may be that I'm a total nub, or it may be that Makefiles are horrid and nmake even worse, but I've never really cared for building at the command-line. VC++ is a much more pleasing environment to work in when doing my Windows development. Anyway, the makefile versions have been annoying me for some time because I'm never sure which CRT they're using; there also seems to be something strange with MS's makefile header thing that seems to make non-debug builds be debug builds when building x64 binaries or something. Anyway. I want to put together VC++ 2008 project files to build the Cartesian product of { win32, win64 } × { debug, release } × { static, DLL }, all multithreaded (as VC++ nowadays only has a multithreaded runtime, it seems appropriate). I'm currently using the following #defines (in addition to VC++'s defaults, so _WIN32, _DEBUG/NDEBUG, and the various processor type #defines etc.): All builds: GC_THREADS, GC_WIN32_THREADS, GC_BUILD, ALL_INTERIOR_POINTERS, __STDC__ Debug builds: GC_DEBUG DLL builds: GC_DLL Static builds: GC_NOT_DLL, THREAD_LOCAL_ALLOC My question is, are these reasonable? The project files in CVS also define SAVE_CALL_COUNT=8. If I define that then I get crashes inside GC_save_callers; the memcpy (BCOPY) is passed bad parameters and crashes. Don't know why. I also have a .lib for C++ programs to override global operator new/global operator delete (this can only be done with a .obj or a static .lib; it can't use a DLL, or so says MS). I'm building all the tests except the trace_test (which won't link; I think the immediate problem is that GC_generate_random_backtrace needs to be marked with GC_API, but that's not enough to make it work anyway). Each test is constructed with each possible GC library (i.e. 8 versions of each are produced). I have made some changes to the thread_leak_test so that it can use Win32 threads as well as pthreads. I think the tests all run successfully. The project structure I'm using is quite different from that of the projects that are in CVS. Instead of a project each for static/DLL I have one project with multiple configurations. I find this makes organization a little easier. DrPizza 2008-06-30T22:23:01 http://permalink.gmane.org/gmane.comp.programming.garbage-collection.boehmgc/2222 Is this single-threaded? Is there a danger of a block being deallocated while this is running? Does p look reasonable and hhdr just turns out to be null? Or is something else going on? (GC_find_header() can be used to get the header from a debugger.) I don't immediately see why this shouldn't work, aside from the fact that some of this code is generally sloppy about locking. The collector itself should invoke it with the allocation lock held, but it's meant to be invocable from a debugger. Hans Boehm, Hans 2008-06-30T17:59:10 http://permalink.gmane.org/gmane.comp.programming.garbage-collection.boehmgc/2221 I'm trying to dump all active blocks along with the debugging information in GC 6.7. Seems a straightforward task, but I can't find any routines to do it except GC_FIND_LEAKS which is not what I want. I tried to piece together code, but it insist on crapping out near the top of dumpblock. This has to be possible. Is there some way to do so? #include "private/gc_priv.h" #include "private/dbg_mlc.h" static void dumpblock(struct hblk *p, word dummy) { register hdr * hhdr = HDR(p); register size_t bytes = WORDS_TO_BYTES(hhdr -> hb_sz); //craps out near here because p or hhdr are invalid //even though this is a straight copy from GC_print_block_list >> GC_print_block_descr //and the GC_print_block_list works fine GC_err_printf3("(%lu:%lu,%lu)", (unsigned long)(hhdr -> hb_obj_kind), (unsigned long)bytes, (unsigned long)(GC_n_set_marks(hhdr))); if (hhdr -> hb_obj_kind == PTRFREE) { GC_err_printf0(" atomic object at "); } else { GC_err_printf0(" composite object at "); } register oh * ohdr = (oh *)GC_base(p); if (GC_HAS_DEBUG_INFO(p)) { GC_err_printf1("0x%lx (", ((unsigned long)ohdr + sizeof(oh))); GC_err_puts(ohdr -> oh_string); # ifdef SHORT_DBG_HDRS GC_err_printf1(":%ld, ", (unsigned long)(ohdr -> oh_int)); # else GC_err_printf2(":%ld, sz=%ld, ", (unsigned long)(ohdr -> oh_int), (unsigned long)(ohdr -> oh_sz)); # endif GC_print_type((ptr_t)(ohdr + 1)); GC_err_puts(")\n"); PRINT_CALL_CHAIN(ohdr); } else { ptr_t base = GC_base(p); GC_err_printf2("start: 0x%lx, appr. length: %ld", base, GC_size (base)); } GC_err_printf0("\n"); } static void dumpallblocks(void) { GC_err_printf0("=======================================\n"); GC_print_block_list(); // this works fine GC_apply_to_all_blocks(dumpblock, (word)0); GC_err_printf0("---------------------------------------\n"); } S M Ryan 2008-06-29T09:11:38 http://permalink.gmane.org/gmane.comp.programming.garbage-collection.boehmgc/2220 At Thu, 19 Jun 2008 13:09:04 +0200, Neal H. Walfield wrote: Turns out that it was a bug in my run-time. For the interested, it had to do with map splitting, i.e., unmapping part of an mmaped region. Neal H. Walfield 2008-06-21T18:18:29 http://permalink.gmane.org/gmane.comp.programming.garbage-collection.boehmgc/2219 At Thu, 19 Jun 2008 00:28:16 +0200, Neal H. Walfield wrote: After some more investigation, it seems that it is not GC_malloc that is failing but something else (quite possibly my run time). Running gctest, I also see failures, usually complains that: List reversal produced incorrect list - collector is broken Other times, it simply terminates due to a NULL pointer dereference. Occasionally, the test passes. This happens either with or without my scheduler enabled. USE_MMAP and USE_MUNMAP are defined. When I disable USE_MUNMAP, the test runs to completion successfully. I've observed that no other configuration currently uses USE_MUNMAP. When enabling USE_MUNMAP and USE_MMAP on GNU/Linux running on an x86 processor, the test passes. When I enable my scheduler (by fixing the available parameter), however, it fails: Completed 1 tests Finalized 2206/2206 objects - finalization is probably ok Total number of bytes allocated is 60591488 Final heap size is 44679168 bytes Unexpected heap growth - collector may be broken Test failed Aborted This is clearly different from my error and thus I suspect it is unrelated. Any advice would be appreciated. By the way, I am using gc 7.0. Neal Neal H. Walfield 2008-06-19T11:09:04 Search the mailing list at Gmane query http://search.gmane.org/?group=$group=gmane.comp.programming.garbage-collection.boehmgc