http://permalink.gmane.org/gmane.comp.db.leveldb hourly 1 1901-01-01T00:00+00:00 http://gmane.org/img/gmane-25t.png http://gmane.org http://permalink.gmane.org/gmane.comp.db.leveldb/1522 <pre> Comment #3 on issue 169 by DavidJoelSchwartz&lt; at &gt;gmail.com: Leveldb keeps generating small sst file http://code.google.com/p/leveldb/issues/detail?id=169 I believe I'm seeing this too. ... 270 small files created 03:13:21.412464 Generated table #1073824: 13 keys, 7209 bytes 03:13:21.504346 Generated table #1073825: 15 keys, 8345 bytes 03:13:21.597557 Generated table #1073826: 4 keys, 2437 bytes 03:13:21.679429 Generated table #1073827: 9 keys, 4581 bytes 03:13:21.771315 Generated table #1073828: 6 keys, 2784 bytes 03:13:21.853079 Generated table #1073829: 12 keys, 5856 bytes 03:13:21.934807 Generated table #1073830: 10 keys, 13350 bytes 03:13:22.016480 Generated table #1073831: 11 keys, 6459 bytes 03:13:22.108478 Generated table #1073832: 10 keys, 5504 bytes 03:13:22.200398 Generated table #1073833: 17 keys, 17318 bytes 03:13:22.282151 Generated table #1073834: 13 keys, 23513 bytes 03:13:22.384259 Generated table #1073835: 9 keys, 4937 bytes 03:13:22.466041 Generated table #1073836: 7 keys, 3846 bytes 03:1</pre> leveldb< at >googlecode.com 2013-05-17T13:23:18 http://permalink.gmane.org/gmane.comp.db.leveldb/1521 <pre> I'm seeing large amounts of compaction and high read or write latency with LevelDB. I noticed just recently that two issues were opened that seem to relate to a very similar problem: http://code.google.com/p/leveldb/issues/detail?id=164 http://code.google.com/p/leveldb/issues/detail?id=168 The issue only seems to arise when a background compaction is in progress and it is worse on I/O constrained machines. Frequently, the background compaction thread is blocked in 'fdatasync' and the read thread is blocked in 'ReadBlock' . I've also noticed huge bursts of compaction (going on for several minutes) shortly after a database is opened. Like the people who opened those issues, I'm using LevelDB 1.9.0 on Linux. Filesystem is ext4, scheduler is deadline. Any hints? Any OS, filesystem, or LevelDB tuning I should be looking at? DS </pre> David Schwartz 2013-05-16T00:24:07 http://permalink.gmane.org/gmane.comp.db.leveldb/1520 <pre>Status: New Owner: ---- Labels: Type-Defect Priority-Medium New issue 170 by wangteng...&lt; at &gt;gmail.com: oen advice about snapshot http://code.google.com/p/leveldb/issues/detail?id=170 What steps will reproduce the problem? 1.all code adn information showed in the additional information 2. 3. What is the expected output? What do you see instead? What version of the product are you using? On what operating system? leveldb 1.9 fedora17 Please provide any additional information below. code begin================================================= const leveldb::Snapshot* snap_shot = db-&gt;GetSnapshot(); leveldb::ReadOptions read_options; // change "key0"'s value from "value0" to "new" db-&gt;Put(leveldb::WriteOptions(),"key0","new"); string val; db-&gt;Get(read_options,"key0",&amp;val); cout&lt;&lt;"no snapshot value: "&lt;&lt;val&lt;&lt;endl;//output " new" . read_options.snapshot = snap_shot; db-&gt;Get(read_options,"key0",&amp;val); cout&lt;&lt;"snapshot value: "&lt;&lt;val&lt;&lt;endl;//output "value0" . //</pre> leveldb< at >googlecode.com 2013-05-15T11:37:39 http://permalink.gmane.org/gmane.comp.db.leveldb/1519 <pre> Comment #2 on issue 169 by wuzuy...&lt; at &gt;gmail.com: Leveldb keeps generating small sst file http://code.google.com/p/leveldb/issues/detail?id=169 I see this line in leveldb doc: ```We also switch to a new output file when the key range of the current output file has grown enough to overlap more then ten level-(L+2) files.``` I assume that the only file in level-2 overlaps many files in level-4, so it splits into 32 level-3 files. I don't think this mechanism is good, because later Get() operations will force seek_compactions on all these level-3 files to be merged with level-4 files. </pre> leveldb< at >googlecode.com 2013-05-15T05:39:22 http://permalink.gmane.org/gmane.comp.db.leveldb/1518 <pre> Comment #1 on issue 169 by wuzuy...&lt; at &gt;gmail.com: Leveldb keeps generating small sst file http://code.google.com/p/leveldb/issues/detail?id=169 This show how it happens: Compactions Level Files Size(MB) Time(sec) Read(MB) Write(MB) -------------------------------------------------- 0 0 0 0 0 11 2 1 39 2 0 118 3 0 0 2 90 88 4 386 10062 354 30228 30227 5 1798 53047 1897 158227 158183 Compactions Level Files Size(MB) Time(sec) Read(MB) Write(MB) -------------------------------------------------- 0 0 0 0 0 11 2 0 0 2 0 118 3 32 39 2 130 127 4 386 10062 354 30228 30227 5 1798 53047 </pre> leveldb< at >googlecode.com 2013-05-15T05:08:42 http://permalink.gmane.org/gmane.comp.db.leveldb/1517 <pre>Status: New Owner: ---- Labels: Type-Defect Priority-Medium New issue 169 by wuzuy...&lt; at &gt;gmail.com: Leveldb keeps generating small sst file http://code.google.com/p/leveldb/issues/detail?id=169 here is leveldb.stats ouputs: Compactions Level Files Size(MB) Time(sec) Read(MB) Write(MB) -------------------------------------------------- 0 0 0 0 0 36 2 0 0 9 0 519 3 30 4 12 594 580 4 530 10070 1187 101893 101892 5 1750 52946 7101 534959 534716 Level-3 has 30 files, but it only has 4MB size. Then these 30 files will be merged to level-4, but the newly created level-4 sst files is small too, I can see that with ls command. This leads to frequently compaction after written 4MB data. What is the expected output? What do you see instead? small sst file should be merged. What version of the product are you using? On what operating system? Linux Please provide any additional information below. kTargetFileSize = 32 * 1048576 </pre> leveldb< at >googlecode.com 2013-05-15T02:09:29 http://permalink.gmane.org/gmane.comp.db.leveldb/1516 <pre>Updates: Status: Fixed Comment #1 on issue 166 by dgrogan&lt; at &gt;chromium.org: assertion failed http://code.google.com/p/leveldb/issues/detail?id=166 (No comment was entered for this change.) </pre> leveldb< at >googlecode.com 2013-05-15T00:13:51 http://permalink.gmane.org/gmane.comp.db.leveldb/1515 <pre>Fixes issues 147 - thanks feniksgordonfreeman 153 156 166 Additionally, * Remove calls to exit(1). * Fix unused-variable warnings from clang. * Fix possible overflow error related to num_restart value &gt;= (2^32/4). * Add leveldbutil to .gitignore. * Add better log messages when Write is stalled on a compaction. </pre> David Grogan 2013-05-15T00:13:15 http://permalink.gmane.org/gmane.comp.db.leveldb/1514 <pre>Updates: Status: Fixed Comment #1 on issue 156 by dgrogan&lt; at &gt;chromium.org: "leveldbutil dump" on manifest produces truncated output if keys contain NUL http://code.google.com/p/leveldb/issues/detail?id=156 (No comment was entered for this change.) </pre> leveldb< at >googlecode.com 2013-05-15T00:12:51 http://permalink.gmane.org/gmane.comp.db.leveldb/1513 <pre>Updates: Status: Fixed Comment #1 on issue 153 by dgrogan&lt; at &gt;chromium.org: /dev/null gets deleted during compilation on some systems http://code.google.com/p/leveldb/issues/detail?id=153 (No comment was entered for this change.) </pre> leveldb< at >googlecode.com 2013-05-15T00:11:51 http://permalink.gmane.org/gmane.comp.db.leveldb/1512 <pre>Updates: Status: Fixed Comment #1 on issue 147 by dgrogan&lt; at &gt;chromium.org: Handling multiworld variables (CC and others) http://code.google.com/p/leveldb/issues/detail?id=147 (No comment was entered for this change.) </pre> leveldb< at >googlecode.com 2013-05-15T00:10:51 http://permalink.gmane.org/gmane.comp.db.leveldb/1511 <pre>Status: New Owner: ---- Labels: Type-Defect Priority-Medium New issue 168 by wuzuy...&lt; at &gt;gmail.com: [Feature Required] Compaction read/write speed limit http://code.google.com/p/leveldb/issues/detail?id=168 We are running SSDB(a leveldb server, https://github.com/ideawu/ssdb), with 28G data total, 400k updates per day. LevelDB compaction happends 6-10 times per day. While compaction is in progress, all reads and writes made to leveldb is significant slow. The disk IO is almost full, this is the reason(I think) why leveldb slows down. What is the expected output? What do you see instead? Compaction won't slow down normal operations, leveldb's compaction thread should limit the disk read/write speed. What version of the product are you using? On what operating system? 1.9.0 Please provide any additional information below. </pre> leveldb< at >googlecode.com 2013-05-14T03:52:40 http://permalink.gmane.org/gmane.comp.db.leveldb/1510 <pre> The C++ implevementation never produces a block with zero restart points. Hence the assertion. It seems like the port to Java behaves differently? In any case, it is fine to make the C++ code more resilient and we have made a corresponding change in the C++ implementation so it deals with such blocks. The fix was slightly more involved than your suggestion, and also involved a fix to a potential overflow problem. The next release will contain the fix. Thanks. </pre> Sanjay Ghemawat 2013-05-13T19:56:56 http://permalink.gmane.org/gmane.comp.db.leveldb/1509 <pre>I don't know this for sure, but you can try to compile by: *CFLAGS="-arch x86_64" make *and force it to compile with the architecture. Oliver On Saturday, May 11, 2013 6:01:56 AM UTC-4, HA YOUNG PARK wrote: </pre> Oliver Wang 2013-05-13T19:42:12 http://permalink.gmane.org/gmane.comp.db.leveldb/1508 <pre> Comment #1 on issue 167 by kkooporation&lt; at &gt;googlemail.com: Error if directory we're trying to create the database in doesn't exist http://code.google.com/p/leveldb/issues/detail?id=167 Note as requested: this is a specific feature-request and does not apply to the broad LevelDB community. Also, this is mostly for convenience. </pre> leveldb< at >googlecode.com 2013-05-11T11:21:57 http://permalink.gmane.org/gmane.comp.db.leveldb/1507 <pre>Hi, I am new to levelDB and just trying to compile/run a simple example included in a tutorial after having compiled leveldb from a source successfully. Unfortunately, I get link error complaining leveldb::DB::Open() can't be found: Undefined symbols for architecture x86_64: "leveldb::DB::Open(leveldb::Options const&amp;, std::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;, leveldb::DB**)", referenced from: _main in cchhRCqZ.o ld: symbol(s) not found for architecture x86_64 I've googled about this but ended up just finding someone else having the same issue that hasn't been solved yet. http://ubuntuforums.org/showthread.php?t=2126618 I'm so fooled as it appears nothing wrong after thoroughly inspecting the source files such as db_impl.cc in which DB::Open() is implemented. Could anyone help me get though this? I'm on Mac OS X, and the g++ command options are listed below: OPTS = -v -I $(LEVELDB_DIR)/include -lpthread $(LEVELDB_DIR)/libleveldb.dylib Thank you in adv</pre> HA YOUNG PARK 2013-05-11T10:01:56 http://permalink.gmane.org/gmane.comp.db.leveldb/1506 <pre>Status: New Owner: ---- Labels: Type-Defect Priority-Medium New issue 167 by kkooporation&lt; at &gt;googlemail.com: Error if directory we're trying to create the database in doesn't exist http://code.google.com/p/leveldb/issues/detail?id=167 What steps will reproduce the problem? 1. open a database 2. bazinga! (we should ditch the standard questions here..) /*What is the expected output? What do you see instead? What version of the product are you using? On what operating system? Please provide any additional information below.*/ When a database is being opened in a folder which itself does not exist, LevelDB errors with a dubious error message: OpenError: IO error: ./db/root.db/LOCK: No such file or directory at [..]/levelup.js:113:25 As this came up quite some times, we discussed the process here; either the path will recursively be created or instead an error being thrown indicating the specific issue. We've been talking about this issue on GitHub regarding LevelDOWN / UP [the NodeJS bindings</pre> leveldb< at >googlecode.com 2013-05-11T09:57:58 http://permalink.gmane.org/gmane.comp.db.leveldb/1505 <pre>Status: New Owner: ---- Labels: Type-Defect Priority-Medium New issue 166 by pmarc...&lt; at &gt;gmail.com: assertion failed http://code.google.com/p/leveldb/issues/detail?id=166 *What steps will reproduce the problem? 1. Create a leveldb database with this leveldb implementation (in my case, it was with the bitcoind client) 2. Open the database with the java version of leveldb (https://github.com/dain/leveldb) 3. Open the database with the C++ version of leveldb *What is the expected output? What do you see instead? The c++ leveldb will crash with an assert failure for table/block.cc, at line 19. IMHO, the code should be: assert(size_ &gt;= sizeof(uint32_t)); and table/block.cc, at line 256 should be if (size_ &lt; sizeof(uint32_t)) { *What version of the product are you using? On what operating system? 1.6.0 on Ubuntu Linux kernel 3.5.0-28-generic, x86_64 I changed the code and tested a bit and everything seems fine. Please let me know how this goes, so I can propagate the changes to the leveldb versi</pre> leveldb< at >googlecode.com 2013-05-11T06:15:54 http://permalink.gmane.org/gmane.comp.db.leveldb/1504 <pre>hi all, I did two db_bench runs: one on a Linux VM against a spinning disk, the other against a SATA3 SSD. The later (SSD) block level tests shows a consistent R/W bandwidth at 520MB/s. The fillseq (seq write) is quite good on spinning disk, but miserable on SSD (ext4). However, the seq-read on spinning disk seems to be an anomaly due to cache effect. My questions: 1. are there any explanations for the bad fillseq results on SSD? 2. are there ways to minimize the cache effect from db_bench? 3. Suppose I care only about aggregated write bandwidth, what are recommended settings for it? Best, Oliver // results against Spinning disk LevelDB: version 1.9 CPU: 2 * Intel(R) Core(TM) i7-2600 CPU &lt; at &gt; 3.40GHz CPUCache: 8192 KB Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) ------------------------------------------------ fillseq : 1.237 micros/op; 89.4 MB/s </pre> Oliver Wang 2013-05-10T15:54:02 http://permalink.gmane.org/gmane.comp.db.leveldb/1503 <pre>Please disregard this thread. After finding that all occurrences were on one machine out of our hundreds and seeing similar I/O errors from other applications on that machine, we've concluded that this is an I/O system issue. </pre> David Strauss 2013-05-10T00:38:59 http://permalink.gmane.org/gmane.comp.db.leveldb/1502 <pre>This issue completely goes away if we shut down the application, delete the LevelDB directory (which is just a cache for the application), and start the application up again (which creates a new LevelDB database in that directory). </pre> David Strauss 2013-05-09T22:55:12 Search the mailing list at Gmane query http://search.gmane.org/?group=$group=gmane.comp.db.leveldb