gmane.comp.security.openwall.john.user

Re: Core 2 Duo benchmarks JtR 1.7.3.1

Dhirendra Singh Kholia — 2008-07-23T03:35:55

.... did some further benchmarking. Intel Core 2 Duo E6750 < at >3.6GHz Linux kernel 2.6.24 GCC 4.2.3 [64-bit] make linux-x86-64 Benchmarking: Traditional DES [128/128 BS SSE2-16]... DONE Many salts:3486K c/s real, 3493K c/s virtual Only one salt:2989K c/s real, 2995K c/s virtual Benchmarking: FreeBSD MD5 [32/64 X2]... DONE Raw:15580 c/s real, 15580 c/s virtual Benchmarking: OpenBSD Blowfish (x32) [32/64 X2]... DONE Raw:943 c/s real, 945 c/s virtual Benchmarking: LM DES [128/128 BS SSE2-16]... DONE Raw:17754K c/s real, 17719K c/s virtual * Thanks Solar Designer, I have now finally shifted to 64-bit distro. * Memory BW and Latency didn't make any difference in the benchmarking. Wondering if latest gcc 4.3.1 or gcc 4.4.x branch would make a difference? < Added results to wiki >

Re: testing on little/big-endian and 32/64-bit (was: raw-md5 module improvement)

Kasza Péter — 2008-07-22T16:32:36

I've resolved the endianness issues (tested on Mac OS X with i386, x86-64 and ppc arches), plus made some other small improvements. Kasza Peter

Re: Core 2 Duo benchmarks JtR 1.7.3.1

Solar Designer — 2008-07-20T21:39:11

Thanks. Please add this benchmark to the wiki: http://openwall.info/wiki/john/benchmarks ... Why did you omit "Traditional DES"?.. That one is the "sorting key" for the table on the wiki. BTW, you'd get much better performance by going for a 64-bit Linux distro and make target. Please refer to this thread: http://www.openwall.com/lists/john-users/2008/06/01/1 (click "thread-next" to see other messages). Alexander

Re: Core 2 Duo benchmarks JtR 1.7.3.1

Sebastian Rother — 2008-07-20T20:56:10

Please do talk to Xorg developers and ATI/nVidia and explain them you wanna get a GENERIC framework. Using cuda wont work. Cuda is not supported on most Systems. Neither is ATIs stream. Xorg is the only alternative to get a universal framework somehow. But this needs modifications to the specific drivers. My oppinion pls. right me if I am wrong. ATI did not answered so far to my requests but the Xorg developers should have better contacts to ATI somebody told me. If you think about offloading the workd to the GPU please consider a generic framework so Most OSs participate, except of Windows but MS plans a own generic framework too! Kind regards, Sebastian

Core 2 Duo benchmarks JtR 1.7.3.1

Dhirendra Singh Kholia — 2008-07-20T13:17:05

Hi, CONFIGURATION: John-1.7.3.1, Kernel 2.6.26, GCC 4.2.3 on Core 2 Duo 6750 < at >3.4GHz [32-bit] Benchmarking: BSDI DES (x725) [128/128 BS SSE2]... DONE Many salts:94515 c/s real, 94515 c/s virtual Only one salt:91673 c/s real, 92041 c/s virtual Benchmarking: FreeBSD MD5 [32/32]... DONE Raw:9370 c/s real, 9370 c/s virtual Benchmarking: OpenBSD Blowfish (x32) [32/32]... DONE Raw:574 c/s real, 574 c/s virtual Benchmarking: LM DES [128/128 BS SSE2]... DONE Raw:15534K c/s real, 15565K c/s virtual Regards, dsk PS: Is there any work going on in utilizing GPUs (using CUDA, CTM) in JtR for password cracking?

Re: JtR 1.7.3.1

Henri Salo — 2008-07-19T21:52:31

On Sat, 19 Jul 2008 23:37:50 +0400 Solar Designer public.gmane.org> wrote: My system is GNU/Linux Debian etch 2.6.18-6-amd64. Hardware is Q6600 with 7907M of ram. I noticed same kind of improvement on another amd64-machine in c/s-rate. John was configured with linux-x86-64-option. You are correct. I only have few (1-5) hashes usually. Maybe I did something different or something has changed (in OS or hardware) since last time. I built last version of john again with same option and didn't recognise any difference on memory usage. At the moment those processes are still taking only 6M of memory.

Re: JtR 1.7.3.1

Solar Designer — 2008-07-19T19:37:50

If you were using the same make target before, then there should be no improvement for this hash type between 1.7.2 and 1.7.3.1. However, if you were using a "plain x86" make target, then going for a -x86-64 one should improve performance at this hash type greatly (maybe by 50%, although this differs across CPUs). -x86-64 targets were not previously available for Mac OS X and Solaris, so if you're using one of those systems then this can be regarded as a speedup with the new version. That's weird. It should not have been taking 20 MB before, unless you were loading a lot of hashes (tens of thousands), but you're not now. You never mentioned what OS/version/hardware this applies to. Your mail headers and "ps" command options suggest Linux, but like I mentioned x86-64 was fully supported under Linux before. And it's a bit tricky to build a "plain x86" binary under typical Linux distributions for x86-64 (there exist undocumented/unlisted make targets for that; I doubt that you've been using one of those). So while I am happy that you like the new version, I am really puzzled by your comments. Something must have gone wrong on your end when you were trying the previous version out. Thanks, Alexander

benchmarks on the wiki

Solar Designer — 2008-07-19T19:22:10

Hi, I've added many new benchmarks to the wiki, including the current winner entry and many non-x86 entries. Also, Erik contributed a curious entry - the very last one. http://openwall.info/wiki/john/benchmarks The fastest system in the table is currently based on a Q6700 CPU running a 64-bit Linux distribution. This one achieves an equivalent of just over 2.5 million of traditional DES-based crypt(3) checks per second per core, but since this CPU is quad-core, this translates to over 10 million of checks per second per CPU chip, with proper parallelization. The slowest is an iPod, which does around one thousand of traditional crypt(3)'s per second. If you have a system substantially different from those listed, please submit your results. Thanks, Alexander

Re: JtR 1.7.3.1

Henri Salo — 2008-07-19T14:47:20

On Sat, 19 Jul 2008 16:54:32 +0400 Solar Designer public.gmane.org> wrote: I noticed a big improvement on FreeBSD MD5 [32/64 X2]. I have no idea why it takes less memory, but it used to take aroun 20M per proccess and now it is like 6M. I'm using ps -eorss,cmd I don't really care about it, but less is better.

Re: JtR 1.7.3.1

Solar Designer — 2008-07-19T12:54:32

You're welcome, but the new version is not supposed to take less memory - this must be a glitch in your testing/measurements. Also, I am surprised that you care about memory consumption - JtR does not take much memory, considering today's typical RAM sizes. It's just around 7 MB in "incremental mode" (and a lot less in other modes) plus whatever it takes to load the password files (of course, it can be twice the size of password files or so because the in-memory "password database" has some structure and because it stores the hashes in both "source" and "binary" forms, although it also omits some irrelevant data). As to the speedup with this new version, it depends on hash type and make target. For some combinations of these, there's speedup of up to 65% (in the tests I ran), but for others there's no speedup (e.g., with 32-bit *-x86-* targets, the same code is used, so there should be no change). Alexander

Re: JtR 1.7.3.1

Henri Salo — 2008-07-18T18:02:10

On Fri, 18 Jul 2008 16:08:40 +0400 Solar Designer public.gmane.org> wrote: This new version takes a lot less memory than last one. It also seems much faster. Thank you a lot!

Re: JtR 1.7.3.1

Solar Designer — 2008-07-18T14:17:30

Hi Sebastian, On Fri, Jul 18, 2008 at 02:41:25PM +0200, Sebastian Rother wrote: Indeed, but unfortunately not soon. So far, "stable" releases of JtR assumed testing on a lot of platforms and pre-building for Windows and DOS, which I did not do this time. (I did not even try compiling this release on Windows, although that should work.) Also, the plan has been to make major releases of JtR infrequently, and to issue minor updates to those for critical bugs only (this was the case for 1.7.0.1 and 1.7.0.2, although I also included some reliability enhancements along with the bugfixes). What if the web page explicitly says what it does now? - In practice, 1.7.2 (a "development" version) has been around for a long time now, and it has been found to be at least as stable as 1.7.0.2, so the use of 1.7.2 is recommended. I can't say that of 1.7.3.1 yet, but I hope that eventually I will. Any Itanium. Alpha 21264 (EV6) or newer. Pretty much anything "recent". Also, testing with very recent gcc (such as 4.3.1) and with other C compilers on RISC architectures makes sense. You may either use "make generic", which autodetects BF_X2, or (re)set BF_X2 in the .h file for your arch manually. Then let me know whether BF_X2=0 or BF_X2=1 works better (and by how much) for your specific CPU, compiler, version. I've already tested on UltraSPARC IIi with gcc up to 3.4.5, so I am interested in tests on newer UltraSPARC and SPARC64 CPUs and/or with newer/other compilers. Yes, I did mention that this optimization is most effective for CPUs that can issue more than 2 integer instructions per cycle, but in practice things are more complicated - it's not only about peak issue rate, but it's also about latencies. So it could make sense on some dual-issue and even single-issue CPUs as well, if the compiler generates good code (those versions of gcc failed at that) and if some of the instructions involved have large latencies. Thanks, Alexander

JtR Pro 1.7.3+ for Linux and Mac OS X

Solar Designer — 2008-07-18T12:52:40

Hi, This is likely the last announcement posting for today, and maybe for this month. It is to announce availability of John the Ripper 1.7.3 Pro for Linux (stable release) and 1.7.3.1 Pro for Mac OS X (currently in public beta). The corresponding URLs are: http://www.openwall.com/john/pro/linux/ http://www.openwall.com/john/pro/macosx/ I've included more detail on each of these two versions below: 1. Besides the update to 1.7.3 (and thus including all of the improvements of that version in an extensively tested native package), new with John the Ripper 1.7.3 Pro for Linux is official support for Windows NTLM (MD4-based) and Mac OS X 10.4+ salted SHA-1 hashes. Also included for the first time is a pre-built RPM package for 64-bit capable systems, which makes use of 64-bit mode extended SSE2. This is a free upgrade for everyone who has purchased the product in the past (the new packages are found in the existing download directories). As usual, new copies/licenses may be purchased at: http://www.openwall.com/john/pro/linux/ I'd like to thank Alain Espinosa for the optimized NTLM code, and for kindly placing it in the public domain. This release of JtR Pro includes Alain's code with slight modifications, as well as replacement code for the password file loader; I am going to roll these into the next revision of the jumbo patch. 2. There's a beta version of John the Ripper Pro 1.7.3.1 for Mac OS X included with every new purchase, and free for everyone who has purchased the product in the past (it is found in the beta/ subdirectory in the existing download directories). As usual, new copies/licenses may be purchased at: http://www.openwall.com/john/pro/macosx/ Similarly to JtR Pro for Linux, besides the update to 1.7.3+, this version adds official support for Windows NTLM (MD4-based) and Mac OS X 10.4+ salted SHA-1 hashes, and it includes native support for 64-bit capable Intel CPUs on Mac OS X 10.5+ (Leopard) within the Universal binary, which makes use of 64-bit mode extended SSE2. Also included is the brand new XPWDUMP tool, which dumps password hashes from Mac OS X systems for subsequent auditing/cracking. As a bonus, the full source code is also provided (it was not provided with 1.7.2 Pro for Mac OS X, the focus of which was on packaging, but with so many exciting new features, we're also being generous and we do share the revised and enhanced source code this time, including several added source files). A "final" (non-beta) release of 1.7.3.1 Pro (or newer) for Mac OS X is coming soon. This will be a free upgrade for everyone who has purchased the product. I'd like to thank Jen Linkova aka Furry for her research into proper packaging for Mac OS X, and for XPWDUMP. Alexander Peslyak GPG key ID: 5B341F15 fp: B3FB 63F4 D7A3 BCCC 6F6E FC55 A2FC 027C 5B34 1F15 http://www.openwall.com - bringing security into open computing environments

Re: JtR 1.7.3.1

Sebastian Rother — 2008-07-18T12:41:25

Hi Solar, At first I'd like to tell you: A realy great release :-) But I'd like to ask if you may could decide to release a new "stable" version some time. The problem with the ports-maintainer on OpenBSD is, that he wont include "unstable" (aka development) versions so far. In case you could do it it would be great. Otherwise I continiue to build from source. :) Kind regards, Sebastian p.s. Could you name some CPUs wich would help you? You wrote something about RISC and 2 int instruction per clock. Could you name some SPARC CPUs in case you know some? I may could ask somebody to do the test in case I'd know the SPARC-CPU wich is capable of this.

JtR 1.7.3.1

Solar Designer — 2008-07-18T12:08:40

Hi, This is to announce availability of John the Ripper 1.7.3.1, which is a "development" version, from the usual location: http://www.openwall.com/john/ John the Ripper 1.7.3 and 1.7.3.1 focus on better x86-64 support. Most notably, two Blowfish-based crypt(3) hashes may now be computed in parallel for much better performance on x86-64 CPUs (on Intel Core 2 and AMD CPUs tested, the speedup was 50% to 65%), and new make targets have been added for Mac OS X 10.5+ (Leopard) and recent versions of Solaris on 64-bit capable x86 processors, producing 64-bit builds that make use of the 64-bit mode extended SSE2 for DES-based hashes. Previously, such "fully native" builds were only supported on Linux and *BSDs. This addition has required making the assembly source files more portable, including conversion to instruction pointer relative addressing on x86-64 as it is required for Mac OS X and avoiding constructs not supported by Sun's assembler. Hopefully, this has not made things any worse for any other systems - and some testing has been done to confirm this, but further testing is desired. A number of other make targets have been added as well, including for 32-bit with SSE2 and 32-bit with MMX builds on Solaris (with Sun Studio C compiler or gcc), for Linux/IA64 (Itanium), and for building three-architecture Universal binaries with Xcode 3.0 on Mac OS X. I'd like to thank Anatoly Pugachev for his assistance with testing the Solaris builds. As a bonus, "DumbForce" and "KnownForce" external mode samples have been added to the default john.conf. DumbForce is a generic implementation of "dumb" exhaustive search, given a range of lengths and an arbitrary charset. The sample is pre-configured for trying 8-bit characters (except for known terminal controls) against LM hashes, but this can be changed easily. KnownForce is a generic implementation of exhaustive search for a partially-known password - arbitrary character sets or even fixed characters may be specified for each position separately. Wanted: native make target vs. "make generic" benchmarks on Itanium and on newer RISC processors (those capable of issuing more than 2 integer instructions per clock cycle), with gcc and with other compilers. Right now, the "dual Blowfish" code is only enabled for x86-64 (where it is known to work well) and IA64 (where it is expected to work well), but not for any other architectures (where at least some versions of gcc are known to produce poor code at least for some dual-issue RISC processors). Please post any follow-ups to this message to the john-users mailing list (be sure to join the list first if you're not on it already). I will post another message on JtR Pro updates. Thanks, Alexander Peslyak GPG key ID: 5B341F15 fp: B3FB 63F4 D7A3 BCCC 6F6E FC55 A2FC 027C 5B34 1F15 http://www.openwall.com - bringing security into open computing environments

testing on little/big-endian and 32/64-bit (was: raw-md5 module improvement)

Solar Designer — 2008-07-16T13:28:26

Bucsay, Erik - On Tue, Jul 15, 2008 at 11:53:28PM -0400, Erik Winkler wrote: By the way, a convenient way to test on all four common combinations of endianness (little vs. big) and word size (32- vs. 64-bit) without having access to multiple machines at once is to use recent versions of Mac OS X and Xcode, which includes support for cross-compilation and for running binaries built for the other architecture (PPC vs. Intel) and word size. At least on my Core 2 Duo MacBook with Mac OS X 10.5 and Xcode 3.0, I am able to compile for all four combinations (by adding the proper "-arch ..." options to CFLAGS/ASFLAGS/LDFLAGS) and run all of the resulting binaries (with transparent emulation involved when needed). Also, "make macosx-universal" produces a three-arch binary, and I am able to choose which of the embedded binaries to run with "arch -... ./john", e.g. "arch -ppc ./john" will run the 32-bit PowerPC binary embedded in the universal binary. (There's some known breakage of macosx-x86-* and macosx-universal targets in the 1.7.3 "development" release; I will repair this in 1.7.3.1 shortly.) Alexander

Re: raw-md5 module improvement

Erik Winkler — 2008-07-16T03:53:28

Ran this 1.7.3 with the rawMD5 patch on powerpc and I get the following error now: Benchmarking: Raw MD5 [raw-md5]... FAILED (cmp_exact) Erik

Re: raw-md5 module improvement

Bucsay Balázs — 2008-07-15T17:28:22

Hello! Thx for the bugreport, i didnt have any time at the past, but now i revised it. I have to compile on big-endian machines. I fixed too, the hash-table thing in the patches of mine. Its work me fine. Urls in the right schemas (i hope so :) ): http://rycon.hu/tools/john-1.7.3-rawMD5_fast-2.diff http://rycon.hu/tools/john-1.7.3-MYSQL_fast-2.diff Balázs Bucsay http://rycon.hu/ Erik Winkler wrote:

Re: cracking md5 passwords not working

Bill Gurley — 2008-07-10T13:11:27

I managed to do this a while back in a not-very-elegant way. First you should make sure that you have a pretty "clean" file that does not have any empty entries, and has two columns of usernames and base64 strings. Then strip out the usernames, so that you have a second file with JUST the base64 strings. Then I made a script that works on the file that just has the base64 strings, converting them to hashes. The relevant command is something like this: perl -MMIME::Base64 -ne 'print decode_base64($_) Then you have to make a script that puts the uid's back with the hashes in the same order. (this is why you don't want any that have blank passwords in the list.) Sorry, but I don't have my scripts cleaned up and in a format that I'm comfortable posting here, but I'll try to do that soon and post it. Once you have the file with ssha hashes instead of base64, then you can use jtr on it. -Bill- --------------------------------- Bill Gurley, Technical Director Department of Chemistry Univ. of Tennessee, Knoxville 865-974-3145

Re: cracking md5 passwords not working

ram — 2008-07-10T06:25:51

On Thu, 2008-07-10 at 07:47 +0200, Simon Marechal wrote: Sorry maybe off topic here , how do I convert base64 to hex. I am searching for some perl module .. cant get

Re: cracking md5 passwords not working

Simon Marechal — 2008-07-10T05:47:49

Yes, by converting the base64 encoding into hex encoding. Or you can write your own plugin.