http://blog.gmane.org/gmane.comp.apache.mahout.devel hourly 1 1901-01-01T00:00+00:00 http://gmane.org/img/gmane-25t.png http://gmane.org http://permalink.gmane.org/gmane.comp.apache.mahout.devel/26210 <pre>On Wed, Jun 19, 2013 at 12:09 PM, Dan Filimon &lt;dangeorge.filimon&lt; at &gt;gmail.com&gt;wrote: It would still implement Vector? So you could pass in int indexes and get back values? And if you iterated over it, you'd iterate over Vector.Element objects, which have int keys... I'm not sure this would be as simple as you imagine it would... Yeah, we could pull up all the codegen from the mahout-collections up to Vector. That's a possibility. Integrating that with the Mapreduce code might be a little bit complex, but maybe not. </pre> Jake Mannix 2013-06-19T19:42:10 http://permalink.gmane.org/gmane.comp.apache.mahout.devel/26209 <pre>I understand that it might not be worth it to have all vectors have 64bit indices all the time. My use case is very sparse vectors, that have IPs in them for example. A particular vector will likely have at most 20 IPs, but since Java doesn't have unsigned, even IPs need to be hashed (and they're 32bit! :). As for actual hashes, for the data I'm working with, I see 0.4% of the data being lost because of collisions. Granted, this is not much, but there's also not that much data... It may also be true that the hash function I'm using isn't the best (Java's hashCode() (on an unrelated note, any suggestions for a better one?). Sean, is this structure you're using available anywhere? What I'm proposing would not change all the Vectors (at least not at once). For instance, I'm thinking of a SequentialAccesSparseLongVector class. This would be straightforward, as we only need an OrderedLongDoubleMapping (which could be templatized like other math containers) and a long size. It could still support the same interface as Vector but add a few more functions... This would be my hack. :) As for a more long-term solution, perhaps also templatizing Vector to support ints and longs and have the primitive classes generated at compile time? On Wed, Jun 19, 2013 at 9:22 PM, Jake Mannix &lt;jake.mannix&lt; at &gt;gmail.com&gt; wrote: </pre> Dan Filimon 2013-06-19T19:09:23 http://permalink.gmane.org/gmane.comp.apache.mahout.devel/26208 <pre>See &lt;https://builds.apache.org/job/Mahout-Examples-Cluster-Reuters-II/517/&gt; ------------------------------------------ [...truncated 6787 lines...] INFO: File Output Format Counters Jun 19, 2013 6:29:19 PM org.apache.hadoop.mapred.Counters log INFO: Bytes Written=6742193 Jun 19, 2013 6:29:19 PM org.apache.hadoop.mapred.Counters log INFO: FileSystemCounters Jun 19, 2013 6:29:19 PM org.apache.hadoop.mapred.Counters log INFO: FILE_BYTES_READ=2593987492 Jun 19, 2013 6:29:19 PM org.apache.hadoop.mapred.Counters log INFO: FILE_BYTES_WRITTEN=2122540279 Jun 19, 2013 6:29:19 PM org.apache.hadoop.mapred.Counters log INFO: File Input Format Counters Jun 19, 2013 6:29:19 PM org.apache.hadoop.mapred.Counters log INFO: Bytes Read=13093153 Jun 19, 2013 6:29:19 PM org.apache.hadoop.mapred.Counters log INFO: Map-Reduce Framework Jun 19, 2013 6:29:19 PM org.apache.hadoop.mapred.Counters log INFO: Map output materialized bytes=6404538 Jun 19, 2013 6:29:19 PM org.apache.hadoop.mapred.Counters log INFO: Map input records=21578 Jun 19, 2013 6:29:19 PM org.apache.hadoop.mapred.Counters log INFO: Reduce shuffle bytes=0 Jun 19, 2013 6:29:19 PM org.apache.hadoop.mapred.Counters log INFO: Spilled Records=40 Jun 19, 2013 6:29:19 PM org.apache.hadoop.mapred.Counters log INFO: Map output bytes=6689280 Jun 19, 2013 6:29:19 PM org.apache.hadoop.mapred.Counters log INFO: Total committed heap usage (bytes)=1663172608 Jun 19, 2013 6:29:19 PM org.apache.hadoop.mapred.Counters log INFO: CPU time spent (ms)=0 Jun 19, 2013 6:29:19 PM org.apache.hadoop.mapred.Counters log INFO: SPLIT_RAW_BYTES=120 Jun 19, 2013 6:29:19 PM org.apache.hadoop.mapred.Counters log INFO: Combine input records=20 Jun 19, 2013 6:29:19 PM org.apache.hadoop.mapred.Counters log INFO: Reduce input records=20 Jun 19, 2013 6:29:19 PM org.apache.hadoop.mapred.Counters log INFO: Reduce input groups=20 Jun 19, 2013 6:29:19 PM org.apache.hadoop.mapred.Counters log INFO: Combine output records=20 Jun 19, 2013 6:29:19 PM org.apache.hadoop.mapred.Counters log INFO: Physical memory (bytes) snapshot=0 Jun 19, 2013 6:29:19 PM org.apache.hadoop.mapred.Counters log INFO: Reduce output records=20 Jun 19, 2013 6:29:19 PM org.apache.hadoop.mapred.Counters log INFO: Virtual memory (bytes) snapshot=0 Jun 19, 2013 6:29:19 PM org.apache.hadoop.mapred.Counters log INFO: Map output records=20 Jun 19, 2013 6:29:19 PM org.slf4j.impl.JCLLoggerAdapter info INFO: About to run iteration 20 of 20 Jun 19, 2013 6:29:19 PM org.slf4j.impl.JCLLoggerAdapter info INFO: About to run: Iteration 20 of 20, input path: /tmp/mahout-work-jenkins/reuters-lda-model/model-19 Jun 19, 2013 6:29:19 PM org.apache.hadoop.mapreduce.lib.input.FileInputFormat listStatus INFO: Total input paths to process : 1 Jun 19, 2013 6:29:19 PM org.apache.hadoop.mapred.JobClient monitorAndPrintJob INFO: Running job: job_local_0020 Jun 19, 2013 6:29:19 PM org.apache.hadoop.mapred.Task initialize INFO: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin&lt; at &gt;44b7f7 Jun 19, 2013 6:29:19 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer &lt;init&gt; INFO: io.sort.mb = 100 Jun 19, 2013 6:29:20 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer &lt;init&gt; INFO: data buffer = 79691776/99614720 Jun 19, 2013 6:29:20 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer &lt;init&gt; INFO: record buffer = 262144/327680 Jun 19, 2013 6:29:20 PM org.slf4j.impl.JCLLoggerAdapter info INFO: Retrieving configuration Jun 19, 2013 6:29:20 PM org.slf4j.impl.JCLLoggerAdapter info INFO: Initializing read model Jun 19, 2013 6:29:20 PM org.slf4j.impl.JCLLoggerAdapter info INFO: Initializing write model Jun 19, 2013 6:29:20 PM org.slf4j.impl.JCLLoggerAdapter info INFO: Initializing model trainer Jun 19, 2013 6:29:20 PM org.slf4j.impl.JCLLoggerAdapter info INFO: Starting training threadpool with 4 threads Jun 19, 2013 6:29:20 PM org.apache.hadoop.mapred.JobClient monitorAndPrintJob INFO: map 0% reduce 0% Jun 19, 2013 6:29:25 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate INFO: Jun 19, 2013 6:29:26 PM org.apache.hadoop.mapred.JobClient monitorAndPrintJob INFO: map 27% reduce 0% Jun 19, 2013 6:29:28 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate INFO: Jun 19, 2013 6:29:29 PM org.apache.hadoop.mapred.JobClient monitorAndPrintJob INFO: map 41% reduce 0% Jun 19, 2013 6:29:31 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate INFO: Jun 19, 2013 6:29:32 PM org.apache.hadoop.mapred.JobClient monitorAndPrintJob INFO: map 55% reduce 0% Jun 19, 2013 6:29:34 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate INFO: Jun 19, 2013 6:29:34 PM org.apache.hadoop.mapred.JobClient monitorAndPrintJob INFO: map 70% reduce 0% Jun 19, 2013 6:29:37 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate INFO: Jun 19, 2013 6:29:37 PM org.apache.hadoop.mapred.JobClient monitorAndPrintJob INFO: map 84% reduce 0% Jun 19, 2013 6:29:40 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate INFO: Jun 19, 2013 6:29:40 PM org.apache.hadoop.mapred.JobClient monitorAndPrintJob INFO: map 99% reduce 0% Jun 19, 2013 6:29:41 PM org.slf4j.impl.JCLLoggerAdapter info INFO: Stopping model trainer Jun 19, 2013 6:29:41 PM org.slf4j.impl.JCLLoggerAdapter info INFO: Initiating stopping of training threadpool Jun 19, 2013 6:29:41 PM org.slf4j.impl.JCLLoggerAdapter info INFO: threadpool took: 23.765603ms Jun 19, 2013 6:29:42 PM org.slf4j.impl.JCLLoggerAdapter info INFO: readModel.stop() took 971.411152ms Jun 19, 2013 6:29:42 PM org.slf4j.impl.JCLLoggerAdapter info INFO: writeModel.stop() took 28.976268ms Jun 19, 2013 6:29:42 PM org.slf4j.impl.JCLLoggerAdapter info INFO: Writing model Jun 19, 2013 6:29:42 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer flush INFO: Starting flush of map output Jun 19, 2013 6:29:42 PM org.apache.hadoop.mapred.MapTask$MapOutputBuffer sortAndSpill INFO: Finished spill 0 Jun 19, 2013 6:29:42 PM org.apache.hadoop.mapred.Task done INFO: Task:attempt_local_0020_m_000000_0 is done. And is in the process of commiting Jun 19, 2013 6:29:42 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate INFO: Jun 19, 2013 6:29:42 PM org.apache.hadoop.mapred.Task sendDone INFO: Task 'attempt_local_0020_m_000000_0' done. Jun 19, 2013 6:29:42 PM org.apache.hadoop.mapred.Task initialize INFO: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin&lt; at &gt;4e7958 Jun 19, 2013 6:29:42 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate INFO: Jun 19, 2013 6:29:42 PM org.apache.hadoop.mapred.Merger$MergeQueue merge INFO: Merging 1 sorted segments Jun 19, 2013 6:29:42 PM org.apache.hadoop.mapred.Merger$MergeQueue merge INFO: Down to the last merge-pass, with 1 segments left of total size: 6402201 bytes Jun 19, 2013 6:29:42 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate INFO: Jun 19, 2013 6:29:42 PM org.apache.hadoop.mapred.JobClient monitorAndPrintJob INFO: map 100% reduce 0% Jun 19, 2013 6:29:43 PM org.apache.hadoop.mapred.Task done INFO: Task:attempt_local_0020_r_000000_0 is done. And is in the process of commiting Jun 19, 2013 6:29:43 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate INFO: Jun 19, 2013 6:29:43 PM org.apache.hadoop.mapred.Task commit INFO: Task attempt_local_0020_r_000000_0 is allowed to commit now Jun 19, 2013 6:29:43 PM org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter commitTask INFO: Saved output of task 'attempt_local_0020_r_000000_0' to /tmp/mahout-work-jenkins/reuters-lda-model/model-20 Jun 19, 2013 6:29:43 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate INFO: reduce &gt; reduce Jun 19, 2013 6:29:43 PM org.apache.hadoop.mapred.Task sendDone INFO: Task 'attempt_local_0020_r_000000_0' done. Jun 19, 2013 6:29:43 PM org.apache.hadoop.mapred.JobClient monitorAndPrintJob INFO: map 100% reduce 100% Jun 19, 2013 6:29:43 PM org.apache.hadoop.mapred.JobClient monitorAndPrintJob INFO: Job complete: job_local_0020 Jun 19, 2013 6:29:43 PM org.apache.hadoop.mapred.Counters log INFO: Counters: 20 Jun 19, 2013 6:29:43 PM org.apache.hadoop.mapred.Counters log INFO: File Output Format Counters Jun 19, 2013 6:29:43 PM org.apache.hadoop.mapred.Counters log INFO: Bytes Written=6742193 Jun 19, 2013 6:29:43 PM org.apache.hadoop.mapred.Counters log INFO: FileSystemCounters Jun 19, 2013 6:29:43 PM org.apache.hadoop.mapred.Counters log INFO: FILE_BYTES_READ=2731420291 Jun 19, 2013 6:29:43 PM org.apache.hadoop.mapred.Counters log INFO: FILE_BYTES_WRITTEN=2234553269 Jun 19, 2013 6:29:43 PM org.apache.hadoop.mapred.Counters log INFO: File Input Format Counters Jun 19, 2013 6:29:43 PM org.apache.hadoop.mapred.Counters log INFO: Bytes Read=13093153 Jun 19, 2013 6:29:43 PM org.apache.hadoop.mapred.Counters log INFO: Map-Reduce Framework Jun 19, 2013 6:29:43 PM org.apache.hadoop.mapred.Counters log INFO: Map output materialized bytes=6402205 Jun 19, 2013 6:29:43 PM org.apache.hadoop.mapred.Counters log INFO: Map input records=21578 Jun 19, 2013 6:29:43 PM org.apache.hadoop.mapred.Counters log INFO: Reduce shuffle bytes=0 Jun 19, 2013 6:29:43 PM org.apache.hadoop.mapred.Counters log INFO: Spilled Records=40 Jun 19, 2013 6:29:43 PM org.apache.hadoop.mapred.Counters log INFO: Map output bytes=6689280 Jun 19, 2013 6:29:43 PM org.apache.hadoop.mapred.Counters log INFO: Total committed heap usage (bytes)=1487536128 Jun 19, 2013 6:29:43 PM org.apache.hadoop.mapred.Counters log INFO: CPU time spent (ms)=0 Jun 19, 2013 6:29:43 PM org.apache.hadoop.mapred.Counters log INFO: SPLIT_RAW_BYTES=120 Jun 19, 2013 6:29:43 PM org.apache.hadoop.mapred.Counters log INFO: Combine input records=20 Jun 19, 2013 6:29:43 PM org.apache.hadoop.mapred.Counters log INFO: Reduce input records=20 Jun 19, 2013 6:29:43 PM org.apache.hadoop.mapred.Counters log INFO: Reduce input groups=20 Jun 19, 2013 6:29:43 PM org.apache.hadoop.mapred.Counters log INFO: Combine output records=20 Jun 19, 2013 6:29:43 PM org.apache.hadoop.mapred.Counters log INFO: Physical memory (bytes) snapshot=0 Jun 19, 2013 6:29:43 PM org.apache.hadoop.mapred.Counters log INFO: Reduce output records=20 Jun 19, 2013 6:29:43 PM org.apache.hadoop.mapred.Counters log INFO: Virtual memory (bytes) snapshot=0 Jun 19, 2013 6:29:43 PM org.apache.hadoop.mapred.Counters log INFO: Map output records=20 Jun 19, 2013 6:29:43 PM org.slf4j.impl.JCLLoggerAdapter info INFO: Completed 20 iterations in 441 seconds Jun 19, 2013 6:29:43 PM org.slf4j.impl.JCLLoggerAdapter info INFO: Perplexities: () Jun 19, 2013 6:29:43 PM org.slf4j.impl.JCLLoggerAdapter info INFO: About to run: Writing final topic/term distributions from /tmp/mahout-work-jenkins/reuters-lda-model/model-20 to /tmp/mahout-work-jenkins/reuters-lda Jun 19, 2013 6:29:44 PM org.apache.hadoop.mapreduce.lib.input.FileInputFormat listStatus INFO: Total input paths to process : 1 Jun 19, 2013 6:29:44 PM org.slf4j.impl.JCLLoggerAdapter info INFO: About to run: Writing final document/topic inference from /tmp/mahout-work-jenkins/reuters-out-matrix/matrix to /tmp/mahout-work-jenkins/reuters-lda-topics Jun 19, 2013 6:29:44 PM org.apache.hadoop.mapred.Task initialize INFO: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin&lt; at &gt;1b9e7fc Jun 19, 2013 6:29:44 PM org.apache.hadoop.mapred.Task done INFO: Task:attempt_local_0021_m_000000_0 is done. And is in the process of commiting Jun 19, 2013 6:29:44 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate INFO: Jun 19, 2013 6:29:44 PM org.apache.hadoop.mapred.Task commit INFO: Task attempt_local_0021_m_000000_0 is allowed to commit now Jun 19, 2013 6:29:44 PM org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter commitTask INFO: Saved output of task 'attempt_local_0021_m_000000_0' to /tmp/mahout-work-jenkins/reuters-lda Jun 19, 2013 6:29:44 PM org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate INFO: Jun 19, 2013 6:29:44 PM org.apache.hadoop.mapred.Task sendDone INFO: Task 'attempt_local_0021_m_000000_0' done. Jun 19, 2013 6:29:44 PM org.apache.hadoop.mapred.JobClient$2 run INFO: Cleaning up the staging area file:/tmp/hadoop-jenkins/mapred/staging/jenkins963194369/.staging/job_local_0022 Jun 19, 2013 6:29:44 PM org.apache.hadoop.security.UserGroupInformation doAs SEVERE: PriviledgedActionException as:jenkins cause:org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory /tmp/mahout-work-jenkins/reuters-lda already exists Exception in thread "main" org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory /tmp/mahout-work-jenkins/reuters-lda already exists at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:137) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:949) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:912) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:912) at org.apache.hadoop.mapreduce.Job.submit(Job.java:500) at org.apache.mahout.clustering.lda.cvb.CVB0Driver.writeDocTopicInference(CVB0Driver.java:463) at org.apache.mahout.clustering.lda.cvb.CVB0Driver.run(CVB0Driver.java:339) at org.apache.mahout.clustering.lda.cvb.CVB0Driver.run(CVB0Driver.java:198) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.mahout.clustering.lda.cvb.CVB0Driver.main(CVB0Driver.java:534) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:195) Build step 'Execute shell' marked build as failure </pre> Apache Jenkins Server 2013-06-19T18:29:45 http://permalink.gmane.org/gmane.comp.apache.mahout.devel/26207 <pre>long keys are super useful for rows in a matrix (ids for documents), and basically free in terms of memory (only one per document), but then for symmetry we really do need them in the columns (keying on e.g. termId), which is a not-insubstantial cost, but possibly worth it. Our vectors would be (16* numNonZeroEntries) bytes in footprint. That's pretty hefty, but not too much more than 12. There are arguments that most of the time, we don't need double values either. Sometimes, we don't need values at all (boolean data), but we could certainly have special-purpose Vectors which carry no values and yet return 1d for when the key is present. But changing over all of our keys to long is a pretty big change. Is it worth it? On Wed, Jun 19, 2013 at 10:25 AM, Sean Owen &lt;srowen&lt; at &gt;gmail.com&gt; wrote: </pre> Jake Mannix 2013-06-19T18:22:37 http://permalink.gmane.org/gmane.comp.apache.mahout.devel/26206 <pre>Nick, thank you for the hints and poniters! I will check out the Breeze. Let me take a look. as far as collaboration, unfortunately i think the only way to go for me and my employer is to cut it, test it and then (after long negotiations with CEO) donate if accepted. They are ok with my small participation in OSS but I don't have permission to outwardly start a fundamentally new OSS stuff on my paid time any more. (They seemed to be OSS friendly when they were hiring me, but once i was in, the situation seemed to have reversed itself). On Wed, Jun 19, 2013 at 3:50 AM, Nick Pentreath &lt;nick.pentreath&lt; at &gt;gmail.com&gt;wrote: </pre> Dmitriy Lyubimov 2013-06-19T17:53:24 http://permalink.gmane.org/gmane.comp.apache.mahout.devel/26205 <pre>Thank you, Sebastian. Actually ALS flavours are indeed one of my first pragmatic goals -- i have also done a few customization for my employer -- so i probably will pragmatically pursue those customizations first. In particular, i do use Koren-Volinsky confidence weighting, but assume we still work with sparse observations and therefore sparse algebra of ALS-WR still applies. I provide fold-in routine for users with fewer than N observations and just new users thus adding incremental approach to learning. I also spend a lot of time of adaptive validation of weights and regularization (which is why my R prototypes are no longer sufficient here, actually, my prototype doesn't take the load of a midsize customer anymore.) On Wed, Jun 19, 2013 at 3:54 AM, Sebastian Schelter &lt;ssc&lt; at &gt;apache.org&gt; wrote: </pre> Dmitriy Lyubimov 2013-06-19T17:44:10 http://permalink.gmane.org/gmane.comp.apache.mahout.devel/26204 <pre>Thank you, Ted. On Wed, Jun 19, 2013 at 12:20 AM, Ted Dunning &lt;ted.dunning&lt; at &gt;gmail.com&gt; wrote: </pre> Dmitriy Lyubimov 2013-06-19T17:38:38 http://permalink.gmane.org/gmane.comp.apache.mahout.devel/26203 <pre>I use 64-bit keys for vector-like data structures, and indeed you may pay a cost in extra RAM, but it has a lot of benefits in simplicity mostly, and making the probability of hash collisions ignorable even at huge scale. I think it's worthwhile overall. On Wed, Jun 19, 2013 at 6:16 PM, Robin Anil &lt;robin.anil&lt; at &gt;gmail.com&gt; wrote: </pre> Sean Owen 2013-06-19T17:25:02 http://permalink.gmane.org/gmane.comp.apache.mahout.devel/26202 <pre>&lt;rant&gt; Which joker thought of removing uint from Java? &lt;/rant&gt; Dan, the cost of moving to 64 bit for the index is extra RAM usage. My experiments show that 32 bits is enough to hash down billions of features. Do we ever need such Quadrillions of features? Can Machine learning truly work at that scale. Think about these. Robin Anil | Software Engineer | +1 312 869 2602 | Google Inc. On Wed, Jun 19, 2013 at 5:16 AM, Dan Filimon &lt;dangeorge.filimon&lt; at &gt;gmail.com&gt;wrote: </pre> Robin Anil 2013-06-19T17:16:53 http://permalink.gmane.org/gmane.comp.apache.mahout.devel/26201 <pre>I have a JBlas version of our ALS solving code lying around [1], feel free to use it. Would also be interested to see the Spark port. -sebastian [1] https://github.com/sscdotopen/mahout-als/blob/jblas/math/src/main/java/org/apache/mahout/math/als/JBlasAlternatingLeastSquaresSolver.java On 19.06.2013 12:50, Nick Pentreath wrote: </pre> Sebastian Schelter 2013-06-19T10:54:47 http://permalink.gmane.org/gmane.comp.apache.mahout.devel/26200 <pre> [ https://issues.apache.org/jira/browse/MAHOUT-1266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Illecker resolved MAHOUT-1266. ------------------------------------- Resolution: Not A Problem -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira </pre> Martin Illecker (JIRA 2013-06-19T10:53:20 http://permalink.gmane.org/gmane.comp.apache.mahout.devel/26199 <pre>Hi Dmitriy I'd be interested to look at helping with this potentially (time permitting). I've recently been working on a port of Mahout's ALS implementation to Spark. I spent a bit of time thinking about how much of mahout-math to use. For now I found that using the Breeze linear algebra library I could get what I needed, ie DenseVector, SparseVector, DenseMatrix, all with in-memory multiply and solve that is backed by JBLAS (so very quick if you have the native libraries active). It comes with very nice "Matlab-like" syntax in Scala. So it ended up being a bit of a rewrite rather than a port of the Mahout code. The sparse matrix support is however a bit... well, sparse :) There is a CSC matrix and some operations but Sparse SVD is not there, and the solvers I think are not there just yet (in-core). But of course the linear algebra objects are not easily usable from Java due to the syntax and the heavy use of implicits. So for a fully functional Java API version that can use the vectors/matrices directly, the options would be to create a Java bridge to the Breeze vectors/matrices, or to instead look at using mahout-math to drive the linear algebra. In that case the Scala syntax would not be as nice, but some sugar can be added again using implicits for common operations (I've tested this a bit and it can work and probably be made reasonably efficient if copies are avoided in the implicit conversion). Anyway, I'd be happy to offer assistance. Nick On Wed, Jun 19, 2013 at 8:09 AM, Sebastian Schelter &lt;ssc&lt; at &gt;apache.org&gt; wrote: </pre> Nick Pentreath 2013-06-19T10:50:33 http://permalink.gmane.org/gmane.comp.apache.mahout.devel/26198 <pre> [ https://issues.apache.org/jira/browse/MAHOUT-1266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&amp;focusedCommentId=13687859#comment-13687859 ] Martin Illecker commented on MAHOUT-1266: ----------------------------------------- I read the javadoc of *DistributedRowMatrix.times* but in *MatrixMultiplicationJob*[1] is no hint that it's not calculating A * B. Sorry I thought *mahout matrixmult* would calculate A * B. Thanks for your time! [1] https://github.com/apache/mahout/blob/trunk/core/src/main/java/org/apache/mahout/math/hadoop/MatrixMultiplicationJob.java -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira </pre> Martin Illecker (JIRA 2013-06-19T10:49:20 http://permalink.gmane.org/gmane.comp.apache.mahout.devel/26197 <pre> [ https://issues.apache.org/jira/browse/MAHOUT-1266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&amp;focusedCommentId=13687841#comment-13687841 ] Sebastian Schelter commented on MAHOUT-1266: -------------------------------------------- Yes, the name of the job is misleading, but the javadoc says it computes A'*B. You can do A*B either by first transposing A via org.apache.mahout.math.hadoop.TransposeJob or by simply storing A' when you create your sequence files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira </pre> Sebastian Schelter (JIRA 2013-06-19T10:39:20 http://permalink.gmane.org/gmane.comp.apache.mahout.devel/26196 <pre> [ https://issues.apache.org/jira/browse/MAHOUT-1266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&amp;focusedCommentId=13687840#comment-13687840 ] Martin Illecker commented on MAHOUT-1266: ----------------------------------------- Thanks Sebastian, I got it. You are calculating A' * B not A * B. But then it's not a matrix multiplication defined as A * B! Normally when executing a matrix multiplication of matrix A and matrix B, you would believe to get A * B not A' * B? So, how can I do a matrix multiplication (A * B) in Mahout, because I think there is no way to get A * B from A' * B? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira </pre> Martin Illecker (JIRA 2013-06-19T10:35:22 http://permalink.gmane.org/gmane.comp.apache.mahout.devel/26195 <pre> [ https://issues.apache.org/jira/browse/MAHOUT-1267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sebastian Schelter updated MAHOUT-1267: --------------------------------------- Fix Version/s: 0.9 Don't think this is necessary for 0.8, pushing to 0.9 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira </pre> Sebastian Schelter (JIRA 2013-06-19T10:23:20 http://permalink.gmane.org/gmane.comp.apache.mahout.devel/26194 <pre> [ https://issues.apache.org/jira/browse/MAHOUT-1266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&amp;focusedCommentId=13687834#comment-13687834 ] Sebastian Schelter commented on MAHOUT-1266: -------------------------------------------- Martin, the magic trick here is that A is *never* explicitly transposed. There are various ways to compute a matrix multiplication, I'll show you row outer producter formulation on a small example: http://latex.codecogs.com/gif.latex?A%20%3D%20%5Cbegin%7Bbmatrix%7D%201%20%26%202%5C%5C%203%20%26%204%20%5Cend%7Bbmatrix%7D%20B%20%3D%5Cbegin%7Bbmatrix%7D%205%20%26%206%5C%5C%207%20%26%208%20%5Cend%7Bbmatrix%7D%20A%5ETB%20%3D%5Cbegin%7Bbmatrix%7D%2026%20%26%2030%5C%5C%2038%20%26%2044%20%5Cend%7Bbmatrix%7D%20%5C%5C%20A%5ETB%20%3D%20a_%7B%5Cbullet%201%7D%20%5Ctimes%20b_%7B1%20%5Cbullet%7D%20&amp;plus;%20a_%7B%5Cbullet%202%7D%20%5Ctimes%20b_%7B2%20%5Cbullet%7D%20%3D%20%5Cbegin%7Bbmatrix%7D%201%5C%5C%202%20%5Cend%7Bbmatrix%7D%20%5Ctimes%20%5Cbegin%7Bbmatrix%7D%205%20%26%206%20%5Cend%7Bbmatrix%7D%20&amp;plus;%20%5Cbegin%7Bbmatrix%7D%203%5C%5C%204%20%5Cend%7Bbmatrix%7D%20%5Ctimes%20%5Cbegin%7Bbmatrix%7D%207%20%26%208%20%5Cend%7Bbmatrix%7D%20%3D%20%5Cbegin%7Bbmatrix%7D%205%20%26%206%5C%5C%2010%20%26%2012%20%5Cend%7Bbmatrix%7D%20&amp;plus;%20%5Cbegin%7Bbmatrix%7D%2021%20%26%2024%5C%5C%2038%20%26%2044%20%5Cend%7Bbmatrix%7D%20%3D%20%5Cbegin%7Bbmatrix%7D%2026%20%26%2030%5C%5C%2038%20%26%2044%20%5Cend%7Bbmatrix%7D -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira </pre> Sebastian Schelter (JIRA 2013-06-19T10:21:22 http://permalink.gmane.org/gmane.comp.apache.mahout.devel/26193 <pre>Also, this is particularly problematic because indices can't be negative so only 2^31 elements are actually possible. On Wed, Jun 19, 2013 at 1:15 PM, Dan Filimon &lt;dangeorge.filimon&lt; at &gt;gmail.com&gt;wrote: </pre> Dan Filimon 2013-06-19T10:16:15 http://permalink.gmane.org/gmane.comp.apache.mahout.devel/26192 <pre>Hi everyone! The current Vector API only supports 32bit maximum indices for Vectors. I feel that 64bits would be more appropriate especially because the indices are likely to be hash values of other data and 32bit will result in quite a few collisions. Also, for some jobs, notably ItemSimilarityJob, this restriction means that we need a special id to index map where we'll collide anyway. What do you think about adding support for 64bit indices? Is anyone at all interested? </pre> Dan Filimon 2013-06-19T10:15:14 http://permalink.gmane.org/gmane.comp.apache.mahout.devel/26191 <pre> [ https://issues.apache.org/jira/browse/MAHOUT-1266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&amp;focusedCommentId=13687820#comment-13687820 ] Sebastian Schelter commented on MAHOUT-1266: -------------------------------------------- Matlab says, Mahout's result is correct: {noformat} 1 8 6 3; 8 3 3 9; 7 1 9 6] A = 9 4 1 9 1 8 6 3 8 3 3 9 7 1 9 6 7 8 9 5; 2 1 5 8; 7 4 4 9] B = 2 1 6 5 7 8 9 5 2 1 5 8 7 4 4 9 ans = 90 53 131 177 77 75 115 93 113 88 111 140 99 66 150 186 {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira </pre> Sebastian Schelter (JIRA 2013-06-19T10:02:20 http://permalink.gmane.org/gmane.comp.apache.mahout.devel/26190 <pre> [ https://issues.apache.org/jira/browse/MAHOUT-1266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&amp;focusedCommentId=13687804#comment-13687804 ] Martin Illecker edited comment on MAHOUT-1266 at 6/19/13 9:49 AM: ------------------------------------------------------------------ I have executed the following commands, based on the example before: {{mahout matrixmult --numRowsA 4 --numColsA 4 --numRowsB 4 --numColsB 4 --inputPathA /pathTo/MatrixA.seq --inputPathB /pathTo/MatrixB.seq}} {{mahout vectordump -i /pathTo/MatrixA.seq}} [1] {0:9.0,1:4.0,2:1.0,3:9.0} {0:1.0,1:8.0,2:6.0,3:3.0} {0:8.0,1:3.0,2:3.0,3:9.0} {0:7.0,1:1.0,2:9.0,3:6.0} {{mahout vectordump -i /pathTo/MatrixB.seq}} [2] {0:2.0,1:1.0,2:6.0,3:5.0} {0:7.0,1:8.0,2:9.0,3:5.0} {0:2.0,1:1.0,2:5.0,3:8.0} {0:7.0,1:4.0,2:4.0,3:9.0} {{mahout vectordump -i /pathTo/productWith-80}} {0:90.0,1:77.0,2:113.0,3:99.0} {0:53.0,1:75.0,2:88.0,3:66.0} {0:131.0,1:115.0,2:111.0,3:150.0} {0:177.0,1:93.0,2:140.0,3:186.0} Shouldn't C be something like this[3]: |111|78|131|154| |91|83|120|120| |106|71|126|160| |81|48|120|166| [1] [https://github.com/millecker/applications/blob/master/hadoop/rootbeer/matrixmultiplication/MatrixA.seq] [2] [https://github.com/millecker/applications/blob/master/hadoop/rootbeer/matrixmultiplication/MatrixB.seq] [3] [http://www.wolframalpha.com/input/?i=%7B%7B9%2C4%2C1%2C9%7D%2C%7B1%2C8%2C6%2C3%7D%2C%7B8%2C3%2C3%2C9%7D%2C%7B7%2C1%2C9%2C6%7D%7D*%7B%7B2%2C1%2C6%2C5%7D%2C%7B7%2C8%2C9%2C5%7D%2C%7B2%2C1%2C5%2C8%7D%2C%7B7%2C4%2C4%2C9%7D%7D] was (Author: bafu): I have executed the following commands, based on the example before: {{mahout matrixmult --numRowsA 4 --numColsA 4 --numRowsB 4 --numColsB 4 --inputPathA /pathTo/MatrixA.seq --inputPathB /pathTo/MatrixB.seq}} {{mahout vectordump -i /pathTo/MatrixA.seq}} {0:9.0,1:4.0,2:1.0,3:9.0} {0:1.0,1:8.0,2:6.0,3:3.0} {0:8.0,1:3.0,2:3.0,3:9.0} {0:7.0,1:1.0,2:9.0,3:6.0} {{mahout vectordump -i /pathTo/MatrixB.seq}} {0:2.0,1:1.0,2:6.0,3:5.0} {0:7.0,1:8.0,2:9.0,3:5.0} {0:2.0,1:1.0,2:5.0,3:8.0} {0:7.0,1:4.0,2:4.0,3:9.0} {{mahout vectordump -i /pathTo/productWith-80}} {0:90.0,1:77.0,2:113.0,3:99.0} {0:53.0,1:75.0,2:88.0,3:66.0} {0:131.0,1:115.0,2:111.0,3:150.0} {0:177.0,1:93.0,2:140.0,3:186.0} Shouldn't C be something like this: |111|78|131|154| |91|83|120|120| |106|71|126|160| |81|48|120|166| [1] [http://www.wolframalpha.com/input/?i=%7B%7B9%2C4%2C1%2C9%7D%2C%7B1%2C8%2C6%2C3%7D%2C%7B8%2C3%2C3%2C9%7D%2C%7B7%2C1%2C9%2C6%7D%7D*%7B%7B2%2C1%2C6%2C5%7D%2C%7B7%2C8%2C9%2C5%7D%2C%7B2%2C1%2C5%2C8%7D%2C%7B7%2C4%2C4%2C9%7D%7D] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira </pre> Martin Illecker (JIRA 2013-06-19T09:50:20 Search the mailing list at Gmane query http://search.gmane.org/?group=$group=gmane.comp.apache.mahout.devel