http://permalink.gmane.org/gmane.comp.nlp.moses.user hourly 1 1901-01-01T00:00+00:00 http://gmane.org/img/gmane-25t.png http://gmane.org http://permalink.gmane.org/gmane.comp.nlp.moses.user/8938 <pre>Hieu, FYI, there have been a couple of edge cases that I've encountered (can't remember exactly what) where it wasn't practical to set LC_ALL=C. I have found that setting LC_COLLATE=C is sufficient instead of setting LC_ALL. LC_COLLATE is what sort and its friends use to determine lexicographic order. Lane On Fri, May 24, 2013 at 7:35 AM, Hieu Hoang &lt;Hieu.Hoang-5WhEfG1TI8k&lt; at &gt;public.gmane.org&gt; wrote: </pre> Lane Schwartz 2013-05-24T13:15:09 http://permalink.gmane.org/gmane.comp.nlp.moses.user/8937 <pre>odd. it's usually a sign that the data is dirty. Make sure it's been cleaned, UTF8. If you run anything outside of the train-model.perl (eg. extract, sort etc), make sure you set LC_ALL=C On 23 May 2013 14:37, Jacob Dlougach &lt;dlougach-Re5JQEeQqe8AvxtiuMwx3w&lt; at &gt;public.gmane.org&gt; wrote: </pre> Hieu Hoang 2013-05-24T11:35:04 http://permalink.gmane.org/gmane.comp.nlp.moses.user/8936 <pre>i only tested the binaries on the following platforms, described in the release notes Ubuntu 12.10, 32 and 64-bit Debian 6.0, 32 and 64-bit Fedora 17, 32 and 64-bit openSUSE 12.2, 32 and 64-bit Redhat releases has a longer lifecycle so it wouldn't surprise me that their kernel is older, hence the error message. In this case, you should compile the code yourself. On 23 May 2013 19:16, Jasper Phillips &lt;jphillips-PZpoF4qebg/zh0hh38GeGa6RkeBMCJyt&lt; at &gt;public.gmane.org&gt;wrote: </pre> Hieu Hoang 2013-05-24T09:07:14 http://permalink.gmane.org/gmane.comp.nlp.moses.user/8935 <pre>Thanks for the quick reply Hieu! I took the linux-64bit tarball directly from the binaries directory at statmt.org to this machine, so, it should be a clean version. If my kernel version is no "too old", are there other reasons for this error? Thanks very much! *M. Jasper Phillips * Operations Department Language Scientific, Inc.® 10 Cabot Road, Suite 209 Medford, MA 02155 T: 617.621.0940 x152 F: 617.621.2552 jphillips-PZpoF4qebg/zh0hh38GeGa6RkeBMCJyt&lt; at &gt;public.gmane.org www.languagescientific.com *FKA: RIC International, Inc.* On Thu, May 23, 2013 at 2:05 PM, Hieu Hoang &lt;Hieu.Hoang-5WhEfG1TI8k&lt; at &gt;public.gmane.org&gt; wrote: _______________________________________________ Moses-support mailing list Moses-support-3s7WtUTddSA&lt; at &gt;public.gmane.org http://mailman.mit.edu/mailman/listinfo/moses-support </pre> Jasper Phillips 2013-05-23T18:16:27 http://permalink.gmane.org/gmane.comp.nlp.moses.user/8934 <pre>I don't think that's an error from the Moses code. Its likely that you've compiled on 1 machine, and running on another machine with incompatible kernels/libraries. Try and compile and run on the same machine and see what happens. There's no specific minimum kernel version, if it's not too ancient we will support it. Redhat 5.9 is only Jan 2013 so if you're still having trouble, please let us know On 23 May 2013 18:56, Jasper Phillips &lt;jphillips-PZpoF4qebg/zh0hh38GeGa6RkeBMCJyt&lt; at &gt;public.gmane.org&gt;wrote: </pre> Hieu Hoang 2013-05-23T18:05:19 http://permalink.gmane.org/gmane.comp.nlp.moses.user/8933 <pre>Good Afternoon All, I am running the EMS included in the Moses 1.0 64-bit Linux Binary to train a baseline system. However, it fails at the [TRAINING: prepare data] step with the error "FATAL: kernel too old". I'm wondering what is the minimum kernel version required for the 1.0 binaries? I'm trying to run Moses on a Red Hat Enterprise Linux Client release 5.9 with kernel version 2.6.18-308.11.1.e15. The following is the step where it fails: number of steps doable or running: 1 at Thu May 23 12:36:49 EDT 2013 doable: TRAINING:prepare-data executing /home/g/grad/icos/Moses/moses_64/scripts/ems/test_model_basic/steps/1/TRAINING_prepare-data.1 via sh (1 active) FATAL: kernel too old step TRAINING:prepare-data crashed number of steps doable or running: 0 at Thu May 23 12:40:16 EDT 2013 If any other info would be helpful, please let me know. Thanks in advance! *M. Jasper Phillips * Operations Department Language Scientific, Inc.® 10 Cabot Road, Suite 209 Medford, MA 02155 T: 617.6</pre> Jasper Phillips 2013-05-23T17:56:56 http://permalink.gmane.org/gmane.comp.nlp.moses.user/8932 <pre> Apologies for cross posting. Please forward to interested parties. ********************************************************************************************** At the Centre for Next Generation Localisation (CNGL) in Dublin, Ireland, we have a number of internships available covering a wide range of topics in Natural Language Processing and Machine Translation based at our Dublin City University site. The internships are available for both basic research and more applied research projects (including development-focused work). Candidates are required to be registered as MSc or PhD students (by research) in their home universities while carrying out their internship in Dublin and need to provide written confirmation of this from their home institute. Please find the internship advertisement attached. Details of a number of specific internships can be found at: http://www.cngl.ie/outreach/graduate-programme/postgradinternships/ Closing date: 31st May 2013 For any informal enquiries please cont</pre> Cara Greene 2013-05-23T14:23:01 http://permalink.gmane.org/gmane.comp.nlp.moses.user/8931 <pre>I am constantly getting the following message while trying to build a baseline model for Turkish-English: (6.6) consolidating the two halves &lt; at &gt; Thu May 23 14:44:11 MSK 2013 Executing: /opt/mosesdecoder/scripts/../bin/consolidate /home/jacob/tr-en/phrasebased/training/model/phra se-table.half.f2e.gz /home/jacob/tr-en/phrasebased/training/model/phrase-table.half.e2f.gz /dev/stdout -- GoodTuring /home/jacob/tr-en/phrasebased/training/model/phrase-table.half.f2e.gz.coc | gzip -c &gt; /home/ja cob/tr-en/phrasebased/training/model/phrase-table.gz Consolidate v2.0 written by Philipp Koehn consolidating direct and indirect rule tables adjusting phrase translation probabilities with Good Turing discounting ERROR: source phrase does not match in line 1201: 'ruhe ! ! !' != 'take part ! !' Executing: rm -f /home/jacob/tr-en/phrasebased/training/model/phrase-table.half.* My training script looks like this: export MOSES_DIR=/opt/mosesdecoder nohup $MOSES_DIR/scripts/training/train-model.perl \ -corpus clean \ -root-d</pre> Jacob Dlougach 2013-05-23T13:37:11 http://permalink.gmane.org/gmane.comp.nlp.moses.user/8930 <pre>FYI everyone. You can find a list of all package dependencies on Debian Linux distros (including Ubuntu) here. http://www.precisiontranslationtools.com/products/software-requirements-domt-desktop/ If anyone has a similar list for other distros or Mac, I'm happy to add that to the page. Tom On 05/23/2013 09:21 AM, Kan Zhang wrote: _______________________________________________ Moses-support mailing list Moses-support-3s7WtUTddSA&lt; at &gt;public.gmane.org http://mailman.mit.edu/mailman/listinfo/moses-support </pre> Tom Hoar 2013-05-23T09:55:35 http://permalink.gmane.org/gmane.comp.nlp.moses.user/8929 <pre>Hi Hieu, Thank you very much for your timely reply, the suggestion is very helpful and the installation ERROR is SOLVED! thanks again and i gotta go ahead with the training step and see what would come. FYI, i just subscribed the support post. 2013/5/22 Hieu Hoang &lt;Hieu.Hoang-5WhEfG1TI8k&lt; at &gt;public.gmane.org&gt; </pre> Kan Zhang 2013-05-23T02:21:33 http://permalink.gmane.org/gmane.comp.nlp.moses.user/8928 <pre>actually, this is assuming the sentence-BLEU scores are already scaled in some way, so if you are not doing that the normalisation might have more of an effect Eva On Thu, May 23, 2013 at 1:17 AM, Eva Hasler &lt;evahasler-Re5JQEeQqe8AvxtiuMwx3w&lt; at &gt;public.gmane.org&gt; wrote: </pre> Eva Hasler 2013-05-23T08:44:09 http://permalink.gmane.org/gmane.comp.nlp.moses.user/8927 <pre>Hi Prashant, the parameters you mention are some of the things I tried out to normalize the losses or make the updates dependent on the size of the loss, but there is not comparative study about this (and at the time of the MIRA for Moses paper, they weren't in the code yet). In my experiments, normalization and margin scaling never made much of a difference, but the slack parameter does and setting it to a large value will make the training unstable. Eva On Thu, May 23, 2013 at 12:53 AM, Prashant Mathur &lt;prashant-0WaB7DoCAz4&lt; at &gt;public.gmane.org&gt; wrote: _______________________________________________ Moses-support mailing list Moses-support-3s7WtUTddSA&lt; at &gt;public.gmane.org http://mailman.mit.edu/mailman/listinfo/moses-support </pre> Eva Hasler 2013-05-23T00:17:11 http://permalink.gmane.org/gmane.comp.nlp.moses.user/8926 <pre>Hi All, I am trying to use MIRA for sparse features in moses. In MIRA there are many factors which regulate the final results a lot, such as 1. Normalisation &amp; sigmoid parameter for normalisation 2. Value of aggressive parameter (slack variable) 3. scaling of margin so on. I wanted to know if there is any comparative study done on how much a factor can affect the MT system. I say this because I don't want to just use the default values. 1. When I experiment with MIRA sometimes normalisation works much better than not doing it, 2. changing the slack variable slightly drives the BLEU crazy. So, I use Simplex to optimize the value of slack variable but that also fails to converge on most of the occasions (doesn't converge even after 50 iterations). I read the MIRA for moses paper but that one didn't had the comparative study that I am looking for. Any links? Thanks, Prashant </pre> Prashant Mathur 2013-05-22T23:53:16 http://permalink.gmane.org/gmane.comp.nlp.moses.user/8925 <pre>hi kan you must subscribe to the mailing list to post to it. You can subscribe here http://mailman.mit.edu/mailman/listinfo/moses-support you answer you question-i think you need to install the package sudo apt-get install libbz2-dev also, make sure you have these packages installed too zlib1g-dev libboost-all-dev ---------- Forwarded message ---------- From: &lt;moses-support-owner-3s7WtUTddSA&lt; at &gt;public.gmane.org&gt; Date: 22 May 2013 12:29 Subject: Moses-support post from kanzhang.jerry-Re5JQEeQqe8AvxtiuMwx3w&lt; at &gt;public.gmane.org requires approval To: moses-support-owner-3s7WtUTddSA&lt; at &gt;public.gmane.org As list administrator, your authorization is requested for the following mailing list posting: List: Moses-support-3s7WtUTddSA&lt; at &gt;public.gmane.org From: kanzhang.jerry-Re5JQEeQqe8AvxtiuMwx3w&lt; at &gt;public.gmane.org Subject: MOSES ON Ubuntu 12.04LTS ERROR HELP! Reason: Post by non-member to a members-only list At your convenience, visit: http://mailman.mit.edu/mailman/admindb/moses-support </pre> Hieu Hoang 2013-05-22T12:59:36 http://permalink.gmane.org/gmane.comp.nlp.moses.user/8924 <pre>Hi Johannes An alternative to adding the terminological information to the phrase table is to use Moses xml-input mechanism to override the model. http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc7 I think you will only find the best approach by experimenting, cheers - Barry On 21/05/13 10:44, Johannes Hellrich wrote: </pre> Barry Haddow 2013-05-22T09:13:22 http://permalink.gmane.org/gmane.comp.nlp.moses.user/8923 <pre>NLP People news: 1. We continue to publish the guidelines on career path selection. The article we just made available at NLPPeople.com is called “Why does it really take to make it in academia?”. We and the author (Paige Harris) will very much appreciate if you could share your opinion on the topics discussed in the article. The direct link to the publication: https://nlppeople.com/index.php/publications/articles/111-what-does-it-really-take-to-make-it-in-academia 2. Hot NLP job openings: - talented NLP researcher for a leading software company based in Cambridge (UK):https://nlppeople.com/index.php/job/2083/nlp-researcher-cambridge - lead researcher position within Samsung Advanced Research Institute in Bangalore (India) https://nlppeople.com/index.php/job/2059/researcher-bangalore - NLP software developer advertised by Venbest Recruiting in Kyiv (Ukraine): https://nlppeople.com/index.php/job/2049/software-developer-(nlp)-kyiv 3. Current machine learning and data mining openings: - entry level progr</pre> Alexander Raginsky 2013-05-22T09:05:08 http://permalink.gmane.org/gmane.comp.nlp.moses.user/8922 <pre>Hi Johannes There's 2 main models you should update 1. Language model 2. Translation model For the language model, you can concatenate the (in-domain) biomedical data to any other target-language corpus you might have. Or you can build language models for in-domain and out-of-doman data, the interpolate them to produce 1 overall LM. Or you can use both LMs separate during decoding. For translation models, there are a few choices. Again, you can concatenate the parallel data. Or you can subsample the parallel data from each corpora. Or you can create 2 separate phrase tables and use them both during decoding. If you do this, you can configure them to compete against each other. Or make 1 the primary phrase-table, and 1 a backoff table which is only used if the primary doesn't have translation. There's no explicit support for placeholders in Moses, but you can do it externally. I'm not sure what is current best practise, I suppose it just depends on what is easiest to implement and gives you the d</pre> Hieu Hoang 2013-05-22T08:57:31 http://permalink.gmane.org/gmane.comp.nlp.moses.user/8921 <pre>I used train-model.perl script to get all the steps(1 to 9) done. Now everything is fine. Thank you. On Thu, May 16, 2013 at 5:10 AM, Hieu Hoang &lt;Hieu.Hoang-5WhEfG1TI8k&lt; at &gt;public.gmane.org&gt; wrote: </pre> Nithya 2013-05-22T05:13:30 http://permalink.gmane.org/gmane.comp.nlp.moses.user/8920 <pre>Hi, This was a recursive issue ... libmoses needed libmira to compile and libmira needed libmoses to compile. However, I was able to solve this issue in rather naive way. Instead of using files from mira/ directory I copied all the files that I wanted to use in a separate directory and added the path to the moses/Jamfile. Now everything works perfectly fine. Thanks for the support though, -- Prashant On 05/16/2013 09:00 PM, Hieu Hoang wrote: _______________________________________________ Moses-support mailing list Moses-support-3s7WtUTddSA&lt; at &gt;public.gmane.org http://mailman.mit.edu/mailman/listinfo/moses-support </pre> Prashant Mathur 2013-05-21T11:35:12 http://permalink.gmane.org/gmane.comp.nlp.moses.user/8919 <pre>Hi all, I want to incorporate a (biomedical) multilingual-terminology into a SMT pipeline. I have no current plans to use disambiguation, so I could simply extract a dictionary from my terminology. But how do I incorporate this dictionary with my pipeline? The common solution seems to be adding its entries to the training data. Would on over-sample it or add the entries only once? I also wondered if one couldn't treat it like a transliteration and replace the known translations in the training data with placeholders (and add some kind of wrapping around Moses to insert the correct ones) - this could be more useful if I use disambiguation later on. Are there any best-practices if failed to find or would this be worth a thorough analysis? Thank you very much, Johannes Hellrich ---------------------------------------------------------------- This message was sent through https://webmail.uni-jena.de </pre> Johannes Hellrich 2013-05-21T09:44:42 http://permalink.gmane.org/gmane.comp.nlp.moses.user/8918 <pre> Thanks, Barry. It helps to know where to look. On 2013-05-21 14:50, Barry Haddow wrote: unusual characters in the corpus. If you vocab file (target side - es.vcb I think) then problem is. wrote: MGIZA++ failed below. I think this anyone confirm this or = -1 (fixed value for parameter p_0 in IBM-5 (if negative then it determined in training)) () fertility for fertility models) executed, use 0 if you just want all (fixed value for parameter p_0 in IBM-3/4 (if negative then determined in training)) pegging) from:/opt/domy/TRAININGS/alignments/align-dell2_full-en-es/giza.classes/en.vcb Reading vocabulary file from:/opt/domy/TRAININGS/alignments/align-dell2_full-en-es/giza.classes/es.vcb Source vocabulary list has 85970 unique tokens list has 84643 unique tokens corpus /opt/domy/TRAININGS/alignments/align-dell2_full-en-es/giza.classes/es-en-int-train.snt Reading more sentence pairs into memory ... 118049 is not in the vocabulary list _______________________________________________ list http://mailman.mit</pre> Tom Hoar 2013-05-21T08:03:10 Search the mailing list at Gmane query http://search.gmane.org/?group=$group=gmane.comp.nlp.moses.user