gmane.org.netlabs.libc.user

Re: Built grep 2.5.3 successfully, or not?

John Small — 2008-11-17T12:58:34

** Reply to message from "Dave Yeo" telus.net> on Sun, 16 Nov 2008 14:55:10 -0700 No matter what? For example, the grep uses fopen (without "b") for the "-f" file and then uses fread's to read the file into a buffer. What I find is: #ifdef HAVE_DOS_FILE_CONTENTS # include # ifdef HAVE_SETMODE # define SET_BINARY(fd) setmode (fd, O_BINARY) # else # define SET_BINARY(fd) _setmode (fd, O_BINARY) # endif #endif in a header file and #if defined(SET_BINARY) /* Set input to binary mode. Pipes are simulated with files on DOS, so this includes the case of "foo | grep bar". */ if (!isatty (desc)) SET_BINARY (desc); #endif in the source. Since this code handles the case where files are opened to be searched and since the code for opening a file to read patterns ("-f") works without "-Zbin-files" and without any setmode's (it uses fopen and fread), then I am led to think that I may be covered. What do you think? No you were correct. I changed back to the non-debug options which includes "-O3" and then deleted "-Zbin-files" from the non-debug options The "-f" tests pass when the ONLY difference in the build options is that there is no "-Zbin-files".

Re: Built grep 2.5.3 successfully, or not?

Dave Yeo — 2008-11-16T21:55:10

Yes that is correct and is usually a good default. Lots of porting problems can be caused by the line endings issue. See http://svn.netlabs.org/libc/wiki/kOptions for OS/2 specific flags Yes, it is a libc problem, not GCC problem. Note that it works fine for most files. See http://svn.netlabs.org/libc/ticket/18 for Knuts take on it. It still works fine for normal files, just questionable for stdin, stdout and stderr. Test with and without the flag. I think it is needed for the files.sh test to pass but I might be wrong. I have it here in my config.site and I also pass the files.sh test. Yes that is the best solution. Often programs already have the setmode call in a #ifdef for windows so can just add __EMX__ to the #ifdef. Other programs have a configure check, perhaps using check_headers to check for io.h and wrap the function in a #ifdef HAVE_IO_H wrapper. I like the second solution for portability but the first solution is simpler and quicker to implement. (Note I might be misremembering some of the above syntax) I also just noticed I misread the flags from Paul's config.site, getting the debug flags mixed up with the non-debug flags so perhaps I'm totally of base with -Zbin-files in this case. Anyways doesn't hurt to experiment, maybe you were right about the optimization. Dave

Re: Built grep 2.5.3 successfully, or not?

John Small — 2008-11-16T22:27:11

** Reply to message from "John Small" os2world.net> on Sun, 16 Nov 2008 07:52:46 +0500 Update: I have looked at the CVS repository for grep 2.5.4 at gnu.org and they have commented out these three tests because they "are known to fail". So I don't think it is worth looking into them any further. I have commented them out in my 2.5.3 script. So the only remaining issue to my knowledge is the "-Zbin-files" issue (discussed in a separate thread) which breaks the "-f" option for grep.

Re: Built grep 2.5.3 successfully, or not?

John Small — 2008-11-16T21:05:37

** Reply to message from "Dave Yeo" telus.net> on Sat, 15 Nov 2008 19:19:52 -0700 As I understand it "-Zbin-files", when it works, causes files to be opened, by default, in binary mode. Is this correct? If it worked, is it a good default? Other questions: - Is "-Zbin-files" is broken in all 3.3.5 (and up) versions of GCC for OS/2? - If yes, then why is it included LDFLAGS in u:\moztools\config.site-gcc335b4 in Paul Smedley's build environment? - How does one know when "-Zbin-files" is needed? More specifically to the case at hand, how do I know if "-Zbin-files" is needed for grep to function properly? It seems to pass the "make check" tests without it but how can I tell if something untested would be broken? Should I add setmode calls to the source to compensate for the missing "-Zbin-files"? Thanks for your help!

Re: Built grep 2.5.3 successfully, or not?

Dave Yeo — 2008-11-16T02:19:52

It is the -Zbin-files that is allowing you to pass file.sh. Using -Zbin-files is equal to setmode (file, O_BINARY). note also that there is a ticket filed about -Zbin-files not (always?) working with stdin, stdout etc. Also look at the -u and -U grep options .... Dave

Re: Built grep 2.5.3 successfully, or not?

John Small — 2008-11-16T02:52:46

** Reply to message from "Steven Levine" earthlink.net> on Sat, 15 Nov 2008 07:47:33 -0800 Now I have replaced the above with just two very simple changes to the script. And the output and the byte-offset numbers match (for all but 3 of the 32 tests)! I went by the example in the README file: ./configure CC=c99 CFLAGS=-g LIBS=-lposix In any case, I think I have found the problem with building a debug version. I am using Paul Smedley's build environment. And when the configure script runs it reads one of his files: u:\moztools\config.site-gcc335b4. In this file are settings for CFLAGS and LDFLAGS. My earlier efforts simply added -g to the other options which are set in this file. I also noticed that there were some commented out debug settings for CFLAGS and LDFLAGS. The two sets of settings are: debug: CFLAGS='-D__EMX__ -DOS2 -D__ST_MT_ERRNO__ -g' LDFLAGS='-Zexe -Zomf -Zmap -Zargs-wild -D__ST_MT_ERRNO__' non-debug: CFLAGS='-D__EMX__ -DOS2 -D__ST_MT_ERRNO__ -O3 -mcpu=pentium3' LDFLAGS='-Zexe -Zomf -Zmap -Zargs-wild -Zbin-files -D__ST_MT_ERRNO__ -s' So I commented out the non-debug settings and un-commented the debug settings. After rerunning the configure script and make, I had 1) a grep program with debug info; and 2) a grep program which passes some of the tests which it previously failed!? The tests the debug version passes are the ones testing the "-f" option. Given the differences in the options, I suspect that the optimization level may be breaking grep's "-f" code. If this is the case, then should I just compile the code with a lower level of optimizations (and document this in the future readme.os2)? At this point I have a grep which 1) Now passes one of the three scripts it previously failed 2) Fails tests in the second script which are reported on gnu.org's list of bugs. 3) Passes all but three tests in the third script when the script is changed to work with line endings of \r\n. And I do not yet know if the Now my greatest need is help in determining if the grep output is wrong or if the grep output is correct and the tests are expecting the wrong output (or both). The log of three failing is in the file I uploaded to Netlabs:

Re: Built grep 2.5.3 successfully, or not?

Steven Levine — 2008-11-15T15:47:33

In <200811150706.1l1jUN6mG3Nl36J0< at >mx-various.atl.sa.earthlink.net>, on 11/15/08 at 05:06 PM, "John Small" os2world.net> said: OK. For our purposes, memchr is logically equivalent to strchr. Not everyone has infinite time. That's why I originally recommended that you upload the test results. > I'll look at this later today. FWIW, I recommend that all uploads of this kind contain a datestamp in the name. This is OK for now, but it's not a good long term solution IMO. I say this because it makes you mods harder to use on the next grep release. The way most of us do this is once the port is done, we do something like diff -u -r grep.2.5.3.org grep2.5.3.ported > grep.2.5.3.patches.diff When the next verison of grep arrives, we apply the patches to the new sources with patch -p0

Re: Built grep 2.5.3 successfully, or not?

John Small — 2008-11-15T12:06:06

** Reply to message from "Steven Levine" earthlink.net> on Fri, 14 Nov 2008 20:01:15 -0800 There are no \0 terminators in the data. They end up using: memchr (motif, '\n', total); I was hoping you would copy and paste it into an editor and then delete the line endings to get the 4 lines per test I described. Since the tested command is also very long and, I think, important in figuring out what is wrong (the actual or the expected output), I have uploaded the text in a zip file: There are 32 tests with multiple offsets for each test. Instead of changing all those offsets I 1) Edited the searched file and removed the \r's 2) Made the searched file read-only so the test could not rewrite it 3) Edited the script to remove (tr -d "\'r") the \r's from grep's output This is how I got the number of failures down to just 4. I have tried both sh ./configure CFLAGS=-g LDFLAGS=-g sh ./configure CFLAGS=-g and make CFLAGS=-g (an error occurs if I use "make CFLAGS=-g LDFLAGS=-g" because -g is passed twice on the link step) I have even tried adding "-g" to CFLAGS and LDFLAGS in the makefiles. Yet while I can see the "-g" and the "-Zomf" during the make, when I run grep under the VACPP debugger I am still unable to see or load any source code!? Thanks for your continuing help!!

Re: Built grep 2.5.3 successfully, or not?

Steven Levine — 2008-11-15T04:01:15

In <200811142209.1l1bxw5Y13Nl36v2< at >mx-harris.atl.sa.earthlink.net>, on 11/15/08 at 08:09 AM, "John Small" os2world.net> said: Hi, The --help option is supposed to show all of the available options. That's definitely not going to handle line endings in a platform specific manner. If suspect they are using strchr or strtok to find the line terminators. Interesting. It's still a garbled mess because your mail clint is wrapping the output. What I would do is line up the columns and figure out which one is wrong. That says you could just adjust the offsets, assuming the output is really correct. Add -g to CFLAGS and LDFLAGS. Yes. Link with -Zomf. Have fun, Steven

Re: Built grep 2.5.3 successfully, or not?

John Small — 2008-11-15T03:09:30

** Reply to message from "Steven Levine" earthlink.net> on Fri, 14 Nov 2008 17:17:23 -0800 How does one learn of the available configuration variables? Does ./configure --help just provide a general list of options or would it also include options specific to the software? Will do. The file is read with the fread function into a single block of memory. It will take me a while to find where and how it is parsed. I should have been more specific. The problem is when -i is used in conjunction with certain other options (e.g. -o and --color). Since my earlier message I have checked out gnu.org and found that this is a problem for many platforms, not just OS/2. [snip] Sorry about that. I meant to explain that the text for each test is exactly four lines each: Test # ... output: ... expect ... FAIL So I hope you won;t mind if I try it again: Test #11: { ../src/grep -F -n -b -m 5 -C 1 yes 2>/dev/null; echo ?0; sed 's!^!X!'; } < yesno.txt | tr '\n' ' '/' | tr -d '\r' output: "2-10-[B02 no ]/3:20:[C03 yes]/4:30:[D04 yes]/5:40:[E05 yes]/6-50-[F06 no ]/7-60-[G07 no ]/8:70:[H08 yes]/9:80:[I09 yes]/10-90-[J10 no ]/?0/X[J10 no ]/X[K11 no ]/X[L12 no ]/X[M13 yes]/X[N14 yes]/" expect: "2-10-[B02 no ]/3:20:[C03 yes]/4:30:[D04 yes]/5:40:[E05 yes]/6-50-[F06 no ]/7-60-[G07 no ]/8:70:[H08 yes]/9:80:[I09 yes]/?0/X[J10 no ]/X[K11 no ]/X[L12 no ]/X[M13 yes]/X[N14 yes]/" FAIL Test #27: { ../src/grep -F -n -b -m 2 -v -C 1 yes 2>/dev/null; echo ?0; sed 's!^!X!'; } < yesno.txt | tr '\n' '/' | tr -d '\r' output: "1:0:[A01 no ]/2:10:[B02 no ]/3-20-[C03 yes]/?0/X[C03 yes]/X[D04 yes]/X[E05 yes]/X[F06 no ]/X[G07 no ]/X[H08 yes]/X[I09 yes]/X[J10 no ]/X[K11 no ]/X[L12 no ]/X[M13 yes]/X[N14 yes]/" expect: "1:0:[A01 no ]/2:10:[B02 no ]/?0/X[C03 yes]/X[D04 yes]/X[E05 yes]/X[F06 no ]/X[G07 no ]/X[H08 yes]/X[I09 yes]/X[J10 no ]/X[K11 no ]/X[L12 no ]/X[M13 yes]/X[N14 yes]/" FAIL Test #28: { ../src/grep -F -n -b -m 2 -v -C 1 -o yes 2>/dev/null; echo ?0; sed 's!^!X!'; } < yesno.txt | tr '\n' '/' | tr -d '\r' output: "3-25-yes/?0/X[C03 yes]/X[D04 yes]/X[E05 yes]/X[F06 no ]/X[G07 no ]/X[H08 yes]/X[I09 yes]/X[J10 no ]/X[K11 no ]/X[L12 no ]/X[M13 yes]/X[N14 yes]/" expect: "?0/X[C03 yes]/X[D04 yes]/X[E05 yes]/X[F06 no ]/X[G07 no ]/X[H08 yes]/X[I09 yes]/X[J10 no ]/X[K11 no ]/X[L12 no ]/X[M13 yes]/X[N14 yes]/" FAIL Of course this should be true for grep . It is the test in the script which depends on the line endings. The "expect"-ed output includes byte-offsets which are only correct when there is no \r. P.S. How does one do a debug build? Can one set the build to create files useable by the VACPP debugger?

Re: Built grep 2.5.3 successfully, or not?

Steven Levine — 2008-11-15T01:17:23

In <200811141100.1l115zrL3Nl36r0< at >mx-ailes.atl.sa.earthlink.net>, on 11/14/08 at 08:59 PM, "John Small" os2world.net> said: Hi, I would fix both. It's much easier to be comfortable with a port that passes all the test. Also, if the fixes are not too intrusive, the grep maintainers might accept them. In the case of line endings there may already be a configuration variable to handle this. IMO, yes. Check for how the pattern file is opened or how the code strips line endings. I have to suspect there are some spurious CRs to be accounted for. There are both checking for a match failure. It's likely. This is not necessarily a flaw on on unix box. That's odd. There must a problem with case mapping. This is hard to visualize with the line wrap. It might be better to log to results to a file and attach it. An good eCS/OS2 port should be insensitive to the line endings since both styles occur. According to the man page the format is prefix line(s) matched line suffix lines(s) -- IIRC, the count determines how many lines are printed before and after the matched line. Steven

Re: Built grep 2.5.3 successfully, or not?

John Small — 2008-11-14T15:59:54

** Reply to message from "Steven Levine" earthlink.net> on Tue, 11 Nov 2008 08:34:06 -0800 After looking at the scripts I believe that many of the failures are due to line endings. There are two places where the line endings may be mishandled. One is in the grep executable and the other is in the test scripts themselves. Do they both need to be fixed? IOW, if grep.exe works correctly is it important to fix test scripts which incorrectly indicate a failure? As I indicated earlier 3 scripts report failure. One script, file.sh, tests the "-f" option. Are the following equivalent? Command line (single line, watch wrapping): (echo "miles" | ..\src\grep -i -E -f patfile > /dev/null 2>&1) || echo "File_pattern: Wrong status code, test \#1 failed" Script (4 lines) echo "miles" | ../src/grep -i -E -f patfile > /dev/null 2>&1 if test $? -ne 0 ; then echo "File_pattern: Wrong status code, test \#1 failed" fi If these are equivalent then the grep I built fails this test (and other similar ones in file.sh). Grep 2.5.1 passes them (when I replace ..\src\grep with grep). Am I correct in thinking that the "-f" option broken in the grep I built? A second script, foad1.sh, tests various combinations of command-line options and six of its tests show failures. Two of the tests are flawed because they try to pass a pattern of "y*" without the double quotes. On the other 4 failed tests, problem(s) with the "-i" option is/are indicated. The failures indicated by the third script are mostly, I believe, line endings issues. When I "fix" the script to work around these line ending problems I end up with 3 failed tests. The log (edited to show the full command line (with the line endings fixes)): Test #11: { ../src/grep -F -n -b -m 5 -C 1 yes 2>/dev/null; echo ?0; sed 's!^!X!'; } < yesno.txt | tr '\n' ' '/' | tr -d '\r' output: "2-10-[B02 no ]/3:20:[C03 yes]/4:30:[D04 yes]/5:40:[E05 yes]/6-50-[F06 no ]/7-60-[G07 no ]/8:70:[H08 yes]/9:80:[I09 yes]/10-90-[J10 no ]/?0/X[J10 no ]/X[K11 no ]/X[L12 no ]/X[M13 yes]/X[N14 yes]/" expect: "2-10-[B02 no ]/3:20:[C03 yes]/4:30:[D04 yes]/5:40:[E05 yes]/6-50-[F06 no ]/7-60-[G07 no ]/8:70:[H08 yes]/9:80:[I09 yes]/?0/X[J10 no ]/X[K11 no ]/X[L12 no ]/X[M13 yes]/X[N14 yes]/" FAIL Test #27: { ../src/grep -F -n -b -m 2 -v -C 1 yes 2>/dev/null; echo ?0; sed 's!^!X!'; } < yesno.txt | tr '\n' '/' | tr -d '\r' output: "1:0:[A01 no ]/2:10:[B02 no ]/3-20-[C03 yes]/?0/X[C03 yes]/X[D04 yes]/X[E05 yes]/X[F06 no ]/X[G07 no ]/X[H08 yes]/X[I09 yes]/X[J10 no ]/X[K11 no ]/X[L12 no ]/X[M13 yes]/X[N14 yes]/" expect: "1:0:[A01 no ]/2:10:[B02 no ]/?0/X[C03 yes]/X[D04 yes]/X[E05 yes]/X[F06 no ]/X[G07 no ]/X[H08 yes]/X[I09 yes]/X[J10 no ]/X[K11 no ]/X[L12 no ]/X[M13 yes]/X[N14 yes]/" FAIL Test #28: { ../src/grep -F -n -b -m 2 -v -C 1 -o yes 2>/dev/null; echo ?0; sed 's!^!X!'; } < yesno.txt | tr '\n' '/' | tr -d '\r' output: "3-25-yes/?0/X[C03 yes]/X[D04 yes]/X[E05 yes]/X[F06 no ]/X[G07 no ]/X[H08 yes]/X[I09 yes]/X[J10 no ]/X[K11 no ]/X[L12 no ]/X[M13 yes]/X[N14 yes]/" expect: "?0/X[C03 yes]/X[D04 yes]/X[E05 yes]/X[F06 no ]/X[G07 no ]/X[H08 yes]/X[I09 yes]/X[J10 no ]/X[K11 no ]/X[L12 no ]/X[M13 yes]/X[N14 yes]/" FAIL The input file for both grep and sed, yesno.txt, contains the following lines (with UNIX line endings (0x0a)): [A01 no ] [B02 no ] [C03 yes] [D04 yes] [E05 yes] [F06 no ] [G07 no ] [H08 yes] [I09 yes] [J10 no ] [K11 no ] [L12 no ] [M13 yes] [N14 yes] I don't know the proper format of the -C output so I do not know if this "failed" output is incorrect or the test's expected out is incorrect (or both). Can anyone help me find the correct format so I can work on "fixing" what is not correct? Any help will be appreciated!

Re: Built grep 2.5.3 successfully, or not?

John Small — 2008-11-14T04:47:59

** Reply to message from "Robert Blair" listemail.net> on Tue, 11 Nov 2008 13:43:52 -0800 How do you know that it "works so far"? Have you built anything successfully? If yes, please tell me where I can get the source code so I can try that one too.

Re: Built grep 2.5.3 successfully, or not?

Robert Blair — 2008-11-14T04:09:36

** Reply to message from "John Small" os2world.net> on Fri, 14 Nov 2008 04:26:43 +0500 The only program I compiled runs. I have not tried any other program yet. Yes. What I wanted was a program that could write to a X10 "firecracker" I searched and found the source so I downloaded "flipit-0.3.5.tar.GZ". When you install Paul's build environment two files are put in the root directory. One is gcc335.cmd which you need to modify to match your system. The other file is readme.os2, which explains the very basics of compiling a unix program. From the file readme.os2: "Many things will compile with a simple ash ./configure, make" This did not work for me as the "ash" shell did like the configure file included with the flipit program. This is why I needed to install the "sh" shell. After installing "sh" I did "sh ./configure" followed by "make" and compiled the flipit program. Then I had to make a small change to the flipit source for OS/2 and recompiled (I also had to fix a problem because the original flipit program had some syntax that the gcc 3.3.5 compiler did not like).

Re: Built grep 2.5.3 successfully, or not?

John Small — 2008-11-13T23:26:43

Re: Built grep 2.5.3 successfully, or not?

John Small — 2008-11-13T23:13:47

Re: Built grep 2.5.3 successfully, or not?

Peter Weilbacher — 2008-11-12T08:53:44

GLIBIDL is a special library for Mozilla builds, and I doubt that it is used by any other package that you are trying to build. Indeed, it would confuse things in case you were trying to build GNOME. Peter.

Re: Built grep 2.5.3 successfully, or not?

Robert Blair — 2008-11-11T21:43:52

** Reply to message from "John Small" os2world.net> on Tue, 11 Nov 2008 18:45:01 +0500 I have also recently installed Smedley's build environment. I don't know. I installed both AUTOCONF and GLIBIDL335 from the sCS CD number 2. I appears that AUTOCONF is not needed because it is included with Smedley's build environment but GLIBIDL335 is not. GLIBIDL335 appeared to make a difference in what I was doing but it may have been something else I did. Smedley's build environment comes with "ash" but it did not always work for me. Dave Yeo recommended "sh" which works here. I used Smedley's gcc335.cmd and changed the disk assignments to match with my system and that works so far. I did not use "make check" so I do not know what my results would have been. I just started by diong the "make" and going from there. I know how you feel. Are my problems because I don't know what I am doing or are they real problems. I does appear that at some point the gcc compiler changed so I had one syntax error that I needed to correct. The problem was that the error message dd not indicate clearly that the problem was a syntax error. So if what you are trying to compile is a little old you may need to fix the code.

Re: Built grep 2.5.3 successfully, or not?

Steven Levine — 2008-11-11T16:34:06

In <200811110845.1kZTyl7093Nl34i0< at >mx-bracke.atl.sa.earthlink.net>, on 11/11/08 at 06:45 PM, "John Small" os2world.net> said: Hi, That's not atypical. You should be able to look at the makefiles and the scripts to verify this. They are just trying to make you feel good. You figure this out by looking at the failing test scripts. Some scripts are better than others at identifying the failing tests. I doubt it is anything you did. It's possible that the tests don't take line endings into account or there is something not quite right in the handling of path names. These are typical things we need to fix up for a production quality eCs/OS2 build. As always, general questions will get you general answers. You would have been well served to post the log for the first couple of tests that failed. Zip is your friend. How small is the zipped log file? If you feel this is still too large, you can always upload to Netlabs incoming and post a link. Recall that we often do this for fm/2 testing. If you do post something, don't forget to include the configure command line and config.log. They may or may not contain useful data. Steven

Built grep 2.5.3 successfully, or not?

John Small — 2008-11-11T13:45:01

In my continuing efforts to figure out how to port software I keep trying new source code packages to see if I can get anything to build. Most end in failure and I have not been able to figure out if the problem is - my build environment (Smedley's) - am I missing something? - do I have the wrong version of something? - which shell should I be using? - do I have all the necessary env. vars. set correctly? - my configuration - which options, if any, should I be passing to the configure scripts? - which "can't find" messages during configure are acceptable and which ones indicate an error which needs to be fixed. - which "checking for/whether xyz... no" messages during configure are acceptable and which ones indicate an error which needs to be fixed. - the source code itself - are there changes necessary to make it build under/for OS/2? I finally ran across some source which may have built successfully, grep-2.5.3 (): - running the configure script completes without any obvious-to-me, fatal errors - running "make" completes without any obvious-to-me, fatal errors But then when I run "make check" I get some failures. I see many "FAIL" messages but at the end the summary says: 3 of 13 tests failed (1 tests were not run) (I think the discrepancy (3 vs. many) comes from the fact that there were errors detected in 3 of the 13 shell scripts which were run during "make check". But several of these scripts attempt multiple tests, many of which are failing.) Early in the "make check" process it says: Please, do not be alarmed if some of the tests failed. Report them to gnu.org> How do I determine if my test failures are ones - which I should "not be alarmed" about; or - which need to be fixed before I consider the program "ready for prime time"? I am hesitant to report my failures to gnu.org because I am afraid my failures are due to self-inflicted, rookie mistakes. I am also hesitant to post the log of "make check" without being asked to do so because of its length. Any help will be greatly appreciated!

Re: buildenv problem

Robert Blair — 2008-11-01T07:57:14

** Reply to message from "Robert Blair" listemail.net> on Fri, 31 Oct 2008 22:34:13 -0700 I found what the problem is. It appears that something was changed in the newer versions of the GCC compiler. Searching google for "does not give a valid preprocessing token" got the answer. Removing the ## in conf.h line 33 #define SYSCONFFILE SYSCONFDIR ## "/flipit.conf" solved that problem.