gmane.comp.web.wget.general

Re: alpha release (1.13.4.56-620c) (was: [PATCH]gnutls.c: fix infinite read timeout)

Dagobert Michelsen — 2012-05-21T21:44:10

Hi Guiseppe,

Am 21.05.2012 um 23:23 schrieb Giuseppe Scrivano:

Testsuite passes fully on Solaris 9 Sparc with Sun Studio 12.

BTW, it would be nice if the CA path in src/gnutls.c would be configurable,
but this just as a side note :-)


Best regards

  -- Dago

[PATCH] fix build warnings with newer zlib

Mike Frysinger — 2012-05-21T23:06:01

The gz* API from zlib takes a "gzFile", not a "gzFile*".  Older
versions of zlib didn't trigger a warning due to how the funcs
were written, but newer ones do.  So tweak the type (which in
reality shouldn't change anything since it's a typedefed pointer)
to work with old/new versions.

Signed-off-by: Mike Frysinger <vapier< at >gentoo.org>
---
 src/warc.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/src/warc.c b/src/warc.c
index 57fdcad..9b4329d 100644
--- a/src/warc.c
+++ b/src/warc.c
< at >< at > -75,7 +75,7 < at >< at > static FILE *warc_current_file;
 #ifdef HAVE_LIBZ
 /* The gzip stream for the current WARC file
    (or NULL, if WARC or gzip is disabled). */
-static gzFile *warc_current_gzfile;
+static gzFile warc_current_gzfile;
 
 /* The offset of the current gzip record in the WARC file. */
 static off_t warc_current_gzfile_offset;

[PATCH] fix build warnings due to DEBUG_MALLOC

Mike Frysinger — 2012-05-21T23:05:59

The current cleanup_html_url code warns because it's marked static
but no one calls it.  Reading the code base, we see it only gets
called when DEBUG_MALLOC is defined, so change the static markings
to check that ifdef.

The spider_cleanup has a similar problem.

Signed-off-by: Mike Frysinger <vapier< at >gentoo.org>
---
 src/html-url.c |    4 +++-
 src/init.c     |    2 ++
 src/spider.c   |    4 +++-
 3 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/src/html-url.c b/src/html-url.c
index 55563e2..8a3dc6c 100644
--- a/src/html-url.c
+++ b/src/html-url.c
< at >< at > -830,7 +830,8 < at >< at > get_urls_file (const char *file)
   return head;
 }
 
-static void
+#ifdef DEBUG_MALLOC
+void
 cleanup_html_url (void)
 {
   /* Destroy the hash tables.  The hash table keys and values are not
< at >< at > -840,3 +841,4 < at >< at > cleanup_html_url (void)
   if (interesting_attributes)
     hash_table_destroy (interesting_attributes);
 }
+#endif
diff --git a/src/init.c b/src/init.c
index 57a4f00..0a3c25b 100644
--- a/src/init.c
+++ b/src/init.c
< at >< at > -1

[PATCH] detect openssl via pkg-config if it's available

Mike Frysinger — 2012-05-21T23:05:56

Newer versions of openssl ship with pkg-config files, so if we can detect
it via those, do so.  If that fails, fall back to the classic method.

Signed-off-by: Mike Frysinger <vapier< at >gentoo.org>
---
 configure.ac |   55 ++++++++++++++++++++++++++++++++-----------------------
 1 files changed, 32 insertions(+), 23 deletions(-)

diff --git a/configure.ac b/configure.ac
index 873c3c9..d527587 100644
--- a/configure.ac
+++ b/configure.ac
< at >< at > -241,7 +241,16 < at >< at > AS_IF([test x"$with_zlib" != xno], [
   AC_CHECK_LIB(z, compress)
 ])
 
+PKG_PROG_PKG_CONFIG
+
 AS_IF([test x"$with_ssl" = xopenssl], [
+  PKG_CHECK_MODULES([OPENSSL], [openssl], [
+    AC_MSG_NOTICE([compiling in support for SSL via OpenSSL])
+    AC_LIBOBJ([openssl])
+    LIBS="$OPENSSL_LIBS $LIBS"
+    CFLAGS="$OPENSSL_CFLAGS -DHAVE_LIBSSL $CFLAGS"
+    LIBSSL=" " # ntlm check below wants this
+  ], [
     dnl some versions of openssl use zlib compression
     AC_CHECK_LIB(z, compress)
 
< at >< at > -278,29 +287,29 < at >< at > AS_IF([test x"$with_ssl" = xopenssl], [

Re: [PATCH] Added .gitignore and updated it with filelist to ignore for GIT repo

Mike Frysinger — 2012-05-21T22:33:05

you shouldn't list every .o file.  simply use *.o.

same goes for .deps, and probably Makefile and Makefile.in
-mike

Re: [PATCH] Added .gitignore and updated it with file list to ignore for GIT repo

Sasikanth babu — 2012-05-21T22:03:19

 Here we are ignoring the .o files and auto-generated files alone, not lib
(git submodule) .c and .h files.

 Thanks
 Sasi

alpha release (1.13.4.56-620c) (was: [PATCH] gnutls.c:fix infinite read timeout)

Giuseppe Scrivano — 2012-05-21T21:23:50


I have amended the ChangeLog entry in your patch and pushed it.  I think
this is the last big change before the new release.

I have prepared an alpha release here:

ftp://alpha.gnu.org/gnu/wget/wget-1.13.4.56-620c.tar.bz2
ftp://alpha.gnu.org/gnu/wget/wget-1.13.4.56-620c.tar.bz2.sig

Please test it and let me know of any problem.

If there are not blocking issues, the final release will be rolled out
in 7-10 days.

Cheers,
Giuseppe

Re: Trying to wget files from IEEE802 document server

Robert Moskowitz — 2012-05-21T11:58:47

There are reasons I do not want to use this high of a level of wget, 
principally there are old years that don't have any updates and wget is 
rather inefficient (compared to rsync, but that is not an option here) 
in checking directories for changed files.

But I figured it was a good place to start.  So I ran it and it was 
working ok.

The directory structure this creates is ../mentor.ieee.org/802.18/

There are 3 directories that got created: bp, dcn, and file.  bp has a 
startpage html and file has year 09 (strange). dcn had year 12 that came 
down ok.  But when it got to 11 under dcn and the problems started.  I 
see the following:

--2012-05-20 17:18:36--  
https://mentor.ieee.org/802.18/dcn/11/18-11-0085-00-0000-rr-tag-agenda-november-2011.ppt
Reusing existing connection to mentor.ieee.org:443.
HTTP request sent, awaiting response... No data received.
Retrying.

--2012-05-20 17:18:37--  (try: 2)  
https://mentor.ieee.org/802.18/dcn/11/18-11-0085-00-0000-rr-tag-agenda-november-2011.ppt
Connecting to

Re: Trying to wget files from IEEE802 document server

Ángel González — 2012-05-20T20:34:39

You can use something like
 wget --no-check-certificate -m -np https://mentor.ieee.org/802.18/documents

Re: Fix: Unused parameter

Giuseppe Scrivano — 2012-05-20T19:02:47


Thanks, applied.

Giuseppe

Re: Trying to wget files from IEEE802 document server

Robert Moskowitz — 2012-05-20T17:45:41

So I see.  Problem with assuming that the URL seen here:

https://mentor.ieee.org/802.18/documents

Would work off the higher level.

The nice page listed above gives a nice user interface into a group's 
documents, hiding which year the rev0 doc was posted.  I like to 
maintain the by year orientation of wg/dcn/yr/.


So what are they doing there?  Well I am going to ask the group that 
maintains the local replication.  Though they use WebDev, but they may 
have some insight for me.  But if you see anything from the user page, 
please tell me....

Re: Trying to wget files from IEEE802 document server

Ángel González — 2012-05-20T16:34:40

This is odd. Apparently, there's a trust difference between the two UA.
I first thought that Verisign root CA wasn't available in the store used
by wget, but wget downloads https://www.verisign.com/ fine, and both
sites have the same certificate root "VeriSign Class 3 Public Primary
Certification Authority - G5" (serial
18:DA:D1:9E:26:7D:E8:BB:4A:21:58:CD:CC:6B:3B:4A).
The difference may lie in that www.verisign.com is providing the three
certificates, but mentor.ieee.org only the last two.

That for the certificate failure.

The problem you get when using --no-check-certificate is due to
https://mentor.ieee.org/802.18/dcn/12/ not containing a file list. TRy
going to https://mentor.ieee.org/802.18/dcn/12/ in your browser: "Not
Found The page requested was not found or is not a functioning properly"

Obviously,
wget --no-check-certificate
https://mentor.ieee.org/802.15/dcn/12/15-12-0290-01-0009-tg9-closing-report-may-2012.ppt

works properly.

Performance statistics

Asen Bozhilov — 2012-05-19T12:28:22

I would like to have a dump of performance statistic, e.g. dns resolving
time, connection time, first byte, last byte etc. For example curl supports
it trough the -w option. Would be pretty nice especially to test the full
page performance.

Re: Fix: Unused parameter

illusionoflife — 2012-05-19T14:49:16

Thanks for clarification.
Here is the patch.

Re: [PATCH] gnutls.c: fix infinite read timeout

Giuseppe Scrivano — 2012-05-19T11:08:44

thanks again for your contribution!  The patch seems ok to me.  Could
you please also provide a complete entry for the ChangeLog file?


As a side note, if you would like to include the ChangeLog entry in the
commit message, please keep a simple description of the change as the
first line.

Giuseppe



Tim Ruehsen <tim.ruehsen< at >gmx.de> writes:

Re: Fix: Unused parameter

Giuseppe Scrivano — 2012-05-19T10:52:18

Hi!

illusionoflife <illusion.of.life92< at >gmail.com> writes:


thanks for your patch.  The preferred way is to generate a patch with
"git format-patch", then send it (either as attachment or inline).
Please also add the entries for the ChangeLog file.

Giuseppe

Re: [PATCH] Added .gitignore and updated it with filelist to ignore for GIT repo

Giuseppe Scrivano — 2012-05-19T10:35:34


Do we really need to track these files at all?  lib is a git submodule,
so it is a repository by itself.

Cheers,
Giuseppe

Trying to wget files from IEEE802 document server

Robert Moskowitz — 2012-05-18T20:57:44

This server is open to the public. For example for 802.11 (WiFi), the 
url is:

https://mentor.ieee.org/802.11/documents

Now when I am at the 802 meetings (as I was earlier this week) we have a 
local replicate that I wget the docs from with:

wget -m -nH --cut-dirs=3 --no-parent -R.html 
http://griffin.events.ieee.org/docs/802.15/12/

I have to do this for each of the directories for each of the groups 
that have had updates. A bit more pain that rsync would be, but they 
don't support rsync.

So....

Now that I am home, I want to go to the main server and I tried:

wget -m -nH --cut-dirs=3 --no-parent -R.html 
http://mentor.ieee.org/802.18/dcn/12/

And got a warning telling me I needed to use '--no-check-certificate'

So both wget -m -nH --cut-dirs=3 --no-parent --no-check-certificate 
-R.html http://mentor.ieee.org/802.18/dcn/12/

and wget -m -nH --cut-dirs=3 --no-parent --no-check-certificate -R.html 
https://mentor.ieee.org/802.18/dcn/12/

gets me:

--2012-05-18 16:44:52-- https://mentor.ieee.org/802.1

Fix: Unused parameter

illusionoflife — 2012-05-18T18:23:09

Hello! Here is very little patch, that fix unused parameter warning and
make 2 functions to not take not-used parameter. I am very new to big 
projects, so any feedback is very welcome.
I do not know, whether inline diff or attach prefered, 
so I used linux style - inline.

diff --git a/src/convert.c b/src/convert.c
index 6cf6f27..e1c58e9 100644
--- a/src/convert.c
+++ b/src/convert.c
< at >< at > -869,9 +869,9 < at >< at > register_delete_file (const char *file)
 
 /* Register that FILE is an HTML file that has been downloaded. */
 
 void
-register_html (const char *url, const char *file)
+register_html (const char *file)
 {
   if (!downloaded_html_set)
     downloaded_html_set = make_string_hash_table (0);
   string_set_add (downloaded_html_set, file);
< at >< at > -879,9 +879,9 < at >< at > register_html (const char *url, const char *file)
 
 /* Register that FILE is a CSS file that has been downloaded. */
 
 void
-register_css (const char *url, const char *file)
+register_css (const char *file)
 {
   if (!downloaded_css_set)
     downloaded_cs

[PATCH] gnutls.c: fix infinite read timeout

Tim Ruehsen — 2012-05-18T11:37:28

Hi,

I finally got --timeout=0 working together with gnutls.

After all it is about two lines in wgnutls_peek().

I lost a few hours in debugging the polling/select/peek/blocking mechanism of 
wget which came together with reading into gnutls. But who knows what it is 
good for ;-)

Tim

Re: help requested in downloading my website

Henrik Holst — 2012-05-18T08:51:15

fre 2012-05-18 klockan 10:47 +0200 skrev Tim Ruehsen:

Good thinking, if he's lucky he has been using relative links though.

/HH