gmane.network.wwwoffle.user

Re: www.consortiuminfo.org/standardsblog/ blockswwwoffle?

Andrew M. Bishop — 2008-07-23T18:51:36

This is strange. Perhaps different versions of the program work differently. I made it the default because it is a new feature in HTTP/1.1 so I thought that it must be a good idea. I did e-mail what I thought was the right address (after removing the SPAM trap e-mail address). I didn't receive a reply.

Re: www.consortiuminfo.org/standardsblog/ blockswwwoffle?

Martin Bähr — 2008-07-23T15:12:08

and it is used by sites where wwwoffle is not blocked (blogs.gnome.org for example claims to use it) oh phantastic, if it's configurable, then there is no need making it the default (i could easely apply it to <*> if i wanted to) i'd be curious. i think i tried emailing them without any results, but then i didn't know what you know, so maybe since you can explain the problem they will actually listen. oh wow, this is really nice. i never even considered that wwwoffle might have such a feature. you just made it a lot harder to replace wwwoffle by any competition greetings, martin.

Re: www.consortiuminfo.org/standardsblog/blockswwwoffle?

Max Kirillov — 2008-07-11T01:36:22

Usually such blockers caches theirs descisions. Not for a long may be an hour, or even less.

Re: www.consortiuminfo.org/standardsblog/ blockswwwoffle?

Andrew M. Bishop — 2008-07-10T18:34:01

I can verify that WWWOFFLE is blocked by this blog and some others because of the set of headers that it sends. The script that is doing the blocking is called "bad behaviour" and it has been around for a while. When requesting a non-persistent connection and chunked encoding (which is a valid combination in HTTP/1.1) WWWOFFLE sends the headers: Connection: close TE: chunked In discussion with the author of the bad behaviour script he said this (it is a couple of years old now): : Yes, the correct Connection: header that wwwoffle should be sending : is "Connection: TE, close" (or "Connection: close, TE") when : requesting chunked data. This would be a bug in wwwoffle. Reference: : RFC 2616, section 14.39: : : The TE header field only applies to the immediate connection. : Therefore, the keyword MUST be supplied within a Connection header : field (section 14.10) whenever TE is present in an HTTP/1.1 message. : : The test is in Bad Behaviour because several malicious proxy servers :

www.consortiuminfo.org/standardsblog/ blockswwwoffle?

Martin Bähr — 2008-07-09T11:36:13

hi, is anyone here able to access www.consortiuminfo.org/standardsblog/ through wwwoffle? i keep getting this message below, and i just don't get why they would block a proxy, and also how they detect something anyways. what does wwwoffle do to the request? is it possible to make wwwoffle send out the request exactly as it received it from the browser? ----------------- error message: --------------------- Precondition Failed We're sorry, but we could not fulfill your request for /standardsblog/article.php?story=20080708052706429 on this server. We have established rules for access to this server, and any person or robot that violates these rules will be unable to access this site. To resolve this problem, please try the following steps: * Ensure that your computer is free of viruses, Trojan horses, spyware or any other sort of malicious software. * If you are using any sort of personal firewall or browser privacy software, check to ensure that its settings do not cause your web browser to i

Re: forcing charset of certain URLs

Andrew M. Bishop — 2008-06-22T15:39:50

In the particular index page that you mention WWWOFFLE will try and display the URLs in a "nice" format. This means that rather than just use the raw URLs which may contain URL-encoded characters this encoding will be undone. There is no good solution to the problem that you mention. The page could contain a mixture of different character sets and there is no solution to this. There are workarounds that you could try in place of the one that you already have. You could modify the page so that the raw URLs are displayed instead of decoded ones. You could add a "meta" tag to the index page HTML with the charset of your choice (see example below). An example meta tag that would probaably work for you would be:

Re: DontCache still ends up in outgoing

2008-03-07T05:39:38

Well, from the point of view of the user offline doing $ wwwoffle URL1 URL2 URL3... there is no feedback on the differing plights of the URLs, already decided by wwwoffle here before we go online, but not to be revealed to the user, who has just loaded tens of them for fetching, from his PDA. AMB> Shame that you don't have the latest version of the program AMB> rather than one that is nearly two years old. Don't worry, I use Debian Sid, the cutting edge. The maintainer will surely upgrade it when he sees this post.

Re: DontCache still ends up in outgoing

Paul Slootman — 2008-03-06T20:26:42

A 2.9d-1 version has been available for some time, unfortunately not for i386 because of a problem with the build daemon systems for the different architectures. I've just prodded some people who are hopefully in a position to fix this. Paul Slootman

Re: DontCache still ends up in outgoing

Andrew M. Bishop — 2008-03-06T19:58:15

You are correct, the WWWOFFLE server doesn't intend to do any fetching. But the wwwoffle program is correct because it says that it is requesting the URL - which it does. For the DontGet URLs it is difficult because if you have a replacement URL configured then you will get that. There is no way to know that it is the replacement and not the real thing. For the DontCache URLs it would be possible to recognise the error header in this case but I don't really see the point of it. You can only do that if the log files are readable though. It is a bug that a request is put into the outgoing directory. This happens because the request is for the WWWOFFLE refresh URL not the original URL. I have fixed this for the next version. It wouldn't work to return an error condition because things would fall over when multiple URLs were specified on the command line and only some were not to be fetched. I don't see the point of adding a feature that only partly works. Congratulations on the first post of th

DontCache still ends up in outgoing

2008-03-01T18:11:30

Complaint: offline both $ wwwoffle SomeDontGetURL $ wwwoffle SomeDontCacheURL both say Requesting ThatURL and return $?=0 to the shell, even though WWWOFFLE intends to do no such fetching. At least one can do # grep 'not to get' /var/log/syslog wwwoffles[5218]: The URL 'http://example.net/f.jpg' matches one in the list not to get. to know about the former, but what about the latter? The latter still ends up in http://localhost:8080/index/outgoing/ Clicking on it there, still here offline, says Your request for URL http://en.wikipedia.org/w/index.php?title=List_of_thinking_errors&action=edit failed because it is on the list of hosts and/or paths that are not to be cached and cannot be requested when offline. Well, OK, then it should be barred from ending up in outgoing too. OK, to check for the latter one would do, after fetching, # less +/not\ possible /var/log/syslog (note I use maximum debug level for my messages) Anyway, if the shell returned 1 and a message for both, one could much easie

Re: executives still use WWWOFFLE to keep track ofwhat's already read

Ian Stirling — 2007-11-28T19:03:58

I also find it invaluable for searching for stuff I've read. It's much easier to find that website you saw a few months ago and can't really remember the name of, if you can search your cache, not the whole internet.

executives still use WWWOFFLE to keep track ofwhat's already read

2007-11-27T18:46:50

Gentlemen, assuming that one has "the works" in the latest computing equipment and network connections, why would one still use WWWOFFLE? Well, certainly one cannot keep track of what articles one has already read on a sites with many articles. So with Purge { age=-1 } one will always know that one has already read which articles by looking at their link colors, (etc. as I mentioned in one of my previous postings.)

Re: keep wget --recursive away from "delete" links

Max Kirillov — 2007-11-05T03:03:51

BTW this does not prevent the robot from requesting (and fetching in online?) unspooled pages So user-agent is still better way

Re: 404 with wwwoffle, page delivered without

Karsten Kruse — 2007-10-16T23:40:41

Andrew M. Bishop schrieb: I didn't pay for realtime support anyways ;). Ok, i understand the problem and will inform the serveradmins. Hopefully they can fix it. Thanks so far. MFG, Karsten Kruse

Re: 404 with wwwoffle, page delivered without

Andrew M. Bishop — 2007-10-16T17:40:15

[Sorry for the long delay between the original message and it appearing on the list.] Karsten Kruse public.gmane.org> writes: I can explain the problem - the server is broken. If you try the two following URLs without WWWOFFLE you should see that the first of them doesn't work but the second one does: http://www.instructables.com/id/Bicyle-Power-for-Your-Television,-Laptop,-or-Cell-/ http://www.instructables.com/id/Bicyle-Power-for-Your-Television%2C-Laptop%2C-or-Cell-/ The two should give the same result since it is valid to encode the ',' character as the sequence %2C or not to encode it since ',' is not a special character in a path name. When you request the URL with WWWOFFLE it always uses the first format rather than the second so that whichever you use only a single one gets cached. Keeping a character or encoding it still references the same entity even if the URL looks different (unless the character is being used for its reserved purpose at that time). It isn't pos

404 with wwwoffle, page delivered without

Karsten Kruse — 2007-10-06T18:28:43

Hi, when i visit a certain URL i get error 404 with wwwoffle and the page without a proxy. Here is the URL: http://www.instructables.com/id /Bicyle-Power-for-Your-Television,-Laptop,-or-Cell-/?relatedLink (note that i broke the URL to fit it in here, it should read ...id/Bi...) Can anyone confirm and/or explain that? Here is my environment: Server - NetBSD 4.0_BETA2 i386 - wwwoffle version 2.9a Client - Debian 4.0r1 i386 - Mozilla Firefox 2.0.0.5 Thanks for your time. MFG, Karsten Kruse

Re: Not quite wwwoffle - for mobiles and low bandwidth use.

Ian Stirling — 2007-09-11T11:58:58

Fair enough. This is basically the sole thing I'm targetting. You have a mobile phone (or dialup), with a slow or expensive connection to the internet. You choose to install software on your phone and home computer (or a server farm, or in principle a paid service providing this cache) to reduce your bills, or speed up your internet. Webservers supporting rsync extensions is irrelevant, as you get better performance if it all goes through one proxy cache - as it can know exactly what files you have. As a goal, it's nice, but expecting ebay/microsoft/google/... to deploy it in the near term is unlikely. The one user proxy really is quite simple. Where it gets rather more complex is a public proxy with a few dozen users. Then rather more with a few thousand. Initial proof of concept would be basically using two stock WWWOFFLEs, and a couple of tiny servers that poke about in the cache. Then it would be nice to teach wwwoffle how to store, list and manage versioned pages (by date and po

Re: Not quite wwwoffle - for mobiles and lowbandwidth use.

Andrew M. Bishop — 2007-09-10T16:35:27

I remembered having seen the combination of rsync and HTTP before so it was easy for me to find. The reason that I say I don't plan to add it in to WWWOFFLE is that this is my standard reply when people ask me to extend WWWOFFLE way beyond its primary task. WWWOFFLE is a proxy cache that keeps a copy for offline viewing, allows offline requesting and automated online fetching and to help with this provides lots of options that reduce re-requesting cached URLs. There only needs to be one WWWOFFLE proxy and this resides at the user's end of the link. What you are suggesting is a pair of proxies, one each end of the low bandwidth link, that act together to minimise the data flowing over that link. The one on the internet end of the link (not the client end) could get new pages from the internet, or from WWWOFFLE or use the WWWOFFLE cache directly. The most general way of implementing it is for it to have an upstream proxy that it gets pages from. There is no need for it to keep cached pages itself, it

Re: Not quite wwwoffle - for mobiles and low bandwidth use.

Ian Stirling — 2007-09-08T11:32:13

Interesting, thanks! Googling had found me little. Initial design was planned to be a little server on host and client, that implemented this by changing the files in the wwwoffle cache, rather than properly implementing stuff. If eventually I generated a patch for WWWOFFLE, that mostly 'just worked', without greatly complicating the core logic, are there any circumstances in which it might be included? Thanks again for the leads!

Re: Not quite wwwoffle - for mobiles and lowbandwidth use.

Andrew M. Bishop — 2007-09-07T15:28:27

Have you seen the (no longer maintained) rsync-over-HTTP protocol that is at http://rproxy.samba.org/. This combines rsync and HTTP to be able to only transfer the updates just like you want. They describe just what you are looking for, but since nobody was interested about using it they seem to have given up. Before you ask, I don't plan to add it into WWWOFFLE.

Re: only-same-host-frames

Andrew M. Bishop — 2007-09-06T16:06:29

The only-same-host-images option is a FetchOptions option so it only works when WWWOFFLE is fetching a request made offline. When you are online there is no way for WWWOFFLE to know that a particular request is an iframe and not a normal page. Using the ModifyHTML option to block iframes (disable-iframe) or using the DontGet options to block based on URL would probably do most of what you want and would work when online as well.