gmane.comp.web.squid.devel

A youtube and windows update cache thing.

Eliezer Croitoru — 2013-05-25T00:17:47

Since the development of StoreID there were some advances in couple 
areas and it seems like the idea of 206 responses caching can be moved 
on to the next level.

Based on my work with GrreasySpoon for youtube (mentiond here: 
http://wiki.squid-cache.org/ConfigExamples/DynamicContent/Coordinator#Implementing_ICAP_solution) 
I took it to the next level and I am now trying to proof with almost the 
same way another thing.

The idea is to proof that caching partial content as a stand alone 
object in squid is better then trying to cache the full object each time 
it's being requested.
This approach is basically meant to work only for windows updates which 
tries to do all updates in 206 chunks.
They now do use a lot of SSL traffic in order to secure their clients 
but still I have seen non SSL updates.

How will it be done?
in youtube case we have used only the url to get info on the object that 
the client requests.

Squid in a way can cache everything if there was someguy inside the 
system who actually will

[PATCH] Support forwarding intercepted but not bumped connections to cache_peers

Alex Rousskov — 2013-05-24T23:58:49

Hello,

    When talking to a cache_peer (i.e., sending a CONNECT request before
tunneling the transaction), tunnel code is using a clever hack: Squid
does not parse the CONNECT response from peer but blindly forwards it to
the client. This works great and simplifies code a lot, except when the
client connection was intercepted and, hence, the client did not send a
CONNECT request and is not expecting a CONNECT response.

In those situations, the patch accumulates, parses, and strips the peer
CONNECT response (or closes connection on errors).

The existing tunnel I/O code is too simple to accommodate that task --
it cannot accumulate read data (its I/O buffers work in lockstep
fashion, writing everything it reads before reading again). Instead of
rewriting the entire tunnel code to use more complex buffers, I added a
temporary accumulation buffer for the CONNECT response. That buffer is
not allocated unless it is needed and does not grow beyond
SQUID_TCP_SO_RCVBUF size, just like the simple buffers.


Thank

[PATCH] mDNS support

Amos Jeffries — 2013-05-24T16:51:09

The last outstanding reason to retain dnsserver component in Squid seems 
to be lack of mDNS support in our client resolver. This patch seeks to 
close that hole and add mDNS support to the Squid internal DNS resolver.

* adds the mDNS multicast group IPs as always-present entries in the 
nameservers list.

* filters each request. ".local" lookups are permitted to both the mDNS 
resolvers and the recursive resolvers, other requests are only permitted 
to the regular recursive resolvers.

I have not done much testing of this yet. However the DNS protocol 
operations are completely unchanged, and the only questions are whether 
the nameservers responses are accepted back into Squid or prevented by 
the ignore_unknown_nameservers option (or not)  - that depends on the 
mDNS responses. And whether the mDNS produces NXDOMAIN in a useful way 
for Squid.

Amos

=== modified file 'src/dns_internal.cc'
--- src/dns_internal.cc2013-03-03 07:10:22 +0000
+++ src/dns_internal.cc2013-05-24 16:42:52 +0000
< at >< at > -141,6 +141,7

Possible external_acl_type format tag

quickdry21 — 2013-05-23T23:51:05

I was wondering if it would be possible to include an a HTTP response status
line tag to the external_acl_type, or even possibly the HTTP-Version,
Status-Code and Reason-Phrase as three separate format tags.

The reason I ask is I'm trying to hook some of my code that needs the status
code from the response into an external_acl_type call. This doesn't really
seem like the purpose of the external_acl_type, but I'm hoping it will suit
my purpose. It doesn't seem like an unreasonable thing to be able do,
considering you can already pass response headers as format tags.

If this is possible to do, I'm willing to write the additional code myself
if someone could point me in the right direction.





--
View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Possible-external-acl-type-format-tag-tp4660222.html
Sent from the Squid - Development mailing list archive at Nabble.com.

[PATCΗ] Quoted values in squid.conf

Tsantilas Christos — 2013-05-23T19:38:10

This patch :

- adds support for quoted values in the entire squid.conf

- warn about or prohibit values that can no longer be interpreted as
either quoted strings or simple tokens

- support file:"path/to/file.name" syntax to load external configuration
files

- support macros in "double quoted" values

- support 'single quoted' values that do not expand macros

- replaces the strtok() calls with calls to the new
ConfigParser::NextToken()

- modify strtokFile to use new ConfigParser::NextToken()

- Add the new configuration_includes_quoted_values configuration option,
to control the squid parser behaviour. If set to on Squid will recognize
each "quoted string" after a configuration directive as a single parameter.


This is a Measurement Factory project

Build failed in Jenkins: 3.HEAD-amd64-CentOS-5.3 #2426

2013-05-23T12:53:25

See <http://build.squid-cache.org/job/3.HEAD-amd64-CentOS-5.3/2426/changes>

Changes:

[Christos Tsantilas] Cleanup: Merge AccessLogEntry 'helperNotes' and 'configNotes' members to 'notes' member

There is not any need to store notes added using Note cfg option and notes added
from helper to separated member. This patch merge them to the same
AccessLogEntry::note member.

This is a Measurement Factory project

------------------------------------------
[...truncated 56493 lines...]
 ( cd '<http://build.squid-cache.org/job/3.HEAD-amd64-CentOS-5.3/ws/btlayer-05-nodeps-esi/squid-3.HEAD-BZR/_inst/share/man/man8'> && rm -f ext_file_userip_acl.8 )
make[4]: Leaving directory `<http://build.squid-cache.org/job/3.HEAD-amd64-CentOS-5.3/ws/btlayer-05-nodeps-esi/squid-3.HEAD-BZR/_build/helpers/external_acl/file_userip'>
Making uninstall in kerberos_ldap_group
make[4]: Entering directory `<http://build.squid-cache.org/job/3.HEAD-amd64-CentOS-5.3/ws/btlayer-05-nodeps-esi/squid-3.HEAD-BZR/_build/helpers/external_acl/kerber

Recommended method for building your own helpers?

Chris Ross — 2013-05-22T16:45:56

  I've written a log_daemon helper, and am writing an external_url helper.  So far, I've just been patching the configure and Makefile's in the squid sources, then moving my code into the expected place in the tree.  But, I was wondering if there is another better/easier method to build helpers _outside_ of the squid source tree.  Is there a recommended system to effectively reach into a squid source tree from outside to get headers and library code?

  Thanks….

                                       - Chris

external_acl helper question

Chris Ross — 2013-05-22T15:48:15

  I'm writing an external_acl helper for a project where we want to make decisions about choosing an outgoing address based on the destination of the connection.  I've written a program that will take in an argument (from the acl) and has a %DST format.

  However, in my testing, it's never used.  It starts up, because I set children-startup=1, but looking at the log I'm never seeing any of the debugging printf's I put in it that I do see if I run it by hand and feed it data.

  I noticed inside of forward.cc, in getOutgoingAddress, the ACL checking it's doing calls cf->fastCheck().  Does that mean that it will avoid calling "slow" acl mechanisms for some reason?  Or am I inferring too much?

  Either way, I wanted to ask, because I can tell that I'm seeing connections and it's trying to choose an outgoing address, but seems to never choose the ones attached to the external_acl helper, and seems to never inquire of it.

  Thanks.

                                    - Chris

--
external_acl_type region chil

Build failed in Jenkins: 3.HEAD-amd64-CentOS-5.3 #2424

2013-05-22T02:41:22

See <http://build.squid-cache.org/job/3.HEAD-amd64-CentOS-5.3/2424/changes>

Changes:

[Amos Jeffries] Detect and use -march=natuve when possible

Clang++ 3.2 fails to detect some CPUs correctly and requires the
additional checks enabled by this option to build working executables.

This option supported by GCC 4.3 and later enables additional CPU
detection and enables CPU-specific optimizations. In the interests of
better performance this patch enables it whenever is it available and
possible to use (cross-compilers cannot use it).

[Amos Jeffries] Bug 1991: kqueue causes SSL to hang

Compare the code in normal select and epoll v.s. kqueue. The select use a 0
wait time to get out of select wait in order to handle a list of read_pendings.
However, epoll add the read_pending to read and write event monitor. At a first
look, this seems strange as why read pending has anything to do with write. It
became obvious when the write ready event is triggered. During a write ready
event, if read_pending is on, the read

[RFC] remove X-Cache and X-Cache-Lookup headers

Amos Jeffries — 2013-05-20T14:23:00

These two headers are marked as experimental, but have been in there so 
long they seem to have become defacto standards for HIT/MISS debugging.

1) they are not necessary on 99% (or more) of traffic.
2) they *actualy* present bogus information, and most of the blog / 
tutorial / how-to I have found document some strange behaviour which is 
*not* how Squid has aways used them.
3) X-Cache-Lookup, when it works at all it is redundant behind X-Cache. 
When it does not work presents "MISS", always.


X-Cache-Lookup I think we can just drop.

X-Cache is trickier due to the defacto nature it has gained. I think we 
can suppress its addition unless TRACE method or Max-Forwards header are 
present as a sign of manual debugging. Alternatively we could make it 
configurable. Either way the minor information leak and bandwidth waste 
that it presents can be resolved.
  PS. if X-Cache remains I think we should extend it slightly to 
indicate REFRESH operations not just HIT/MISS.

Amos

Build failed in Jenkins: 3.3-matrix » ubuntu-precise-x64 #53

2013-05-20T01:30:28

See <http://build.squid-cache.org/job/3.3-matrix/./label=ubuntu-precise-x64/53/changes>

Changes:

[Amos Jeffries] Allocate ClientInfo::hash.key using malloc() instead of new char[]

... because it is destroyed using safe_free() instead of delete[].

Added ClientInfo construction/destruction debugging to find leaking objects.

[Amos Jeffries] Bug 3851: Delay Pool class 5 tag:levels displayed incorrectly in cache manager

[Amos Jeffries] Use case-insensitive comparison for HTTP header names in *_header_access

[Amos Jeffries] Bug 3744: squid terminated: FATAL: Bungled (null) line 3: sslproxy_cert_sign signTrusted all

This bug is a Makefile dependencies  problem.

- The cf_gen includes the  cf_gen_defines.cci so this file should included in
cf_gen dependencies.
- Currently the cf_gen_defines.cci exist in cf_gen.$(OBJEXT) dependencies but
does not have any effect because the obj file never build and used.
- Also the  cf_gen_defines.cci file depends on autoconf.h so this file should
added to to cf_gen_defines.c

Build failed in Jenkins: 3.2-matrix » master #314

2013-05-20T00:22:15

See <http://build.squid-cache.org/job/3.2-matrix/./label=master/314/changes>

Changes:

[Amos Jeffries] Allocate ClientInfo::hash.key using malloc() instead of new char[]

... because it is destroyed using safe_free() instead of delete[].

Added ClientInfo construction/destruction debugging to find leaking objects.

[Amos Jeffries] Remove origin_tries limiter on forwarding

This limit seems to have been set to prevent large amount of looping when
DIRECT attempts fail under the old model of constant DNS lookups and
retries.

However it is hard-coded and has no configuration knob visible. Under
the curent model of all destinations being enumerated once and tried
sequentially this protection would seem to be no longer necessary and
somewhat harmful as it will be preventing retries reaching destinations
with more than 2 unreachable IPs (think 3 IPv6 and an IPv4 on IPv4-only
network).

[Amos Jeffries] Fixed leaking configurable SSL error details.

Trunk r11496 "Configurable SSL error details messages" correctly d

Build failed in Jenkins: 3.HEAD-amd64-CentOS-5.3 #2421

2013-05-18T18:19:07

See <http://build.squid-cache.org/job/3.HEAD-amd64-CentOS-5.3/2421/changes>

Changes:

[Amos Jeffries] autoconf.h exist in buildir, not srcdir.

rev.12827 only works for in-tree builds.

------------------------------------------
[...truncated 55745 lines...]
make[4]: Leaving directory `<http://build.squid-cache.org/job/3.HEAD-amd64-CentOS-5.3/ws/btlayer-05-nodeps-esi/squid-3.HEAD-BZR/_build/helpers/external_acl/LDAP_group'>
Making uninstall in SQL_session
make[4]: Entering directory `<http://build.squid-cache.org/job/3.HEAD-amd64-CentOS-5.3/ws/btlayer-05-nodeps-esi/squid-3.HEAD-BZR/_build/helpers/external_acl/SQL_session'>
 ( cd '<http://build.squid-cache.org/job/3.HEAD-amd64-CentOS-5.3/ws/btlayer-05-nodeps-esi/squid-3.HEAD-BZR/_inst/libexec'> && rm -f ext_sql_session_acl )
 ( cd '<http://build.squid-cache.org/job/3.HEAD-amd64-CentOS-5.3/ws/btlayer-05-nodeps-esi/squid-3.HEAD-BZR/_inst/share/man/man8'> && rm -f ext_sql_session_acl.8 )
make[4]: Leaving directory `<http://build.squid-cache.org/job/3.HEAD-amd64

[PATCH] Enable -march-native when available

Amos Jeffries — 2013-05-18T11:36:15

Clang++ 3.2 fails to detect some CPUs correctly and requires the 
additional checks enabled by this option to build working executables.

This option supported by GCC 4.3 and later enables additional CPU 
detection and enables CPU-specific optimizations. In the interests of 
better performance this patch enables it whenever is it available and 
possible to use (cross-compilers cannot use it).

Amos

=== modified file 'configure.ac'
--- configure.ac2013-05-15 15:41:43 +0000
+++ configure.ac2013-05-18 09:48:20 +0000
< at >< at > -35,9 +35,20 < at >< at >
 AC_LANG([C++])
 AC_CANONICAL_HOST
 
-# might be cross-compiling
+# Clang 3.2 on some CPUs requires -march-native to detect correctly
+# GCC 4.3+ can also produce faster executables when its used
+SQUID_CC_CHECK_ARGUMENT([ac_cv_check_marchnative],[-march=native])
+
+# might be cross-compiling.
 if test "x$HOSTCXX" = "x"; then
   HOSTCXX="$CXX"
+  if test "x$ac_cv_check_marchnative" = "xyes"; then
+    CXXFLAGS="$CXXFLAGS -march=native"
+  fi
+fi
+if test "x$ac_cv_check_marchnat

Build failed in Jenkins: 3.HEAD-amd64-FreeBSD-9.0-clang #246

2013-05-16T19:24:19

See <http://build.squid-cache.org/job/3.HEAD-amd64-FreeBSD-9.0-clang/246/changes>

Changes:

[Christos Tsantilas] Bug 3744: squid terminated: FATAL: Bungled (null) line 3: sslproxy_cert_sign signTrusted all

This bug is a Makefile dependencies  problem.

- The cf_gen includes the  cf_gen_defines.cci so this file should included in
cf_gen dependencies.
- Currently the cf_gen_defines.cci exist in cf_gen.$(OBJEXT) dependencies but
does not have any effect because the obj file never build and used.
- Also the  cf_gen_defines.cci file depends on autoconf.h so this file should
added to to cf_gen_defines.cc dependencies.
All of the sources has the autoconf.h file in their dependencies.
But the cf_gen_defines.cci is auto-generated and does not exist when the
dependencies computed.

This is a Measurement Factory project

[Christos Tsantilas] "LruMap bug fix and other polishing touches" patch: fix compile error

[Amos Jeffries] Release Notes: rebuild HTML notes for 3.4

[Christos Tsantilas] LruMap bug fix and other po

[PATCH] note acl

Tsantilas Christos — 2013-05-16T16:41:44

Hi all,

I am attaching two patches here:

1) Merge AccessLogEntry 'helperNotes' and 'configNotes' members to
'notes' member (trunk-merge-cfg-helper-notes-v2.patch)

There is not any need to store notes added using Note cfg option and
notes added from helper to separated member. This patch merge them to
the same  AccessLogEntry::note member.


2) note acl ( trunk-note-ACL-v5.patch )
Syntax:
   acl aclname note name [value ...]

Without values, matches any annotation with a given name. With value(s),
matches any annotation with a given name that also has one of the given
values. Annotation sources include note and adaptation_meta directives
as well as helper and eCAP responses.

The second patch based on the first. If the first rejected needs to be
modified a little.

This is a Measurement Factory project

[RFC] ConfigParser.* vs Parsing.*

Amos Jeffries — 2013-05-16T11:20:40

We seem to have a strange separation of logics between the ConfigParser 
class and the Get*() functions in Parsing.h/.cc.

Is anyone able to explain why they are not all just methods of 
ConfigParser ?

Unless there is an reason not to I am proposing we make them such and 
convert their strtok() calls to using the ConfigParser strtokFile() methods.

Amos

Build failed in Jenkins: 3.HEAD-amd64-CentOS-5.3 #2413

2013-05-15T21:56:33

See <http://build.squid-cache.org/job/3.HEAD-amd64-CentOS-5.3/2413/changes>

Changes:

[Christos Tsantilas] LruMap bug fix and other polishing touches

The  LruMap::findEntry may return destroyed entries to the caller.

------------------------------------------
[...truncated 26848 lines...]
libtool: link: ( cd ".libs" && rm -f "liblog.la" && ln -s "../liblog.la" "liblog.la" )
make[4]: Leaving directory `<http://build.squid-cache.org/job/3.HEAD-amd64-CentOS-5.3/ws/btlayer-02-maximus/squid-3.HEAD-BZR/_build/src/log'>
Making all in ipc
make[4]: Entering directory `<http://build.squid-cache.org/job/3.HEAD-amd64-CentOS-5.3/ws/btlayer-02-maximus/squid-3.HEAD-BZR/_build/src/ipc'>
/bin/sh ../../libtool --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -DDEFAULT_STATEDIR=\"<http://build.squid-cache.org/job/3.HEAD-amd64-CentOS-5.3/ws/btlayer-02-maximus/squid-3.HEAD-BZR/_inst/var/run/squid\">  -I../../.. -I../../../include -I../../../lib -I../../../src -I../../include     -I/usr/include/libxml2    -I/usr/include/libxml2

Build failed in Jenkins: 3.HEAD-amd64-FreeBSD-7.2 #1852

2013-05-15T19:10:32

See <http://build.squid-cache.org/job/3.HEAD-amd64-FreeBSD-7.2/1852/changes>

Changes:

[Christos Tsantilas] LruMap bug fix and other polishing touches

The  LruMap::findEntry may return destroyed entries to the caller.

[Christos Tsantilas] Bug 3816: SSL_get_certificate call inside Ssl::verifySslCertificate crashes squid, part3

Change the libraries order in LIBS variable inside SQUID_CHECK_OPENSSL_GETCERTIFICATE_WORKS check.
Looks that play a role in some cases (when openssl provided only as static
library in my tests).

[Christos Tsantilas] Bug 3759: OpenSSL compilation error on stock Fedora17, RHEL, CentOS 6 systems

OpenSSL-1.0.x has changes in TXT_DB interface over the earlier openSSL releases.
Also looks that the IMPLEMENT_LHASH_* macros are not correctly implemented and
causes compile failures.
Some of the linux distributions to overcome the above problems trying to patch
openSSL SDK. For squid this is means that the current checks based on openSSL
version can not work.

This patch try to detect at c

Build failed in Jenkins: 3.HEAD-amd64-CentOS-5.3 #2411

2013-05-14T21:20:57

See <http://build.squid-cache.org/job/3.HEAD-amd64-CentOS-5.3/2411/changes>

Changes:

[Amos Jeffries] Drop terminal comments and garbage options from cache_peer lines

[Amos Jeffries] Release Notes: update status of Squid-2 options

[Amos Jeffries] Port from 2.6: external acl %ACL and %DATA tags

[Christos Tsantilas] Bug 3816: SSL_get_certificate call inside Ssl::verifySslCertificate crashes squid, part2

This patch try to avoid using the SSL_get_certificate function. While configures
squid run tests:
- to examine if the workaround code can be used
- to detect buggy SSL_get_certificate

Inside Ssl::verifySslCertificate try to use workarround code and if this is not
possible uses the SSL_get_certificate if it is not buggy, else hit an assertion

This is a Measurement Factory project

------------------------------------------
[...truncated 9745 lines...]
Making uninstall in LDAP_group
make[4]: Entering directory `<http://build.squid-cache.org/job/3.HEAD-amd64-CentOS-5.3/ws/btlayer-00-default/squid-3.HEAD-BZR

Build failed in Jenkins: 3.HEAD-amd64-CentOS-5.3 #2406

2013-05-13T16:28:23

See <http://build.squid-cache.org/job/3.HEAD-amd64-CentOS-5.3/2406/changes>

Changes:

[Amos Jeffries] Fix pointless comparison of unsigned integer with zero in rev.12805

CommIoCbParams::size and local variable writtenSize are both size_t
which is unsigned and cannot be less than 0.

This produces compiler warnings/errors in ICC.

[Alex Rousskov] Fixed leaking configurable SSL error details.

Trunk r11496 "Configurable SSL error details messages" correctly disabled
collection of HTTP statistics for non-HTTP header fields, such as configurable
SSL error details. However, it also incorrectly disabled deletion of those
non-HTTP header fields.

Configurable SSL error details are only created during [re]configuration time,
so the leak went unnoticed since 2011-06-17, but the same bug caused a major
runtime annotation leak later (r12413) until the new annotation code was
redesigned to avoid using HttpHeader (r12779).

------------------------------------------
[...truncated 10948 lines...]
make[4]: Entering direc