gmane.comp.apache.mod-log-sql

Re: mod_log_sql: how to send patch for NAT proxies

Thomas Goirand — 2012-12-06T09:53:34

No pb.


I only wrote the score_board patch, which is available in
the Debian package of mod_log_sql, however, I am not the
author of the rest of it. This is maintained by upstream
author at:

http://www.outoforder.cc/projects/httpd/mod_log_sql/

The author is contactable through the mod_log_sql mailing
list, and he is usually quite responsive. I have added
the list to this email, so everyone knows about your intend
to patch mod_log_sql (I hope you don't mind me making your
email public).

If you would like to send patches against DTC, it's done
the same way: send a patch as an email attachment (see
below).

Note that the debian package I maintain isn't the very last
version of mod_log_sql, so you might want to use the latest
one from outoforder.cc.


Where exactly did you read this? What file, and line?


Just produce a patch against the source, and send it to
both upstream author through the mailing list, and open
a bug against the libapache-mod-log-sql package in the
Debian bug tracking system. Then it will be processed,
hopefully.


After a fresh clone of DTC, simply type:

make deb

and that will produce some debian packages for you to test.
Once you are done with your patch, you can do:

git commit -a

enter some comments in your $EDITOR, then save and quit.
Then use "git log" to find the last sha256 hash, and do:

git format-patch <sha256-hash-you-have-found-in-git-log>

then send over by email the git format-patch output.

Cheers,

Thomas
_______________________________________________
Download the latest version at http://www.outoforder.cc/projects/apache/mod_log_sql/

To unsubscribe send an e-mail to 
mod_log_sql-unsubscribe< at >lists.outoforder.cc

Re: mod_log_sql

Edward Rudd — 2012-04-24T14:40:28

The transfer log format only affects what is logged.  The table is always created the same way.  I'll have to check, but I may have changed this in the master branch.

Sent from my iPad

On Apr 24, 2012, at 9:36, "Springer, Carl" <Carl.Springer-+dhpKbBLSQ5a4t1NWSe2Bg< at >public.gmane.org> wrote:

_______________________________________________
Download the latest version at http://www.outoforder.cc/projects/apache/mod_log_sql/

To unsubscribe send an e-mail to 
mod_log_sql-unsubscribe-7qY7E20V6GW73k+5HYS8LQqVMODqnSLI< at >public.gmane.org

Re: Fwd: Bug#666797: Building for apache 2.4

Edward Rudd — 2012-04-21T20:29:25

Nice.  Feel free to fork mod log SQL on github and send a pull request.  It's up at https://github.com/OutOfOrder/mod_log_sql/

Sent from my iPad

On Apr 21, 2012, at 15:56, Thomas Goirand <thomas< at >goirand.fr> wrote:

_______________________________________________
Download the latest version at http://www.outoforder.cc/projects/apache/mod_log_sql/

To unsubscribe send an e-mail to 
mod_log_sql-unsubscribe< at >lists.outoforder.cc

Re: Fwd: Bug#666797: Building for apache 2.4

Thomas Goirand — 2012-04-21T19:56:01

Hum... I meant:

#if AP_MODULE_MAGIC_AT_LEAST(20111130-0)
return r->useragent_ip;
#else
return r->connection->remote_ip;
#endif

since 20111130-0 is the first time when the API change was introduced,
according to Arno Toll.

Thomas
_______________________________________________
Download the latest version at http://www.outoforder.cc/projects/apache/mod_log_sql/

To unsubscribe send an e-mail to 
mod_log_sql-unsubscribe< at >lists.outoforder.cc

Re: Fwd: Bug#666797: Building for apache 2.4

Thomas Goirand — 2012-04-21T17:23:18

Also, FYI, we could use this:

#if AP_MODULE_MAGIC_AT_LEAST(19980812,2)
return r->useragent_ip;
#else
return r->connection->remote_ip;
#endif

so that mod_log_sql would build on both old and new apache.

It'd be great to include this change upstream. Edward, do you think you
can release a new version of mod_log_sql with this change?

Also, I still carry my score_board patch in Debian, and it has never
been accepted. Do you feel that one day, it will? What should I add in
my patch to make it acceptable?

Cheers,

Thomas
_______________________________________________
Download the latest version at http://www.outoforder.cc/projects/apache/mod_log_sql/

To unsubscribe send an e-mail to 
mod_log_sql-unsubscribe< at >lists.outoforder.cc

Fwd: Bug#666797: Building for apache 2.4

Thomas Goirand — 2012-04-21T07:27:53

Hi,

Here's the patch from Arno Töll, so that mod_log_sql can be built for
Apache 2.4.

Thomas

-------- Original Message --------
Subject: Bug#666797: Building for apache 2.4
Resent-Date: Fri, 20 Apr 2012 23:48:04 +0000
Resent-From: Arno Töll <arno-8fiUuRrzOP0dnm+yROfE0A< at >public.gmane.org>
Resent-To: debian-bugs-dist-0aAXYlwwYIJuHlm7Suoebg< at >public.gmane.org
Resent-CC: Thomas Goirand <thomas-/lR9RVWmqZhGWvitb5QawA< at >public.gmane.org>
Date: Sat, 21 Apr 2012 01:35:21 +0200
From: Arno Töll <arno-8fiUuRrzOP0dnm+yROfE0A< at >public.gmane.org>
Reply-To: Arno Töll <arno-8fiUuRrzOP0dnm+yROfE0A< at >public.gmane.org>, 666797-61a8vm9lEZVf4u+23C9RwQ< at >public.gmane.org
To: Thomas Goirand <thomas-/lR9RVWmqZhGWvitb5QawA< at >public.gmane.org>
CC: 666797-61a8vm9lEZVf4u+23C9RwQ< at >public.gmane.org,  General mod_log_sql discussion
<mod_log_sql-7qY7E20V6GW73k+5HYS8LQqVMODqnSLI< at >public.gmane.org>, control-61a8vm9lEZVf4u+23C9RwQ< at >public.gmane.org

tags 666797 +upstream +patch
thanks

Hi Thomas,

please try the attached patch. I didn't verify whether it really does
what I think, but it compiles fine now. You may want to forward the
patch to upstream once you tested it.

Re: Bug#666797: Building for apache 2.4

Thomas Goirand — 2012-04-21T07:27:10

Thanks a lot for this patch, Arno.

Do you know how I can test if the module is building under 2.4? That'd
be cool to be able to do a conditional build, depending on the version,
and I believe that this might be a requirement upstream. Also, I'd be
happy to patch the source package now, so that only a rebuild will be
needed when you upload 2.4 to SID.

Cheers,

Thomas
_______________________________________________
Download the latest version at http://www.outoforder.cc/projects/apache/mod_log_sql/

To unsubscribe send an e-mail to 
mod_log_sql-unsubscribe< at >lists.outoforder.cc

Re: Bug#666797: Building for apache 2.4

Thomas Goirand — 2012-04-20T17:47:21

----- Original message -----

Thanks for your quick reply!

Just use the SID package, it's currently unmodified,
and you only will need to change the build-depends.

Thomas

_______________________________________________
Download the latest version at http://www.outoforder.cc/projects/apache/mod_log_sql/

To unsubscribe send an e-mail to 
mod_log_sql-unsubscribe-7qY7E20V6GW73k+5HYS8LQqVMODqnSLI< at >public.gmane.org

Re: High number of MySQL threads when usingmod_log_sql

Robert Krig — 2012-03-01T09:21:56

Thanks for clearing that up for me.
_______________________________________________
Download the latest version at http://www.outoforder.cc/projects/apache/mod_log_sql/

To unsubscribe send an e-mail to 
mod_log_sql-unsubscribe-7qY7E20V6GW73k+5HYS8LQqVMODqnSLI< at >public.gmane.org

Re: High number of MySQL threads when usingmod_log_sql

Edward Rudd — 2012-03-01T04:44:06

On Feb 29, 2012, at 23:19 , Thomas Goirand wrote:

MySQL does handle large # of connections very well in my experience.
Basically the architecture of mod_log_sql is to open and maintain one connection per apache preforked process. So, if you have a lot of traffic going on *now* you'll have a lot of preforked apache processes, thus a large # of mysql connections. If you have apache configured to have a large number of idle processes waiting for new traffic, then yes you'll encounter a large number of idle mysql connections.

There are a few other options if it is becoming a problem.

Use the mod_dbd support driver and a threaded apache MPM. Assuming your application can run in a threaded apache MPM (Worker or event) you can make use of the mod_dbd connection pooling and reduce your connections. This in *theory* should be thread safe due to mod_dbd's architecture. It's been a while since I write it, but I was pretty certain that I added mod_dbd support for a client who wanted to run in threaded mode and use connection pooling.

Another option is to use the force preserve option, and import the preserve files at a regular interval. You do lose some "real-timeness" of the data, but it will reduce connection load to the server.

Like the preserve option, you can use the "cli utility" to import raw apache access logs in. (only available in the latest git master, not in a released version). This was another client request. They moved to an infrastructure that didn't allow for direct db connections (Amazon Cloud), so they wanted a CLI script that would parse the apache logs into the mod_log_sql db format. The latest git code also includes a more powerful configuration syntax to allow better customizing of the db format.

Edward Rudd
OutOfOrder.cc

_______________________________________________
Download the latest version at http://www.outoforder.cc/projects/apache/mod_log_sql/

To unsubscribe send an e-mail to
mod_log_sql-unsubscribe-7qY7E20V6GW73k+5HYS8LQqVMODqnSLI< at >public.gmane.org

Re: High number of MySQL threads when usingmod_log_sql

Thomas Goirand — 2012-03-01T04:19:37

Hi,

FYI, I'm the Debian maintainer for the mod_log_sql package, and I've
been using it for many, many years on all of my servers (I'm tempted to
say a decade, but I'm not sure that's right).

On 02/29/2012 05:47 PM, Robert Krig wrote:

Frankly, I don't see why this should be worrisome.

On 02/29/2012 05:47 PM, Robert Krig wrote:

Yes, for each apache thread, there's a MySQL connection being made.

On 02/29/2012 05:47 PM, Robert Krig wrote:

If by optimize, you mean connecting to MySQL when an apache thread needs
to log something, then immediately after, disconnect, then you'd be
slowing down things. The way things are done right now really *is* the
optimized way.

On 02/29/2012 05:47 PM, Robert Krig wrote:

It is *not* a problem. Just make sure that the max_connection directive
in your MySQL server is set to a high number. The default (100
connections) really is too low, and should be increased to something
like 500 or 1000. My experience shows that 500 MySQL connections isn't
an issue anyway.

I hope that helps.

Thomas
_______________________________________________
Download the latest version at http://www.outoforder.cc/projects/apache/mod_log_sql/

To unsubscribe send an e-mail to 
mod_log_sql-unsubscribe-7qY7E20V6GW73k+5HYS8LQqVMODqnSLI< at >public.gmane.org

High number of MySQL threads when using mod_log_sql

Robert Krig — 2012-02-29T09:47:15

Hi there. I'm the admin for a very high volume website. Wie have 4 
webservers which we use to distribute the load of our website. We would 
like to use mod_log_sql to log apache logs to a central MySQL database.

So far, I've just enabled on of our webservers to log to a MySQL server 
running on a different host.

What I find a bit worrying, is the number of MySQL threads which this 
causes.
The one webserver I have logging to MySQL gets about 30-50 hits per 
second. And already this is causing about 150-200 MySQL server threads.

The load on the MySQL server is extremely low.
The host running the MySQL server is a 2x6 core Xeon Cpu with 
Hyperthreading, totalling 24 threads and equipped with 48GB of RAM.

What I find puzzling is that our central MySQL Database Server, which we 
use as a backend for our website, uses only about 12-30 MySQL threads in 
total. Thats for all our webservers.
Granted, this central database server uses SSD Harddrives for the 
database and handles 1500 Queries per second. But still.

Is it normal for mod_log_sql to be using so many MySQL threads? Is there 
a way I could optimise this?
Perhaps a tweak in the mysql server config or a parameter for 
mod_log_sql which I could try?

I don't know if its gonna cause problems if MySQL has so many threads 
open. Especially considering that most of those seem to be "sleeping" 
when monitored with MyTOP.

The MySQL server I use for logging, is on its own and is only used for 
logging. e.g.

I'm worried about adding our other apache servers into the mix. 
Especially considering that the last two actually deal with 150-200 hits 
per second each.
_______________________________________________
Download the latest version at http://www.outoforder.cc/projects/apache/mod_log_sql/

To unsubscribe send an e-mail to 
mod_log_sql-unsubscribe-7qY7E20V6GW73k+5HYS8LQqVMODqnSLI< at >public.gmane.org

mod_log_sql 1.101 With mod_dbd Support

Rene Kanzler — 2011-08-20T11:54:26

Hello 

Because Apache segfaults with mod_log_sql and mod_dbd and
the latest mod_log_sql version from CVS did not work for me.
Here is the patch to get mod_dbd support in mod_log_sql 1.101.
Of course it includes the "VH_GECOS" and "X-Forwarded-For" header 
stuff from my previous patch for mod_log_sql 1.100.

Hopefully somebody finds this useful. 
The patch is attached ;-) 

Bye
Rene_______________________________________________
Download the latest version at http://www.outoforder.cc/projects/apache/mod_log_sql/

To unsubscribe send an e-mail to 
mod_log_sql-unsubscribe-7qY7E20V6GW73k+5HYS8LQqVMODqnSLI< at >public.gmane.org

Re: [error] Log Format 'i' unknown

Dan Bunyard — 2011-01-02T21:05:52

That got it, thanks so much!!!  I knew I was missing something stupid.
 Thanks again!

On 01/02/2011, Edward Rudd <urkle-l4c2hbCm6dz5FsjnhOxw3g< at >public.gmane.org> wrote:

Re: [error] Log Format 'i' unknown

Edward Rudd — 2011-01-02T19:43:49


You need to load the mod_log_sql_logio module after loading mod_log_sql.


Edward Rudd
OutOfOrder.cc
317-674-3296





_______________________________________________
Download the latest version at http://www.outoforder.cc/projects/apache/mod_log_sql/

To unsubscribe send an e-mail to 
mod_log_sql-unsubscribe-7qY7E20V6GW73k+5HYS8LQqVMODqnSLI< at >public.gmane.org

[error] Log Format 'i' unknown

Dan Bunyard — 2011-01-02T17:41:53

Alright I have been trying to figure this out for a few days now and have
not been able to.  I am attempting to get mod_log_sql working correctly on
my Fedora 14 box.  Everything is working fine until I enable the logging of
bytes_in and bytes_out using "i" and "o" respectively.  As soon as I add
these two letter to the LogSQLTransferLogFormat configuration option my
Apache error_log is flooded with:

[Sun Jan 02 10:26:20 2011] [error] Log Format 'i' unknown
[Sun Jan 02 10:26:20 2011] [error] Log Format 'o' unknown
[Sun Jan 02 10:26:20 2011] [error] Log Format 'i' unknown
[Sun Jan 02 10:26:20 2011] [error] Log Format 'o' unknown
[Sun Jan 02 10:26:23 2011] [error] Log Format 'i' unknown
[Sun Jan 02 10:26:23 2011] [error] Log Format 'o' unknown
[Sun Jan 02 10:26:24 2011] [error] Log Format 'i' unknown
[Sun Jan 02 10:26:24 2011] [error] Log Format 'o' unknown
[Sun Jan 02 10:26:37 2011] [error] Log Format 'i' unknown
[Sun Jan 02 10:26:37 2011] [error] Log Format 'o' unknown

messages.  I am using version 1.101 and have the mod_logio module disabled.
The fields "bytes_in" and "bytes_out" both exist in the table I am logging
to.  Is there something else I need to set for this to work properly?  My
complete httpd.conf file can be found here (username/passwords/emails
removed): http://dbunyard.homeip.net/stuff/httpd.conf and my phpinfo page
can be found here: http://dbunyard.homeip.net/phpinfo.php  Thanks!!
--
Dan

http://www.bunyardpics.com
http://www.linkedin.com/in/danbunyard
http://www.danodemano.com
http://www.dansrandomness.com
http://www.danandshelley.com

This is not a problem that requires infinite wisdom, Benj. This is a problem
that requires enough neural organization to qualify as a vertebrate,
apparently a stretch for some folks these days.
~Cecil Adams.
_______________________________________________
Download the latest version at http://www.outoforder.cc/projects/apache/mod_log_sql/

To unsubscribe send an e-mail to 
mod_log_sql-unsubscribe-7qY7E20V6GW73k+5HYS8LQqVMODqnSLI< at >public.gmane.org

Any plans to support %D ?

James Po — 2010-10-27T13:22:32

Hi,

just started using mod_log_sql, very easy to set up.

I'd be very interested in getting more details on the amount of time a
response takes, similar to the %D operative in flatfile logging.

http://httpd.apache.org/docs/2.0/mod/mod_log_config.html

%...D               The time taken to serve the request, in microseconds.


Any chance of this enhancement being made?


Thanks


James
_______________________________________________
Download the latest version at http://www.outoforder.cc/projects/apache/mod_log_sql/

To unsubscribe send an e-mail to 
mod_log_sql-unsubscribe-7qY7E20V6GW73k+5HYS8LQqVMODqnSLI< at >public.gmane.org

Re: configure: WARNING: ** libDBI client librariesnot found!

Horace Ho — 2010-08-24T13:40:04

Thank you very much!

On Tuesday, August 24, 2010, Edward Rudd <urkle< at >outoforder.cc> wrote:
_______________________________________________
Download the latest version at http://www.outoforder.cc/projects/apache/mod_log_sql/

To unsubscribe send an e-mail to 
mod_log_sql-unsubscribe< at >lists.outoforder.cc

Re: configure: WARNING: ** libDBI client librariesnot found!

Edward Rudd — 2010-08-24T12:23:47

_______________________________________________
Download the latest version at http://www.outoforder.cc/projects/apache/mod_log_sql/

To unsubscribe send an e-mail to 
mod_log_sql-unsubscribe-7qY7E20V6GW73k+5HYS8LQqVMODqnSLI< at >public.gmane.org

Re: configure: WARNING: ** libDBI client libraries not found!

Rene Kanzler — 2010-08-24T10:56:15

If you only want to use MySQL ignore the warning.

Bye
Rene

_______________________________________________
Download the latest version at http://www.outoforder.cc/projects/apache/mod_log_sql/

To unsubscribe send an e-mail to 
mod_log_sql-unsubscribe-7qY7E20V6GW73k+5HYS8LQqVMODqnSLI< at >public.gmane.org

Re: configure: WARNING: ** libDBI client librariesnot found!

Horace Ho — 2010-08-24T09:41:39

_______________________________________________
Download the latest version at http://www.outoforder.cc/projects/apache/mod_log_sql/

To unsubscribe send an e-mail to 
mod_log_sql-unsubscribe-7qY7E20V6GW73k+5HYS8LQqVMODqnSLI< at >public.gmane.org