gmane.comp.jakarta.log4j.user

Re: network logging performance

Paul Smith — 2008-09-05T02:44:26

[oops, I originally sent this reply direct to Johannes. Copying again the list] On 04/09/2008, at 4:19 PM, Johannes Gutleber wrote: fantastic. awesome to hear about this sort of environment. Really appreciate you taking the time to respond in depth here. Are you able to articulate where you thought the bottleneck within the Java process was ? What JRE were you using? Was it GC overhead? Was it log4j itself, or the custom built collector that was the problem? Which version of Chainsaw did you use? Was it the one inbuilt into log4j, or the newer Chainsaw v2? The latter uses a cyclic buffer so as not to consume too much memory. I do agree though that even Chainsaw v2 is far from perfect.. :) My vision, ridiculous as this may sound, is that log4j should be able to support an environment in the new cloud environments. Imagine the Hadoop clusters that are out there and making sense of wtf happened during the dissemination of the MapReduce portions, with thousands of host nodes logging and, somehow, an engineer able to 'see' what happened on his job centrally (even if it's not real time). With a Many->1 collector the collector host is going to have be one grunty box.... Pinpoints current design uses ActiveMQ internally. The Receivers inside accept the remote event and place it on the local log4j bus (just like it was placed on the remote bus that went to the network appender). Internally a local 'appender' routes the event to an internal JMS topic, with ActiveMQ buffering the events to a local persistent store temporarily. A single topic listener then tries to chew through the received events to index them (indexing is always going to be expensive hence a producer/consumer pattern). This temporary JMS buffer I am hoping will allow the receiving of events much faster than they can be consumed which is sort of what JMS (and ActiveMQ) is designed to handle, although it'll be interesting to see how it translate into practice. Where I work that is driving me to develop Pinpoint has no where near the load environment that yours has though. I'd love to hear some actual logging numbers, that is, how many events (log lines would do) that each host on average is producing and their combined total (even a peak load for a given hour or something would be interesting). Your environment really is exactly what I had in mind to support with Pinpoint, and I'm very much focussed on making it as non-intrusive as possible. cheers, Paul Smith

RE: network logging performance

Bruno Melloni — 2008-09-04T13:23:04

**Theoretically**, a logging event should be a 'fire and forget' activity from the perspective of the application. That means that you should be able to submit the logging event to the 'logging system' without blocking, and then the 'logging system' should be able to deliver those logging events to the final destination on its own thread(s) and at its own speed. As long as the average volume of logging does not exceed the capacity of the actual logging mechanism (i.e.: over the network) then there ought to be no impact whatsoever to the application (except perhaps for the cpu overhead of the physical logging thread(s)). I don't know log4j well enough to say whether it is already doing this isolation of 'log event submission' vs 'log event processing'. If it doesn't, you should be able to have your application submit all logging requests through a class written by you that puts all requests into a memory cache, ensuring no blocking. Then have one or more separate threads handling the 'log event processing' independently from the rest of the app. For safety, you should probably add some code that monitors memory, disk and network utilization by this approach and notify you (perhaps by email) when things get backed up - that way you'll detect any volume-caused issues before they have a chance to impact the app. If you really wanted to be bulletproof, implement the cache to use memory first (up to some limit) and then overflow to a local file cache. That way if your network gets bogged down by issues other than your app's you would still be able to continue functioning while the network recovers. I know that did not answer your immediate question, but it should help some in making your choices. bruno -----Original Message----- From: Paul Smith [mailto:psmith< at >aconex.com] Sent: Wednesday, September 03, 2008 8:31 PM To: Log4J Users List Subject: network logging performance If anyone out there is using "Logging over the network" of any form (socket, JMS, Multicast, syslog appenders etc), this topic is for you. I'm wondering whether people could comment on their experience setting up high performance logging of application data over the network. The cost of shipping an event over the wire compared with writing it to a local file is obviously higher, and so one can notice user-visible impact if the logging volume is high when using network-based logging (unless one wraps it in AsyncAppenders). Just curious to hear peoples experiences, strategies, and thoughts. Perhaps people can relate to the flow rate they've manage to achieve under different configurations. The driver to my question is that for my Apache Lab project (Pinpoint) I'm building a central logging repository server to allow data mining of the generated logging events. In a high performance site one can have many logging events shipped from multiple threads in, say, a web- app, being shipped serially over the wire, so slowing down the logging can slow down the response time. cheers, Paul Smith --------------------------------------------------------------------- To unsubscribe, e-mail: log4j-user-unsubscribe< at >logging.apache.org For additional commands, e-mail: log4j-user-help< at >logging.apache.org

CLOB datatype column usage with JDBCPlus jdbc appender

Vaibhav Kumar — 2008-09-04T09:56:58

Hi, I am using JDBCPlus (http://www.dankomannhaupt.de/projects/index.html) as a jdbc appender for logging into database. For a requirement, I had to change the datatype of a column (which stores the message string to be logged) from varchar2(4000) to CLOB. But as soon I do this, the logging in database stops and the stack trace got is : log4j:ERROR JDBCAppender::flush_buffer(), : java.sql.SQLException: Internal Error: Unable to construct a Datum from the specified input at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:168) at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:210) at oracle.jdbc.dbaccess.DBError.check_error(DBError.java:829) at oracle.sql.SQLUtil.makeDatum(SQLUtil.java:645) at oracle.sql.SQLUtil.makeOracleDatum(SQLUtil.java:946) at oracle.jdbc.driver.UpdatableResultSet.updateObject(UpdatableResultSet.java:1568) at oracle.jdbc.driver.OracleResultSet.updateObject(OracleResultSet.java:2787) at org.apache.log4j.jdbcplus.JDBCLogger.append(JDBCLogger.java:742) at org.apache.log4j.jdbcplus.JDBCAppender.flush_buffer(JDBCAppender.java:887) at org.apache.log4j.jdbcplus.JDBCAppender.append(JDBCAppender.java:867) at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:230) at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:65) at org.apache.log4j.Category.callAppenders(Category.java:203) at org.apache.log4j.Category.forcedLog(Category.java:388) at org.apache.log4j.Category.log(Category.java:853) at edu.unc.its.util.UNCLogger.prepareAndLogMessage(UNCLogger.java:445) at edu.unc.its.util.UNCLoggerClient.main(UNCLoggerClient.java:23) log4j:ERROR JDBCAppender::flush_buffer(), : java.sql.SQLException: Internal Error: Unable to construct a Datum from the specified input at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:168) at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:210) at oracle.jdbc.dbaccess.DBError.check_error(DBError.java:829) at oracle.sql.SQLUtil.makeDatum(SQLUtil.java:645) at oracle.sql.SQLUtil.makeOracleDatum(SQLUtil.java:946) at oracle.jdbc.driver.UpdatableResultSet.updateObject(UpdatableResultSet.java:1568) at oracle.jdbc.driver.OracleResultSet.updateObject(OracleResultSet.java:2787) at org.apache.log4j.jdbcplus.JDBCLogger.append(JDBCLogger.java:742) at org.apache.log4j.jdbcplus.JDBCAppender.flush_buffer(JDBCAppender.java:887) at org.apache.log4j.jdbcplus.JDBCAppender.append(JDBCAppender.java:867) at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:230) at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:65) at org.apache.log4j.Category.callAppenders(Category.java:203) at org.apache.log4j.Category.forcedLog(Category.java:388) at org.apache.log4j.Category.log(Category.java:853) at edu.unc.its.util.UNCLogger.prepareAndLogMessage(UNCLogger.java:445) at edu.unc.its.util.UNCLoggerClient.main(UNCLoggerClient.java:23) 14:39:08,463 FATAL UNC:? - [Message logged at =2008-08-07 10:30:00][Process Id =P1][Process Name =Process1][Transaction Id =T1][Source =Source1][Target =Target1][Service Invoked =LOGGING][Instance Id =I1][Message Id =M1] FATAL - This is the error message. It will be great if any suggestions can be provided to implement CLOB with JDBCPlus appender. Regards, Vaibhav

RE: network logging performance

Michael Erskine — 2008-09-04T09:14:13

For convenience I routinely configure SocketHubAppender instances for all my services and typically set a threshold of WARN to reduce traffic to important info. Where performance is deemed more important than the timely delivery of the information I have custom asynchronous appenders with limited BlockingDeque queues and daemon threads to pump out data at their leisure. Where I have multiple services logging across the network to a central place I tend to use a custom JDBC asynchronous appender. In other scenarios I have networked systems that require certain log messages as part of a larger communications protocol. Here I have custom appenders that catch LoggingEvent messages, translate them and drop them in the outgoing BlockingDeque for the target system (I always work with queues on complex systems to encourage loose coupling). I don't tend to pass Java objects over the wire: call me a traditionalist but when I write my protocols they tend to be human-readable 7-bit ASCII where you can telnet to a server, issue commands and get replies. This is great for unit testing, troubleshooting, etc. More recently I've been writing systems that interoperate with 3rd party systems where XML is passed over the wire (so performance is not of major importance :) ) and here I tend to make Apache XMLBeans and StAX parsers do all the work but internally I'll drop down to a human readable (and loggable!) protocol for testing and faster interfacing. I have a strong belief that any non-trivial networked communications system can be generalised to IRC! Just my 2 euros :) Regards, Michael Erskine.

network logging performance

Paul Smith — 2008-09-04T01:31:23

If anyone out there is using "Logging over the network" of any form (socket, JMS, Multicast, syslog appenders etc), this topic is for you. I'm wondering whether people could comment on their experience setting up high performance logging of application data over the network. The cost of shipping an event over the wire compared with writing it to a local file is obviously higher, and so one can notice user-visible impact if the logging volume is high when using network-based logging (unless one wraps it in AsyncAppenders). Just curious to hear peoples experiences, strategies, and thoughts. Perhaps people can relate to the flow rate they've manage to achieve under different configurations. The driver to my question is that for my Apache Lab project (Pinpoint) I'm building a central logging repository server to allow data mining of the generated logging events. In a high performance site one can have many logging events shipped from multiple threads in, say, a web- app, being shipped serially over the wire, so slowing down the logging can slow down the response time. cheers, Paul Smith

RE: New Appender: PatternFileAppender

Bruno Melloni — 2008-09-03T19:12:04

Seems that several of us have been looking at similar things. I looked at the MultiFile appender but it wasn't quite flexible enough, so I created another appender similar to this (DynamicTimestampedFileAppender) and was debating whether to submit it. It is complete but has only been in use for under a week. Please let me know if I should submit it, or perhaps the three appenders ought to be compared and eventually create a hybrid appender that replaces them all. Key features: - Designed as a full lifecycle file appender. - Instead of sequence numbers uses timestamps of file creation (much easier to find entries). Rollovers are really nothing more than closing a file and creating a new one. - Zips and deletes old files. Zips and deletes happen at night and server start (minimizes load when server is running). - Multiple logging file sets are dynamically generated. - Has built-in variables that can be used in defining the file names. - Additional variables can be defined by parsing the message through regular expressions. - Has a maximum file size, but it can be overridden with a minimum amount of time between rollovers (i.e.: 1 hour). - Extra rollovers occur at midnight and at a specified time (i.e.: 3:00am), allowing for clear separation of logs by 'operating window'. log4j.properties instructions and sample setup data: # basePath = The base folder for the server log (./log/server) # parseExpression = A RegularExpression to match. Each group will become a variable {%i}, # with i=1,2,... in the order shown in the regex, with its value parsed # from the message. Currently Station:ClientPC:TransactionID. Only # the station is being used, the other variables in the expression # help prevent mismatches. # filespecFormat = Format to use for the path and file name. # {%T} = Timestamp is required somewhere IN the file name. # {%d} = Current Day. Optional. May be in path, file name, or nowhere. # {%s} = Server Name. Optional. Where this appender is running. # May be in path, file name, or nowhere. # {%i} = Where i is a variable number from the parseExpression. Optional. # May be in path, file name, or nowhere. # defaultFilespec = Format to use for the path and file name when parsing fails to find a match. # For example, when there is no Station. # It MUST include {%T} IN the file name. It may not have any {%i} values. # {%d} and {%s} are allowed as in filespecFormat. # maxDesiredSize = Maximum size desired for individual log files. I.e.: 1GB, 2MB or 4KB. # It may be ignored based on minElapsedMinutes. # minElapsedMinutes = A minimum time that must pass before rolling, even if the maxDesiredSize has # been reached. This will prevent excessive rolling when volume/hr is high. # To ignore it, set it to a low value (i.e.: 1). The default limits rolls to # at most once an hour, or less, depending on the maxDesiredSize. # forceRollTime = A roll is forced on all open files at the time specified. # zipAfterDays = Zip all files that are older than this number of days. # disposeAfterDays = Delete all files that are older than this number of days. # # RELEVANT FEATURES: # - There will be a roll of all open files at 00:00:00 every day, and possibly one when a file # is being opened for the first time in the day if it had not previously rolled. # - zipping and deleting of old files happens during the daily roll. # - Logs are created as .txt files. After zipping they are converted to .zip. # LIMITATIONS (intentional): # Currently using lazy algorithms that have higher performance but may not trigger a roll until # later than normally expected: # - Elapsed time is being measured since start or last roll. # - Only open files are rolled. log4j.appender.LOGFILE=com...logging.DynamicTimestampedFileAppender log4j.appender.LOGFILE.layout=com...logging.PCIPatternLayout log4j.appender.LOGFILE.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} [%t] [%c] %-5p - %m%n log4j.appender.LOGFILE.basePath=${serverLog} log4j.appender.LOGFILE.parseExpression=^(\\w{3}):([\\w\\-]{1,}):([\\d_]{ 1,}) log4j.appender.LOGFILE.filespecFormat={%d}/server_{%1}_{%T}.{%s} log4j.appender.LOGFILE.defaultFilespec={%d}/default.{%T}.{%s} log4j.appender.LOGFILE.maxDesiredSize=2MB log4j.appender.LOGFILE.minElapsedMinutes=60 log4j.appender.LOGFILE.forceRollTime=03:00 log4j.appender.LOGFILE.zipAfterDays=1 log4j.appender.LOGFILE.disposeAfterDays=6 Note: My appender does not limit the number of concurrent open files. May I ask what is the limit in Unix/Linux if anyone knows? bruno -----Original Message----- From: David Britton [mailto:dpb< at >hp.com] Sent: Tuesday, September 02, 2008 4:33 PM To: Log4J Users List Subject: Re: New Appender: PatternFileAppender On Tue, Sep 02, 2008 at 09:04:30PM +0000, Curt Arnold wrote: Hi Curt -- Glad I asked! It looks to be nearly identical in base functionality, but of course has a much more developed interface, and has many more options to it. It looks like it has two main "policies", a Pattern policy and a Size policy. The "Size" policy looks like it provides duplicate functionality that the RollingFileAppender provides. It increments an index appended to the file after they get to a certain size. I may be over simplifying here -- but I think that RollingFileAppender provides this and more. The "Pattern" policy is what my appender was supposed to solve, and I think is what is "missing" from log4j right now. Just from inspection, it appears as though it would be slightly more complex than my PatternFileAppender, but has more configuration options, and of course is 1.2 compatible. :) Is there a reason this has not made it into the official distribution? -- David Britton hp.com> --------------------------------------------------------------------- To unsubscribe, e-mail: log4j-user-unsubscribe< at >logging.apache.org For additional commands, e-mail: log4j-user-help< at >logging.apache.org

Re: Future of JULI<->Log4j bridging

Paul Smith — 2008-09-03T04:57:55

wow! cool! this is great news to hear. Now we have 2 users! :) If you're interested in trialing things one day, look out for Pinpoint (currently an Apache Lab). It's for log data mining. centralised log repository with full text indexing and reporting. I'm right in the middle of a load test for it as we speak (soaking events from 2 hosts, at 100 events/second, this is a Mercury load test of our actual application running realistic production loads). Unfortunately there's no real UI for actually doing the searching as yet, but the indexing side is working very well. cheers, Paul On 03/09/2008, at 2:20 PM, Brett Randall wrote: Paul Smith Core Engineering Manager Aconex The easy way to save time and money on your project 696 Bourke Street, Melbourne, VIC 3000, Australia Tel: +61 3 9240 0200 Fax: +61 3 9240 0299 Email: psmith< at >aconex.com www.aconex.com This email and any attachments are intended solely for the addressee. The contents may be privileged, confidential and/or subject to copyright or other applicable law. No confidentiality or privilege is lost by an erroneous transmission. If you have received this e-mail in error, please let us know by reply e-mail and delete or destroy this mail and all copies. If you are not the intended recipient of this message you must not disseminate, copy or take any action in reliance on it. The sender takes no responsibility for the effect of this message upon the recipient's computer system.

Re: Future of JULI<->Log4j bridging

Brett Randall — 2008-09-03T04:20:44

Thanks - all good and working for now. Bridge version apache-jul-log4j-bridge-1.0.0-20071030.02281 appears to work well. Using Spring Framework, I can actually bootstrap the bridge during the Spring context load using this: org.apache.logging.julbridge.JULLog4jBridge.assimilate ... which I think is nice. No reference or dependency on the jul-log4j-bridge in my code, deploy-time replacement of JUL backend with log4j. Then I can even plug-in Chainsaw: (log4j.xml)

Very happy - thank you for contributing this code. Cheers Brett

Re: Future of JULI<->Log4j bridging

Paul Smith — 2008-09-03T03:58:17

Brett, if you get stuck, I can package something up here locally and email it to you privately if you like (or maybe I should republish to my ~psmith repo.. ) cheers, Paul On 03/09/2008, at 1:54 PM, Brett Randall wrote: Paul Smith Core Engineering Manager Aconex The easy way to save time and money on your project 696 Bourke Street, Melbourne, VIC 3000, Australia Tel: +61 3 9240 0200 Fax: +61 3 9240 0299 Email: psmith< at >aconex.com www.aconex.com This email and any attachments are intended solely for the addressee. The contents may be privileged, confidential and/or subject to copyright or other applicable law. No confidentiality or privilege is lost by an erroneous transmission. If you have received this e-mail in error, please let us know by reply e-mail and delete or destroy this mail and all copies. If you are not the intended recipient of this message you must not disseminate, copy or take any action in reliance on it. The sender takes no responsibility for the effect of this message upon the recipient's computer system.

Re: Future of JULI<->Log4j bridging

Brett Randall — 2008-09-03T03:54:19

Whoops - I lie. I got an older snapshot from that location, and not one later than the same problem described here https://issues.apache.org/bugzilla/show_bug.cgi?id=43567 . Let me go and test a later one. Cheers Brett On Wed, Sep 3, 2008 at 1:51 PM, Brett Randall gmail.com> wrote:

Re: Future of JULI<->Log4j bridging

Brett Randall — 2008-09-03T03:51:02

Hi Paul, Thanks for your fast and informative response. AFAICT there isn't an equivalent to this around, and I suspect that people would find it useful, given peoples intellectual investment and liking of log4j and in recognition of the number of libraries using JUL. I grabbed the source and paused at the point where I needed to build log4j component 1.0 to satisfy that Maven dependency - don't have the cycles to do this today. I grabbed the latest 1.0.0 SNAPSHOT from here http://people.apache.org/~psmith/logging.apache.org/repo/org/apache/logging/apache-jul-log4j-bridge/1.0.0-SNAPSHOT/. This looks like it will work, but perhaps a log4j signature has changed - I wanted to use latest 1.2.15 but I got: java.lang.NoSuchMethodError: org.apache.log4j.spi.LoggingEvent.(Ljava/lang/String;Lorg/apache/log4j/Logger;JLorg/apache/log4j/Level;Ljava/lang/Object;Ljava/lang/String;Lorg/apache/log4j/spi/ThrowableInformation;Ljava/lang/String;Lorg/apache/log4j/spi/LocationInfo;Ljava/util/Map;)V at org.apache.logging.julbridge.JULLog4jEventConverter.convert(JULLog4jEventConverter.java:130) at org.apache.logging.julbridge.JULBridgeHandler.publish(JULBridgeHandler.java:49) at java.util.logging.Logger.log(Logger.java:472) at java.util.logging.Logger.doLog(Logger.java:494) at java.util.logging.Logger.log(Logger.java:517) at java.util.logging.Logger.severe(Logger.java:1004) ... ... and I'm wondering if this is Category versus Logger ... It would be interesting to hear if others would like to, as you suggest, breathe some life back into this towards a release. Do you know whether the log4j component framework which it is dependent on was ever released, as this would seem to be a blocker. Cheers Brett On Wed, Sep 3, 2008 at 12:28 PM, Paul Smith aconex.com> wrote:

Re: Future of JULI<->Log4j bridging

Paul Smith — 2008-09-03T02:28:35

On 03/09/2008, at 12:21 PM, Brett Randall wrote: the jul-log4j-bridge has not been released yet because most of the effort has been towards the log4j 1.2.16 and other sub-modules (receivers, zeroconf). As the sole developer of the bridge, I think it's feature complete, and there was at least one user I know of who's used it. Whether they're still using it, or if there are others using it I can't say as I haven't had any reports. Obviously I'd love to have feedback on it out in the wild as this would give us impetus to go through a formal release. Bit of a chicken/egg problem there though. If you're up for giving it a shot, let me know and I'll support you through the way (it would be a good test of the documentation... (you'd need to generate that via 'mvn site' locally though, since it's not released, there's no public website for it as yet - if that's a burden, I can generate a site here myself and zip it up for you if you like). cheers, Paul Smith

Future of JULI<->Log4j bridging

Brett Randall — 2008-09-03T02:21:03

Hi all, I'm wondering what folks are using when they find they want to bridge an application built with JULI logging to use a log4j logging backend. I'm aware of http://people.apache.org/~psmith/logging.apache.org/sandbox/jul-log4j-bridge/, but it's in the sandbox and there's not a lot of detail on status/roadmap. Has anyone used this, or are there accepted alternatives? Thanks Brett

Re: DEBUG messages appearing on console?

2008-09-02T21:57:57

Thanks Curt. I implemented your suggestion and got what I was looking for. Thanks David for your suggestion as well. Patrick Patrick ----- Original Message ----- From: Curt Arnold [carnold< at >apache.org] Sent: 09/02/2008 04:47 PM To: "Log4J Users List" logging.apache.org> Subject: Re: DEBUG messages appearing on console? On Sep 2, 2008, at 2:00 PM, Patrick.Grimard< at >xtl.com wrote: The filter suggestion is a bit overkill for what you need. What is happening is that you are assigning a DEBUG threshold to all logging requests to loggers starting with "com.dsn.loadlink", all other loggers have a threshold of WARN. You are also attaching a ConsoleAppender to the root logger and a RollingFileAppender to "com.dsn.loadlink", but that is an independent action from setting the thresholds. So if you make a logging request on "com.dsn.loadlink" or a child logger, it is evaluated against the nearest specified threshold and if that is satisfied, then it is processed by all appenders attached in the hierarchy (unless additivity is set to false which will prevent appenders attached to parent loggers from seeing the event). You can set a threshold on the appender which is independent of the threshold on the logger. Unless you have a specific need to separate the appenders, I would attach them both to root, but set a threshold on the ConsoleAppender. log4j.appender.threshold=INFO --------------------------------------------------------------------- To unsubscribe, e-mail: log4j-user-unsubscribe< at >logging.apache.org For additional commands, e-mail: log4j-user-help< at >logging.apache.org

Re: New Appender: PatternFileAppender

David Britton — 2008-09-02T21:32:59

Hi Curt -- Glad I asked! It looks to be nearly identical in base functionality, but of course has a much more developed interface, and has many more options to it. It looks like it has two main "policies", a Pattern policy and a Size policy. The "Size" policy looks like it provides duplicate functionality that the RollingFileAppender provides. It increments an index appended to the file after they get to a certain size. I may be over simplifying here -- but I think that RollingFileAppender provides this and more. The "Pattern" policy is what my appender was supposed to solve, and I think is what is "missing" from log4j right now. Just from inspection, it appears as though it would be slightly more complex than my PatternFileAppender, but has more configuration options, and of course is 1.2 compatible. :) Is there a reason this has not made it into the official distribution?

Re: New Appender: PatternFileAppender

Curt Arnold — 2008-09-02T21:04:30

On Sep 2, 2008, at 3:38 PM, David Britton wrote: Could you compare it with the MultiFileAppender that has been languishing in the sandbox for a while? Search the archives for MultiFileAppender. Source code at http://svn.apache.org/repos/asf/logging/sandbox/log4j/multifile .

Re: DEBUG messages appearing on console?

Curt Arnold — 2008-09-02T20:47:20

On Sep 2, 2008, at 2:00 PM, Patrick.Grimard< at >xtl.com wrote: The filter suggestion is a bit overkill for what you need. What is happening is that you are assigning a DEBUG threshold to all logging requests to loggers starting with "com.dsn.loadlink", all other loggers have a threshold of WARN. You are also attaching a ConsoleAppender to the root logger and a RollingFileAppender to "com.dsn.loadlink", but that is an independent action from setting the thresholds. So if you make a logging request on "com.dsn.loadlink" or a child logger, it is evaluated against the nearest specified threshold and if that is satisfied, then it is processed by all appenders attached in the hierarchy (unless additivity is set to false which will prevent appenders attached to parent loggers from seeing the event). You can set a threshold on the appender which is independent of the threshold on the logger. Unless you have a specific need to separate the appenders, I would attach them both to root, but set a threshold on the ConsoleAppender. log4j.appender.threshold=INFO

New Appender: PatternFileAppender

David Britton — 2008-09-02T20:38:26

Hello -- I'm not sure if this is the correct way to do this, but my colleague and I have developed a new appender called a PatternFileAppender. I did not see anything else like this in the log4j distribution, but it helped us solve our particular problem, and I think it's a good general purpose appender. If someone knows of another appender like this, I would love to hear about it. I went to make an official patch for log4j 1.2, but I found that the code needed to be compatible with 1.2 source compatibility level. Since the code uses generics in it's use of the LinkedHashMap data structure, it needs 1.5 or later to compile correctly. Anyway, I've attached the file, if there is an "official" way to do this, please point me in the right direction. Here is the high-level details of the appender: * Can process a filename as a pattern, with all substitutions available. (e.g., MDC, date, variable, level, class, ...) * Interesting use cases we are using it for: - Separate log file per "thread" using MDC (note that "thread" could be any separation desired, log level, date, time, etc). * Keeps a cache of the latest 20 open file streams, so as not to overload the OS/filesytem/etc * Adds performance penalty in that each log message will generate an extra pattern lookup. Note that this already happens for the message itself, so this does not seem like a huge impact. Let me know any feedback... Thanks!

Re: DEBUG messages appearing on console?

David Britton — 2008-09-02T19:31:47

Please see the following answer I recently gave: http://mail-archives.apache.org/mod_mbox/logging-log4j-user/200808.mbox/%3C20080825151619.GA13754< at >dune.americas.cpqcorp.net%3E

DEBUG messages appearing on console?

2008-09-02T19:00:22

I'm trying to configure log4j so that DEBUG messages or lower priority only appear in a log file and not the console, and anything above DEBUG appears on the console. This is what my log4j.properties file looks like right now, however the DEBUG messages are still appearing on the console. I'm using log4j along with Commons Logging API if that makes any difference. # Set the root logger to the INFO level and CA to be it's appender log4j.rootLogger=WARN, CA log4j.appender.CA=org.apache.log4j.ConsoleAppender log4j.appender.CA.layout=org.apache.log4j.PatternLayout log4j.appender.CA.layout.ConversionPattern=%-5p %d{yyyy-MM-dd HH:mm:ss} [%t] %c%n\t{%m}%n%n # LinkHarvester log4j.logger.com.dsn.loadlink=DEBUG, harvester log4j.appender.harvester=org.apache.log4j.RollingFileAppender log4j.appender.harvester.File=C:/log4j/logs/LinkHarvester.log log4j.appender.harvester.MaxFileSize=100KB log4j.appender.harvester.MaxBackupIndex=1 log4j.appender.harvester.layout=org.apache.log4j.PatternLayout log4j.appender.harvester.layout.ConversionPattern=%-5p %d{yyyy-MM-dd HH:mm:ss} [%t] %c%n\t{%m}%n%n (Embedded image moved to file: pic18467.jpg) --------------------------------------------------------------------- To unsubscribe, e-mail: log4j-user-unsubscribe< at >logging.apache.org For additional commands, e-mail: log4j-user-help< at >logging.apache.org

Re: HOw to replace the system.out.println

Curt Arnold — 2008-09-02T17:42:31

On Sep 2, 2008, at 12:36 PM, Thorbjørn Ravn Andersen wrote: I believe you could still extract it if you wanted it, but it keeps it all in one place and you can decide if you want to go to that expense. You could also check the threshold of the logger hierarchy before attempting to determine the location.