How to log large data?

Topics: Logging Application Block
Jan 17, 2013 at 4:56 PM
Edited Jan 17, 2013 at 4:56 PM

Im having issues while logging large logs. I have the following WCF service operation. I have noticed that if the file size is large the Enterprise library does not log anything. Is there any settings i have to do to log large data?

 

public Upload(UploadData data)
{
 // Serialize and log data here
}

public class UploadData
{

  public string SomeMetaData{get;set;}

  public byte[] File {get;set;}

}
Editor
Jan 18, 2013 at 3:49 AM

What trace listener are you using as the destination?  Is it the event log?  The event log definitely has a maximum message size of around 32K (See http://msdn.microsoft.com/en-us/library/6w20x90k.aspx for the details) so is not a good choice for logging large messages.  

Enterprise Library itself does not enforce any size limitations but the eventual message destinations may.

Probably the best approach for large messages would be to log to disk (e.g. Flat File Trace Listener or Rolling Flat File Trace Listener).

Also, I always recommend setting the errors special source to write to "simple" trace listener such as the Flat File Trace Listener that points to a location where permissions will not be an issue.  This way, if an error occurs logging to the event log due to a large message size the error will be reported in the log file.  The benefit is two fold: you become aware of a potential issue you are encountering and you don't lose the original message.

If this doesn't seem to fit with what you are seeing, I would recommend posting more information about the specifics.

--
Randy Levy
entlib.support@live.com
Enterprise Library support engineer
Support How-to 

Jan 18, 2013 at 8:29 PM

Thanks randy, thats extactly what i was looking for. Im logging to event viewer. There is no issue while logging errors/exception in the event viewer.The Default LogLevel is set to Error.

I was  logging serialized object ( ie. large message) using LogLevel = Info, so that if there is an issue we can just turn the verbose logging on and see what request we are getting. 

1>What listeners are available out of the box to log such large message?

2>If i configured one more listener, Will Logger.Write(LogEntry) will try to log using both the listeners?  

3>Is there anyway to configured the second listeners to use only to log Info messages? ( This is the listener i'll use to log large  message)

Editor
Jan 21, 2013 at 4:29 AM

1>What listeners are available out of the box to log such large message?

The file based trace listeners: FlatFileTraceListener, RollingFlatFileTraceListener, XmlTraceListener as well as built-in System.Diagnostic TraceListeners.

2>If i configured one more listener, Will Logger.Write(LogEntry) will try to log using both the listeners? 

Yes.  I'm assuming you mean adding an additional listener to a category.

3>Is there anyway to configured the second listeners to use only to log Info messages? ( This is the listener i'll use to log large  message)

You can configure 2 trace listeners and set different filter levels for each listener.  For example:

 

<configuration>
    <configSections>
        <section name="loggingConfiguration" type="Microsoft.Practices.EnterpriseLibrary.Logging.Configuration.LoggingSettings, Microsoft.Practices.EnterpriseLibrary.Logging, Version=5.0.505.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" requirePermission="true" />
    </configSections>
    <loggingConfiguration name="" tracingEnabled="true" defaultCategory="General">
        <listeners>
            <add name="Event Log Listener" type="Microsoft.Practices.EnterpriseLibrary.Logging.TraceListeners.FormattedEventLogTraceListener, Microsoft.Practices.EnterpriseLibrary.Logging, Version=5.0.505.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
                listenerDataType="Microsoft.Practices.EnterpriseLibrary.Logging.Configuration.FormattedEventLogTraceListenerData, Microsoft.Practices.EnterpriseLibrary.Logging, Version=5.0.505.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
                source="Enterprise Library Logging" formatter="Text Formatter"
                log="" machineName="." traceOutputOptions="None" filter="Warning" />
            <add name="Info Flat File Trace Listener" type="Microsoft.Practices.EnterpriseLibrary.Logging.TraceListeners.FlatFileTraceListener, Microsoft.Practices.EnterpriseLibrary.Logging, Version=5.0.505.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
                listenerDataType="Microsoft.Practices.EnterpriseLibrary.Logging.Configuration.FlatFileTraceListenerData, Microsoft.Practices.EnterpriseLibrary.Logging, Version=5.0.505.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
                fileName="info.log" filter="Information" />
            <add name="Error Flat File Trace Listener" type="Microsoft.Practices.EnterpriseLibrary.Logging.TraceListeners.FlatFileTraceListener, Microsoft.Practices.EnterpriseLibrary.Logging, Version=5.0.505.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
                listenerDataType="Microsoft.Practices.EnterpriseLibrary.Logging.Configuration.FlatFileTraceListenerData, Microsoft.Practices.EnterpriseLibrary.Logging, Version=5.0.505.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
                fileName="error.log" />
        </listeners>
        <formatters>
            <add type="Microsoft.Practices.EnterpriseLibrary.Logging.Formatters.TextFormatter, Microsoft.Practices.EnterpriseLibrary.Logging, Version=5.0.505.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
                template="Timestamp: {timestamp}{newline}
Message: {message}{newline}
Category: {category}{newline}
Priority: {priority}{newline}
EventId: {eventid}{newline}
Severity: {severity}{newline}
Title:{title}{newline}
Machine: {localMachine}{newline}
App Domain: {localAppDomain}{newline}
ProcessId: {localProcessId}{newline}
Process Name: {localProcessName}{newline}
Thread Name: {threadName}{newline}
Win32 ThreadId:{win32ThreadId}{newline}
Extended Properties: {dictionary({key} - {value}{newline})}"
                name="Text Formatter" />
        </formatters>
        <categorySources>
            <add switchValue="All" name="General">
                <listeners>
                    <add name="Event Log Listener" />
                    <add name="Info Flat File Trace Listener" />
                </listeners>
            </add>
        </categorySources>
        <specialSources>
            <allEvents switchValue="All" name="All Events" />
            <notProcessed switchValue="All" name="Unprocessed Category" />
            <errors switchValue="All" name="Logging Errors & Warnings">
                <listeners>
                    <add name="Error Flat File Trace Listener" />
                </listeners>
            </errors>
        </specialSources>
    </loggingConfiguration>
</configuration>

 

However, if you specify to log Information then all messages with severity of Information and above will be logged.  So it's not strictly logging only Information messages.  In that situation, it would be best to create a new category to log large messages.

--
Randy Levy
entlib.support@live.com
Enterprise Library support engineer
Support How-to 

Jan 22, 2013 at 8:00 PM

 

However, if you specify to log Information then all messages with severity of Information and above will be logged.  So it's not strictly logging only Information messages.  In that situation, it would be best to create a new category to log large messages.

Thats okay

The existing listener is logging errors to event viewer. The second listener will log to flat file ( errors & info) that's what i needed

Jan 22, 2013 at 8:52 PM
Edited Jan 22, 2013 at 9:07 PM

also whats is  <specialSources>, is that only used if there is an internal error while logging?

If that is true and if i only need to log my application errors into flat file. I need to add new flat file listener in <listeners> section and inside <categorySources>. I do not have to configure it inside <specialSources> section. Is that correct?

Editor
Jan 23, 2013 at 3:52 AM

specialSources contains 3 trace sources:

  • allEvents -- this will receive all messages irrespective of whether they are processed by other sources
  • notProcessed -- all categories that are not processed will be sent to this trace source
  • errors -- all internal logging block errors will be sent to this source.  

I recommend always setting the errors specialSource in order to capture logging errors (e.g. mis-configuration or transient error information) as well as the content of the original LogEntry that failed.

--
Randy Levy
entlib.support@live.com
Enterprise Library support engineer
Support How-to 

Jan 23, 2013 at 4:30 PM

<categorySources>
      <add switchValue="All" name="General">
        <listeners>
          <add name="Event Log Listener" />          
        </listeners>
      </add>
      <add switchValue="All" name="Audit">
        <listeners>          
          <add name="Flat File Trace Listener"/>
        </listeners>
      </add>
    </categorySources>

 

Is that a valid configuration for category source?

Editor
Jan 24, 2013 at 5:29 AM

It looks OK.  I would recommend using the Enterprise Library Configuration Tool to validate existing configurations as well as to create new XML configuration.

--
Randy Levy
entlib.support@live.com
Enterprise Library support engineer
Support How-to 

Apr 3, 2013 at 8:47 PM
im still having the issue while logging large messages. Can someone point me to correct library source code for the version mentioned below? I want to debug the code to check what im missing.

Microsoft.Practices.EnterpriseLibrary.Logging.Configuration.LoggingSettings, Microsoft.Practices.EnterpriseLibrary.Logging, Version=5.0.505.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"

I have downloaded source code from http://msdn.microsoft.com/en-us/library/ff648951.aspx (Enterprise Library 5.0 - April 2010) but that 5.0.414.0 not 5.0.505.0
Editor
Apr 3, 2013 at 8:53 PM
You can get the source and bits for version 5.0.505.0 at Microsoft Enterprise Library 5.0 Optional Update 1

~~
Randy Levy
entlib.support@live.com
Enterprise Library support engineer
Support How-to
Apr 3, 2013 at 10:52 PM
well i downloaded that MSI but then couldn't find Microsoft.Practices.EnterpriseLibrary.Logging.Service.dll . Is it suppose to be part of Optional Update 1 MSI?
Editor
Apr 4, 2013 at 4:28 AM
Edited May 1, 2013 at 9:31 PM
lax4u wrote:
well i downloaded that MSI but then couldn't find Microsoft.Practices.EnterpriseLibrary.Logging.Service.dll . Is it suppose to be part of Optional Update 1 MSI?
No, it's not.

It sounds like you are using the Silverlight Integration Pack. If so, then you can get the source code at Silverlight Integration Pack for Microsoft Enterprise Library 5.0.

If you are using the remote logging service then you could be hitting the WCF limit on max message size. Have you configured the service as per Implementing Diagnostic Logging and Tracing in Your Application?

WCF has a lot of knobs to play with. You could try something like this for the web.config
    <bindings>
      <customBinding>
        <binding name="Microsoft.Practices.EnterpriseLibrary.Logging.Service.customBinding">
          <binaryMessageEncoding maxReadPoolSize="2147483647" maxSessionSize="2147483647" maxWritePoolSize="2147483647">
            <readerQuotas maxDepth="32" maxStringContentLength="5242880"
            maxArrayLength="200000" maxBytesPerRead="4096" maxNameTableCharCount="16384" />
          </binaryMessageEncoding>
          <httpTransport maxBufferPoolSize="2147483647" maxBufferSize="2147483647" maxReceivedMessageSize="2147483647"/>
        </binding>
      </customBinding>
    </bindings>
Also you may have to increase the httpRuntime maxRequestLength:
  <system.web>
    <httpRuntime maxRequestLength="2097151" />
    <compilation debug="true" targetFramework="4.0" />
  </system.web>
For troubleshooting WCF you can try Configuring Tracing as well as fiddler.

~~
Randy Levy
entlib.support@live.com
Enterprise Library support engineer
Support How-to
Apr 4, 2013 at 9:25 PM
Finally I was able to debug enterprise library and found the root cause.
My configuration is setup as below
<categorySources>
      <add switchValue="All" name="General">
        <listeners>
          <add name="Event Log Listener" />
          <add name="Flat File Trace Listener" />
        </listeners>
      </add>
    </categorySources>
    <specialSources>
      <allEvents switchValue="All" name="All Events" />
      <notProcessed switchValue="All" name="Unprocessed Category" />
      <errors switchValue="All" name="Logging Errors &amp; Warnings">
        <listeners>
          <add name="Event Log Listener" />
        </listeners>
      </errors>
    </specialSources>
</loggingConfiguration>     
The message I’m trying to log is large, so as expected the message will not get logged into EventViewer and that’s why I have configured “Flat File Trace Listener” to log the large messages. But EnterpriseLibrary does not log into “Flat File” either. After debugging the code here is what I found:
First, It tries to log into Event viewer, which fails with the error {"Log entry string is too long. A string written to the event log cannot exceed 32766 characters."}
Then it tries to log this new exception into event viewer, with the message “Message: Tracing to LogSource 'General' failed. Processing for other sources will continue. See summary information below for more information. Should this problem persist, stop the service and check the configuration file(s) for possible error(s) in the configuration of the categories and sinks.”
but the summary information also include original message and since the original message is large anyway so it fails the second time too.
After this, it does not try to log into "Flat File Trace Listener"

I don’t know if this is a bug or expected behavior, but I solved my issue just reversing the order of Listeners or by simply using “Flat File Trace Lister” alone
Thank you all for your help