MSMQ Distributor Service (EntLib 5) won't start ... sometimes

Topics: Logging Application Block
Jul 25, 2013 at 5:49 PM
We are having weird and inconsistent behaviours trying to run this service on various servers.

At times I was able to run it (as Network Service user) and it was fine. Other times it would only run under a particular users credentials.

Even weirder, if the server was rebooted, or the service stopped manually, it wouldn't run again as the previously designated user, you had to change users and then it would run. If you stopped it and ran it again as the previous user, it would work again.

This is getting pretty frustrating, as there seems to be no rhyme or reason as to why it will run as some and not others, and only sometimes at that!

Does anybody have any ideas?
Jul 26, 2013 at 7:10 AM
The behavior seems very odd. Do you get any exceptions? When you say "it wouldn't run again" what does that mean? That it started but nothing was processed? Is the queue a local queue or remote (to the distributor service)?

~~
Randy Levy
entlib.support@live.com
Enterprise Library support engineer
Support How-to
Jul 26, 2013 at 12:47 PM
Hi Randy. Sorry for not putting more useful information in the post!

I've got to try to reproduce it (shouldn't be hard) and I'll send along anything I'm seeing in eventviewer. Anything else I should look for?

What happens is that it eventually times out when trying to restart the service, so it doesn't end up running at all.

The queue is a private queue on the local machine, nothing remote. I have permissions set correctly and all that as when it does get up and running it processes messages off the queue correctly.

Thanks,

Chris
Jul 26, 2013 at 2:24 PM
An intermittent timeout on startup sounds very odd. Startup doesn't do that much real work -- mostly validation, log some messages and start a timer. If an exception is thrown then that should be logged and the service should fail to start. Definitely check the event log and see what is there.

~~
Randy Levy
entlib.support@live.com
Enterprise Library support engineer
Support How-to
Sep 17, 2013 at 3:25 PM
Hi Randy. It took me a while to get back to this.

I'm still having trouble getting this service started, even though I can get exactly the same service to run on other servers. Right now the problem doesn't seem so intermittent, I just can't get it running on one server.

I checked the event log, and there is very little to go on there. When the service fails to start I get two logs under Event Viewer -> Windows Logs -> System:

First I get this:
  • <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  • <System>
    <Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" EventSourceName="Service Control Manager" />
    <EventID Qualifiers="49152">7009</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8080000000000000</Keywords>
    <TimeCreated SystemTime="2013-09-17T14:15:23.016029100Z" />
    <EventRecordID>33139</EventRecordID>
    <Correlation />
    <Execution ProcessID="516" ThreadID="3724" />
    <Channel>System</Channel>
    <Computer> ... snip ...</Computer>
    <Security />
    </System>
  • <EventData>
    <Data Name="param1">30000</Data>
    <Data Name="param2">Enterprise Library Distributor Service (PALS)</Data>
    </EventData>
    </Event>
And then I get this:
  • <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  • <System>
    <Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" EventSourceName="Service Control Manager" />
    <EventID Qualifiers="49152">7000</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8080000000000000</Keywords>
    <TimeCreated SystemTime="2013-09-17T14:15:23.016029100Z" />
    <EventRecordID>33140</EventRecordID>
    <Correlation />
    <Execution ProcessID="516" ThreadID="3724" />
    <Channel>System</Channel>
    <Computer> ... snip ... </Computer>
    <Security />
    </System>
  • <EventData>
    <Data Name="param1">Enterprise Library Distributor Service (PALS)</Data>
    <Data Name="param2">%%1053</Data>
    </EventData>
    </Event>
Any other ideas as to what I can try? Is there any data in either of these logs that tells you anything?

Thanks,

Chris
Sep 18, 2013 at 1:18 AM
Unfortunately, those messages just look like a service startup timeout which doesn't really help us much.

I wonder if there is an issue writing errors to the Event Log for the Distributor Service. Can you check that the "Enterprise Library Logging Distributor Service" Event Source exists (HKLM\SYSTEM\CurrentControlSet\services\eventlog\Application\Enterprise Library Logging Distributor Service)? If not, then create it using a program or script (http://stackoverflow.com/questions/446691/how-to-create-windows-eventlog-source-from-command-line). Then try and start the service and check the Event Log if it fails to start.

~~
Randy Levy
entlib.support@live.com
Enterprise Library support engineer
Support How-to
Sep 18, 2013 at 12:58 PM
Hi Randy. I checked the registry and I found an entry for the service in the place you specified with the following data:

(Default) REG_SZ (value not set)
EventMessageFile REG_EXPAND_SZ C:\Windows\Microsoft.NET\Framework64\v4.0.30319\EventLogMessages.dll

Does this look right to you?

Any other ideas?

Chris
Sep 18, 2013 at 3:55 PM
Yes, that looks right.

Try adding the following to the config file and see if that makes a difference.
<configuration>
    <runtime>
        <generatePublisherEvidence enabled="false"/>
    </runtime>
</configuration>
~~
Randy Levy
entlib.support@live.com
Enterprise Library support engineer
Support How-to
Marked as answer by randylevy on 10/7/2013 at 10:59 PM
Sep 23, 2013 at 2:58 PM
Randy, you are the man! after adding that it starts up no problem!

Can you tell me what this setting does? I read a little bit about what it does, but I just want to make sure I understand the ramifications of turning it off in a production environment.

Thanks again!
Sep 23, 2013 at 3:46 PM
Edited Sep 23, 2013 at 3:47 PM
Great to hear. Setting generatePublisherEvidence to false tells the runtime to not create Publisher evidence for code access security (CAS). My understanding is that this involves Authenticode but in this case we don't need that so we can turn it off. It's not 100% clear to me what the runtime is doing under the covers since I don't believe Authenticode would be in play here (perhaps going out to a obtain a CRL even if not needed) but the recommendation is to turn it off:

We recommend that services use the <generatePublisherEvidence> element to improve startup performance. Using this element can also help avoid delays that can cause a time-out and the cancellation of the service startup.

This is the best article I've seen about this issue: Best Practice - <GeneratePublisherEvidence> in ASPNET.CONFIG.

~~
Randy Levy
entlib.support@live.com
Enterprise Library support engineer
Support How-to