reactiveRules won't find it's target

Topics: Windows Azure Integration Pack
May 7, 2012 at 12:10 PM
Edited May 7, 2012 at 12:12 PM

Hi,

I'm trying to incorporate the Autoscaling block of P&P in Azure. All seems to work except the reactiveRules (the constraintRules do work).

I've created the performabce counters, diagnostics and all - and still get that the block doesn't see the target of the reactive rules. I get this result (in diagnostics):

 

<TraceSource>Autoscaling General</TraceSource><Object>Rule match.[BEGIN DATA]{"EvaluationId":"794cac89-8cb9-4aed-9e6a-0388fada0360","MatchingRules":[{"RuleName":"Default constraints for all roles","RuleDescription":"SLA rule","Targets":["JobsRole","RESTAPI","Web"]},{"RuleName":"scaleWebApi","RuleDescription":"Scale up when cpu average over 5%","Targets":[]},{"RuleName":"scaledownWebApi","RuleDescription":"Scale down when cpu avg under 50%","Targets":[]}]}</Object>

 

Where as my config files are:

 

<?xml version="1.0" encoding="utf-8"?><rules xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://schemas.microsoft.com/practices/2011/entlib/autoscaling/rules">  <constraintRules>    <rule name="Default constraints for all roles" description="SLA rule" enabled="true" rank="0">      <actions>        <range target="JobsRole" min="1" max="5" />        <range target="RESTAPI" min="1" max="5" /> <range target="Web" min="1" max="5" />      </actions>    </rule>  </constraintRules>  <reactiveRules> <rule name="scaleWebApi" rank="2" enabled="true" description="Scale up when cpu average over 5%" > <when> <any> <greater operand="RESTAPI_CPU_Avg" than="50" /> </any> </when> <actions> <scale target="RESTAPI" by="1" /> </actions> </rule> <rule name="scaledownWebApi" rank="2" enabled="true" description="Scale down when cpu avg under 50%"> <when> <all> <lessOrEqual operand="RESTAPI_CPU_Avg" than="50" /> </all> </when> <actions> <scale target="RESTAPI" by="-1" /> </actions> </rule>  </reactiveRules>  <operands>    <performanceCounter alias="RESTAPI_CPU_Avg" timespan="00:10:00" aggregate="Average" source="RESTAPI" performanceCounterName="\Processor(_Total)\% Processor Time" />  </operands></rules>

 

 

and the service config

 

<?xml version="1.0" encoding="utf-8"?><serviceModel xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://schemas.microsoft.com/practices/2011/entlib/autoscaling/serviceModel">  <subscriptions>    <subscription name="<subs name>" subscriptionId="<subid>" certificateThumbprint="<thumb>" certificateStoreName="My" certificateStoreLocation="LocalMachine">      <services>        <service dnsPrefix="atkapi" slot="Production" scalingMode="Scale">          <roles>            <role alias="JobsRole" roleName="JobsRole" wadStorageAccountName="store" />            <role alias="RESTAPI" roleName="RESTAPI" wadStorageAccountName="store" />          </roles>        </service>        <service dnsPrefix="atk-web" slot="Production" scalingMode="Scale">          <roles> <role alias="Web" roleName="Web" wadStorageAccountName="store" />          </roles>        </service>      </services>      <storageAccounts>        <storageAccount alias="store" connectionString="<connectionstring>">          <queues>            <queue alias="JobsQueue" queueName="userlocations" />          </queues>        </storageAccount>      </storageAccounts>    </subscription>  </subscriptions>  <scaleGroups />  <stabilizer scaleUpCooldown="00:10:00" scaleDownCooldown="00:10:00" notificationsCooldown="00:30:00">    <role roleAlias="JobsRole" scaleUpCooldown="00:08:00" scaleDownCooldown="00:15:00" /> <role roleAlias="RESTAPI" scaleUpCooldown="00:08:00" scaleDownCooldown="00:15:00" /> <role roleAlias="Web" scaleUpCooldown="00:08:00" scaleDownCooldown="00:15:00" />  </stabilizer></serviceModel>

 

All seems to be good - because I get the diagnostic and performance counter.

 

Here is my C# code in my job role that is operating the block:

 

under Run():

            autoscaler = EnterpriseLibraryContainer.Current.GetInstance<Autoscaler>();
            autoscaler.Start();
and under OnStart()

            TimeSpan tsOneMinute = TimeSpan.FromMinutes(1);
 
            DiagnosticMonitorConfiguration dmc = DiagnosticMonitor.GetDefaultInitialConfiguration();
 
            dmc.PerformanceCounters.DataSources.Add(
                 new PerformanceCounterConfiguration() {
                     CounterSpecifier = @"\Processor(_Total)\% Processor Time",
                     SampleRate = TimeSpan.FromMinutes(1)
                 });
 
            dmc.PerformanceCounters.ScheduledTransferPeriod = tsOneMinute;
 
            // Transfer logs to storage every minute
 
            dmc.Logs.ScheduledTransferPeriod = tsOneMinute;
 
            // Transfer verbose, critical, etc. logs
 
            dmc.Logs.ScheduledTransferLogLevelFilter = LogLevel.Verbose;
 
            // Start up the diagnostic manager with the given configuration
 
            DiagnosticMonitor.Start("Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString", dmc);

 

So - to conclude, all seems to work - except the reactiveRules which are not run and the problem seems to be that it won't detect the Target (where the constraintRules will)

 

Ideas? :)

 

Thanks

Roman

May 9, 2012 at 3:15 AM
Edited May 9, 2012 at 4:37 AM

I took your configuration and modified it (removed 2 roles, and the queue, changed accounts etc.) and it seems to run fine against my app.

In what role are you setting the PerformanceCounterConfiguration?  It should be in the OnStart method of the RESTAPI role since that is the source of the performance counters for the reactive rules.

Are the performance counters being collected?  Do you have a WADPerformanceCountersTable created that is being populated?  I've seen where when overwriting a deployment that the WADPerformanceCountersTable was not being created unless the deployment was removed and then re-deployed.  

I notice you have a trace message for "Autoscaling General"; are you collecting trace messages for "Autoscaling Updates"?  Those should provide more details.

In general, the verbose logging provides good visibility into the block.  Autoscaling Application Block Logging provides information about configuring and interpreting logging for autoscaling. 

--
Randy Levy
Enterprise Library support engineer
entlib.support@live.com 

May 9, 2012 at 8:19 AM

Hi Randy,

Thanks for the great effort :)

Regarding your questions - I have the performance counter and anything needed according to the P&P Autoscaling documentation. but just to make sure I'd be glad to post my code.

 

In the RESTAPI role, in the EntryPoint.cs file (OnStart method) I've put this code:

 

            TimeSpan tsOneMinute = TimeSpan.FromMinutes(1);
 
            DiagnosticMonitorConfiguration dmc = DiagnosticMonitor.GetDefaultInitialConfiguration();
 
            CloudStorageAccount.SetConfigurationSettingPublisher(
                (configName, configSetter) =>
                configSetter(RoleEnvironment.GetConfigurationSettingValue(configName))
            );
 
            var cloudStorageAccount =
                CloudStorageAccount.Parse(
                RoleEnvironment.GetConfigurationSettingValue(
                "Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString")
            );
 
            dmc.PerformanceCounters.DataSources.Add(
                 new PerformanceCounterConfiguration()
                 {
                     CounterSpecifier = @"\Processor(_Total)\% Processor Time",
                     SampleRate = tsOneMinute
                 }
            );
 
            dmc.PerformanceCounters.ScheduledTransferPeriod = tsOneMinute;
 
            // Transfer logs to storage every minute
 
            dmc.Logs.ScheduledTransferPeriod = tsOneMinute;
 
            // Transfer verbose, critical, etc. logs
 
            dmc.Logs.ScheduledTransferLogLevelFilter = LogLevel.Verbose;
 
            // Start up the diagnostic manager with the given configuration
 
            DiagnosticMonitor.Start(cloudStorageAccount, dmc);

as it seems, the counter is being collected. In my diagnostics account I have 2 tables WadLogsTable and WADPerformanceCountersTable, where as the performance counter table has the performance for each instance of the RESTAPI.

Regarding the Autoscaling General, here is my diagnostics config for the diagnostics in the JobsRole (which is hosting the AutoScaler):

    <system.diagnostics>
        <sources>
            <source name="Autoscaling General"  switchName="SourceSwitch"
                switchType="System.Diagnostics.SourceSwitch" >
                <listeners>
                    <add name="AzureDiag" />
                    <remove name="Default"  />
                </listeners>
            </source>
            <source name="Autoscaling Updates"  switchName="SourceSwitch"
                switchType="System.Diagnostics.SourceSwitch" >
                <listeners>
                    <add name="AzureDiag" />
                    <remove name="Default"  />
                </listeners>
            </source>
        </sources>
        <switches>
            <add name="SourceSwitch"
                value="Verbose, Information, Warning, Error, Critical" />
        </switches>
        <sharedListeners>
            <add type="Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorTraceListener, Microsoft.WindowsAzure.Diagnostics, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
                name="AzureDiag"/>
        </sharedListeners>
        <trace>
            <listeners>
                <add type="Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorTraceListener, Microsoft.WindowsAzure.Diagnostics, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
                    name="AzureDiagnostics">
                    <filter type="" />
                </add>
            </listeners>
        </trace>
    </system.diagnostics>


Seems I AM collecting them :)

I guess there is some error somewhere which I can't detect. Any ideas where to look for i? :)

Many Thanks,
Roman
May 9, 2012 at 8:21 AM

BTW, I've redeployed it MANY MANY times :) So it's not about that one.

May 12, 2012 at 8:18 AM

Can you post the entire configuration for the JobsRole (which is hosting the AutoScaler)?

--
Randy Levy
Enterprise Library support engineer
entlib.support@live.com 

May 13, 2012 at 7:40 AM

Sure. Thank you for the help.

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
    <configSections>
        <section name="autoscalingConfiguration" type="Microsoft.Practices.EnterpriseLibrary.WindowsAzure.Autoscaling.Configuration.AutoscalingSettings, Microsoft.Practices.EnterpriseLibrary.WindowsAzure.Autoscaling, Version=5.0.1118.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" requirePermission="true" />
        <section name="typeRegistrationProvidersConfiguration" type="Microsoft.Practices.EnterpriseLibrary.Common.Configuration.TypeRegistrationProvidersConfigurationSection, Microsoft.Practices.EnterpriseLibrary.Common, Version=5.0.505.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
    </configSections>
    <autoscalingConfiguration dataPointsStoreAccount="DefaultEndpointsProtocol=https;AccountName=<store name>;AccountKey=<key>"
        ruleEvaluationRate="00:05:00" loggerName="System Diagnostics Logger"
        rulesStoreName="Blob Rules Store" serviceInformationStoreName="Blob Service Information Store">
        <loggers>
            <add name="System Diagnostics Logger" type="Microsoft.Practices.EnterpriseLibrary.WindowsAzure.Autoscaling.Logging.SystemDiagnosticsLogger, Microsoft.Practices.EnterpriseLibrary.WindowsAzure.Autoscaling, Version=5.0.1118.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
        </loggers>
        <rulesStores>
            <add name="Blob Rules Store" type="Microsoft.Practices.EnterpriseLibrary.WindowsAzure.Autoscaling.Rules.Configuration.BlobXmlFileRulesStore, Microsoft.Practices.EnterpriseLibrary.WindowsAzure.Autoscaling, Version=5.0.1118.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
                blobContainerName="autoscaling" blobName="rules-store.xml"
                storageAccount="DefaultEndpointsProtocol=https;AccountName=<store name>;AccountKey=<key>"
                monitoringRate="00:01:00" certificateThumbprint="" />
        </rulesStores>
        <serviceInformationStores>
            <add name="Blob Service Information Store" type="Microsoft.Practices.EnterpriseLibrary.WindowsAzure.Autoscaling.ServiceModel.Configuration.BlobXmlFileServiceInformationStore, Microsoft.Practices.EnterpriseLibrary.WindowsAzure.Autoscaling, Version=5.0.1118.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
                blobContainerName="autoscaling" blobName="service-information-store.xml"
                storageAccount="DefaultEndpointsProtocol=https;AccountName=<store name>;AccountKey=<key>"
                monitoringRate="00:01:00" certificateThumbprint="" />
        </serviceInformationStores>
        <advancedOptions>
            <serviceManagementRequestTracker trackServiceManagementRequests="true"
                storageAccount="DefaultEndpointsProtocol=https;AccountName=<store name>;AccountKey=<key>" />
            <executionLease useBlobExecutionLease="true" blobExecutionLeaseStorageAccount="DefaultEndpointsProtocol=https;AccountName=<store name>;AccountKey=<key>" />
        </advancedOptions>
    </autoscalingConfiguration>
    <typeRegistrationProvidersConfiguration>
        <clear />
        <add name="Caching" sectionName="cachingConfiguration" />
        <add name="Cryptography" sectionName="securityCryptographyConfiguration" />
        <add name="Exception Handling" sectionName="exceptionHandling" />
        <add name="Instrumentation" sectionName="instrumentationConfiguration" />
        <add name="Logging" sectionName="loggingConfiguration" />
        <add name="Policy Injection" sectionName="policyInjection" />
        <add name="Security" sectionName="securityConfiguration" />
        <add name="Data Access" providerType="Microsoft.Practices.EnterpriseLibrary.Data.Configuration.DatabaseSyntheticConfigSettings, Microsoft.Practices.EnterpriseLibrary.Data" />
        <add name="Validation" providerType="Microsoft.Practices.EnterpriseLibrary.Validation.Configuration.ValidationTypeRegistrationProvider, Microsoft.Practices.EnterpriseLibrary.Validation" />
        <add sectionName="autoscalingConfiguration" name="autoscalingConfiguration" />
    </typeRegistrationProvidersConfiguration>
    <system.diagnostics>
        <sources>
            <source name="Autoscaling General"  switchName="SourceSwitch"
                switchType="System.Diagnostics.SourceSwitch" >
                <listeners>
                    <add name="AzureDiag" />
                    <remove name="Default"  />
                </listeners>
            </source>
            <source name="Autoscaling Updates"  switchName="SourceSwitch"
                switchType="System.Diagnostics.SourceSwitch" >
                <listeners>
                    <add name="AzureDiag" />
                    <remove name="Default"  />
                </listeners>
            </source>
        </sources>
        <switches>
            <add name="SourceSwitch"
                value="Verbose, Information, Warning, Error, Critical" />
        </switches>
        <sharedListeners>
            <add type="Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorTraceListener, Microsoft.WindowsAzure.Diagnostics, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
                name="AzureDiag"/>
        </sharedListeners>
        <trace>
            <listeners>
                <add type="Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorTraceListener, Microsoft.WindowsAzure.Diagnostics, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
                    name="AzureDiagnostics">
                    <filter type="" />
                </add>
            </listeners>
        </trace>
    </system.diagnostics>

    <connectionStrings>
        <some string here>
    </connectionStrings>
</configuration>

May 13, 2012 at 9:09 AM
Edited May 13, 2012 at 9:09 AM

Have you configured the certificates as per Deploying the Autoscaling Application Block?  I found that was a key step (that was easy to miss) in running Autoscaling from a worker role.

--
Randy Levy
Enterprise Library support engineer
entlib.support@live.com 

May 13, 2012 at 2:48 PM

Well, just to make sure - I did the whole process again. Still without any results :/

P.S - I've left the old certificates as well. Might it be a problem?

 

Thank you very much!

May 17, 2012 at 8:07 PM

Hi all. I have the same scenario here. Any news about this case?

My log output:

Timestamp: 17/05/2012 19:56:37
Message: Rule match.
Category: Autoscaling General
Priority: -1
EventId: 1002
Severity: Verbose
Title:
Machine: FELIPEPONTES-PC
App Domain: SigHealthMonitoring.exe
ProcessId: 6428
Process Name: C:\Users\felipe.pontes\Dev\PHCServer\Tools\SigHealthMonitoring\SigHealthMonitoring\bin\Release\SigHealthMonitoring.exe
Thread Name: 
Win32 ThreadId:6844
Extended Properties: DETAILS - {"EvaluationId":"a11eb96a-f967-45fc-b504-832506da94f3","MatchingRules":[{"RuleName":"ConstraintRule","RuleDescription":"The default constraint rule","Targets":["Services"]},{"RuleName":"ScaleUpOnHighUtilization","RuleDescription":"Scale up the web role","Targets":[]},{"RuleName":"ScaleDownOnLowUtilization","RuleDescription":"Scale down the web role","Targets":[]}]}

----------------------------------------
----------------------------------------

Timestamp: 17/05/2012 19:56:37
Message: The current deployment configuration for a hosted service is about to be checked to determine if a change is required (for role scaling or changes to settings).
Category: Autoscaling Updates
Priority: -1
EventId: 3001
Severity: Verbose
Title:
Machine: FELIPEPONTES-PC
App Domain: SigHealthMonitoring.exe
ProcessId: 6428
Process Name: C:\Users\felipe.pontes\Dev\PHCServer\Tools\SigHealthMonitoring\SigHealthMonitoring\bin\Release\SigHealthMonitoring.exe
Thread Name: 
Win32 ThreadId:6844
Extended Properties: DETAILS - {"EvaluationId":"a11eb96a-f967-45fc-b504-832506da94f3","HostedServiceDetails":{"Subscription":"Azure Free Trial","HostedService":"sigbox-tests","DeploymentSlot":"Production"},"ScaleRequests":{"Services":{"Min":1,"Max":6,"AbsoluteDelta":0,"RelativeDelta":0,"MatchingRules":"ConstraintRule"}},"SettingChangeRequests":{}}
----------------------------------------
----------------------------------------

Timestamp: 17/05/2012 19:56:41
Message: Some instance count changes will be ignored.
Category: Autoscaling Updates
Priority: -1
EventId: 3012
Severity: Verbose
Title:
Machine: FELIPEPONTES-PC
App Domain: SigHealthMonitoring.exe
ProcessId: 6428
Process Name: C:\Users\felipe.pontes\Dev\PHCServer\Tools\SigHealthMonitoring\SigHealthMonitoring\bin\Release\SigHealthMonitoring.exe
Thread Name: 
Win32 ThreadId:6844
Extended Properties: DETAILS - {"EvaluationId":"a11eb96a-f967-45fc-b504-832506da94f3","HostedServiceDetails":{"Subscription":"Azure Free Trial","HostedService":"sigbox-tests","DeploymentSlot":"Production"},"InstanceChanges":{"Services":{"DesiredInstanceCount":1,"TargetInstanceCount":1}}}

----------------------------------------
----------------------------------------

Timestamp: 17/05/2012 19:56:41
Message: There are no configuration changes to submit for the hosted service.
Category: Autoscaling Updates
Priority: -1
EventId: 3004
Severity: Verbose
Title:
Machine: FELIPEPONTES-PC
App Domain: SigHealthMonitoring.exe
ProcessId: 6428
Process Name: C:\Users\felipe.pontes\Dev\PHCServer\Tools\SigHealthMonitoring\SigHealthMonitoring\bin\Release\SigHealthMonitoring.exe
Thread Name: 
Win32 ThreadId:6844
Extended Properties: DETAILS - {"EvaluationId":"a11eb96a-f967-45fc-b504-832506da94f3","HostedServiceDetails":{"Subscription":"Azure Free Trial","HostedService":"sigbox-tests","DeploymentSlot":"Production"},"ScaleRequests":{"Services":{"Min":1,"Max":6,"AbsoluteDelta":0,"RelativeDelta":0,"MatchingRules":"ConstraintRule"}},"SettingChangeRequests":{}}

----------------------------------------

Thanks in advance.

Best Regards,
Felipe Pontes.