Autoscaling not scaling

Topics: Autoscaling Application Block ("Wasabi")
Feb 20, 2013 at 9:54 AM
Edited Feb 20, 2013 at 10:31 AM
Hi Guys,

I've setup autoscaling on my azure worker role following the tutorial on azure , based on processor time.

The autoscaling is running on the same role that is targeted by autoscaling. By default the service starts one instance and it can scale up to 3 instances.

These are my autoscaling logs:
2013-02-19 23:38:04,943 [353] INFO  LiveRacers - Autoscaling General Verbose: 1002 : 
2013-02-19 23:38:04,943 [353] INFO  LiveRacers - Rule match.
[BEGIN DATA]{"EvaluationId":"1c8ac7bb-9ac7-4814-bfed-1371324339ba","MatchingRules":[{"RuleName":"default","RuleDescription":"The default constraint rule","Targets":["AutoscalingApplicationRole"]},{"RuleName":"ScaleUpOnHighUtilization","RuleDescription":"Scale up the role","Targets":[]},{"RuleName":"ScaleDownOnLowUtilization","RuleDescription":"Scale down the role","Targets":[]}]}
2013-02-19 23:38:04,943 [353] INFO  LiveRacers - Autoscaling Updates Verbose: 3001 : 
2013-02-19 23:38:04,943 [353] INFO  LiveRacers - The current deployment configuration for a hosted service is about to be checked to determine if a change is required (for role scaling or changes to settings).
[BEGIN DATA]{"EvaluationId":"1c8ac7bb-9ac7-4814-bfed-1371324339ba","HostedServiceDetails":{"Subscription":"Windows Azure MSDN - Visual Studio Ultimate","HostedService":"liveracers","DeploymentSlot":"Staging"},"ScaleRequests":{"AutoscalingApplicationRole":{"Min":1,"Max":3,"AbsoluteDelta":0,"RelativeDelta":0,"MatchingRules":"default"}},"SettingChangeRequests":{}}
2013-02-19 23:38:05,771 [353] INFO  LiveRacers - Autoscaling Updates Verbose: 3012 : 
2013-02-19 23:38:05,771 [353] INFO  LiveRacers - Some instance count changes will be ignored.
[BEGIN DATA]{"EvaluationId":"1c8ac7bb-9ac7-4814-bfed-1371324339ba","HostedServiceDetails":{"Subscription":"Windows Azure MSDN - Visual Studio Ultimate","HostedService":"liveracers","DeploymentSlot":"Staging"},"InstanceChanges":{"AutoscalingApplicationRole":{"DesiredInstanceCount":1,"TargetInstanceCount":1}}}
2013-02-19 23:38:05,771 [353] INFO  LiveRacers - Autoscaling Updates Verbose: 3004 : 
2013-02-19 23:38:05,771 [353] INFO  LiveRacers - There are no configuration changes to submit for the hosted service.
[BEGIN DATA]{"EvaluationId":"1c8ac7bb-9ac7-4814-bfed-1371324339ba","HostedServiceDetails":{"Subscription":"Windows Azure MSDN - Visual Studio Ultimate","HostedService":"liveracers","DeploymentSlot":"Staging"},"ScaleRequests":{"AutoscalingApplicationRole":{"Min":1,"Max":3,"AbsoluteDelta":0,"RelativeDelta":0,"MatchingRules":"default"}},"SettingChangeRequests":{}}
2013-02-19 23:38:43,677 [22] INFO  LiveRacers - Tcp Live Connections 110
2013-02-19 23:42:02,740 [321] INFO  LiveRacers - Autoscaling General Verbose: 1002 : 
2013-02-19 23:42:02,740 [321] INFO  LiveRacers - Rule match.
[BEGIN DATA]{"EvaluationId":"30003fae-998f-4b8f-818a-9a857d66c3a6","MatchingRules":[{"RuleName":"default","RuleDescription":"The default constraint rule","Targets":["AutoscalingApplicationRole"]},{"RuleName":"ScaleUpOnHighUtilization","RuleDescription":"Scale up the role","Targets":[]},{"RuleName":"ScaleDownOnLowUtilization","RuleDescription":"Scale down the role","Targets":[]}]}
2013-02-19 23:42:02,740 [321] INFO  LiveRacers - Autoscaling Updates Verbose: 3001 : 
2013-02-19 23:42:02,740 [321] INFO  LiveRacers - The current deployment configuration for a hosted service is about to be checked to determine if a change is required (for role scaling or changes to settings).
[BEGIN DATA]{"EvaluationId":"30003fae-998f-4b8f-818a-9a857d66c3a6","HostedServiceDetails":{"Subscription":"Windows Azure MSDN - Visual Studio Ultimate","HostedService":"liveracers","DeploymentSlot":"Staging"},"ScaleRequests":{"AutoscalingApplicationRole":{"Min":1,"Max":3,"AbsoluteDelta":0,"RelativeDelta":0,"MatchingRules":"default"}},"SettingChangeRequests":{}}
2013-02-19 23:42:04,334 [321] INFO  LiveRacers - Autoscaling Updates Verbose: 3012 : 
2013-02-19 23:42:04,334 [321] INFO  LiveRacers - Some instance count changes will be ignored.
[BEGIN DATA]{"EvaluationId":"30003fae-998f-4b8f-818a-9a857d66c3a6","HostedServiceDetails":{"Subscription":"Windows Azure MSDN - Visual Studio Ultimate","HostedService":"liveracers","DeploymentSlot":"Staging"},"InstanceChanges":{"AutoscalingApplicationRole":{"DesiredInstanceCount":1,"TargetInstanceCount":1}}}
2013-02-19 23:42:04,334 [321] INFO  LiveRacers - Autoscaling Updates Verbose: 3004 : 
2013-02-19 23:42:04,334 [321] INFO  LiveRacers - There are no configuration changes to submit for the hosted service.
[BEGIN DATA]{"EvaluationId":"30003fae-998f-4b8f-818a-9a857d66c3a6","HostedServiceDetails":{"Subscription":"Windows Azure MSDN - Visual Studio Ultimate","HostedService":"liveracers","DeploymentSlot":"Staging"},"ScaleRequests":{"AutoscalingApplicationRole":{"Min":1,"Max":3,"AbsoluteDelta":0,"RelativeDelta":0,"MatchingRules":"default"}},"SettingChangeRequests":{}}
And this is my configuration.

Rules:
<?xml version="1.0" encoding="utf-8" ?>
<rules xmlns="http://schemas.microsoft.com/practices/2011/entlib/autoscaling/rules">
  <constraintRules>
    <rule name="default" enabled="true" rank="1" description="The default constraint rule">
      <actions>
        <range min="1" max="3" target="AutoscalingApplicationRole"/>
      </actions>
    </rule>
  </constraintRules>
  <reactiveRules>
    <rule name="ScaleUpOnHighUtilization" rank="10" description="Scale up the role" enabled="true" >
      <when>
        <any>
          <greaterOrEqual operand="Role_CPU_Avg_5m" than="60"/>
        </any>
      </when>
      <actions>
        <scale target="AutoscalingApplicationRole" by="1"/>
      </actions>
    </rule>
    <rule name="ScaleDownOnLowUtilization" rank="10" description="Scale down the role" enabled="true" >
      <when>
        <all>
          <less operand="Role_CPU_Avg_5m" than="40"/>
        </all>
      </when>
      <actions>
        <scale target="AutoscalingApplicationRole" by="-1"/>
      </actions>
    </rule>
  </reactiveRules>
  <operands>
    <performanceCounter alias="Role_CPU_Avg_5m"
      performanceCounterName="\Processor(_Total)\% Processor Time"
      source ="AutoscalingApplicationRole"
      timespan="00:05:00" aggregate="Average"/>
  </operands>  
</rules>
Service:
<?xml version="1.0" encoding="utf-8" ?>
<serviceModel xmlns="http://schemas.microsoft.com/practices/2011/entlib/autoscaling/serviceModel">
  <subscriptions>
    <subscription name="Windows Azure MSDN - Visual Studio Ultimate"
                  certificateThumbprint="xxx"
                  subscriptionId="xxx"
                  certificateStoreLocation="LocalMachine"
                  certificateStoreName="My">
      <services>
        <service dnsPrefix="liveracers" slot="Staging">
          <roles>
            <role alias="AutoscalingApplicationRole"
                  roleName="LiveRacers.PluginServer"
                  wadStorageAccountName="targetstorage"/>
          </roles>
        </service>
      </services>
      <storageAccounts>
        <storageAccount alias="targetstorage"
          connectionString="DefaultEndpointsProtocol=https;AccountName=xxx;AccountKey=xxx">
        </storageAccount>
      </storageAccounts>
    </subscription>
  </subscriptions>
  <stabilizer scaleUpCooldown="00:10:00"
            scaleDownCooldown="00:10:00"
            scaleUpOnlyInFirstMinutesOfHour="0"
            scaleDownOnlyInLastMinutesOfHour="20" />
</serviceModel>
I can see the AutoscalerDatapoints table generated in the targetstorage account.

Any idea what could be wrong?

Note that I.m definitely hitting the 60% CPU time.When logged on the machine, in task manager, the CPU never goes below 70%. I've also had Performance Monitor open and the average CPU was above 70% for several hours. It's running in Staging.


Thanks,

Stevo
Feb 21, 2013 at 6:55 AM
Just to confirm that you have configured the role to collect the performance counters? Check in Azure Storage for WADPerformanceCountersTable and also check what values have been written. Also, verify that the AutoscalerDatapoints has the \Processor* info and not just RoleInstanceCount.

You could try simplifying by removing the stabilizer and adding one rule that should always be true (e.g. processor > 0 scale up) just to get it working and then add more realistic rules.

~~
Randy Levy
entlib.support@live.com
Enterprise Library support engineer
Support How-to
Feb 21, 2013 at 10:00 AM
Edited Feb 21, 2013 at 11:26 AM
I've put this code into OnStart()
DiagnosticMonitorConfiguration config = DiagnosticMonitor.GetDefaultInitialConfiguration();
config.PerformanceCounters.DataSources.Add(new PerformanceCounterConfiguration()
{
CounterSpecifier = @"\Processor(_Total)\% Processor Time",
SampleRate = TimeSpan.FromSeconds(30)
});
config.PerformanceCounters.ScheduledTransferPeriod = TimeSpan.FromMinutes(1);
DiagnosticMonitor.Start("Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString", config);
But it's not created WADPerformanceCountersTable, neither I have the \Processor data points in AutoscalerDatapoints

Any idea what could be causing this?

Edit:
I'm also reading that the performance counters should be mentioned in the wad-control-container deployment file. However that has no reference to any performance counters.
Feb 22, 2013 at 11:54 AM
Hi

thanks for the support, I've found the problem. It was caused by calling the DiagnosticsMonitor.Start(...) more times in different areas of the application for different reasons. So the second call (that was configuring the performance monitoring) was just ignored.

For future reference it might be beneficial if calling DiagnosticsMonitor.Start() multiple times within one domain would raise an exception.

Regards,

Stevo
Feb 23, 2013 at 3:23 AM
Thanks for posting the issue -- hopefully it can help someone else.

~~
Randy Levy
entlib.support@live.com
Enterprise Library support engineer
Support How-to