Activation error occured while trying to get instance of type Autoscaler, key ""

Topics: Windows Azure Integration Pack
May 7, 2012 at 6:49 AM

I've followed http://www.windowsazure.com/en-us/develop/net/how-to-guides/autoscaling/#Instantiate step by step and getting below exception. I did not see anyone posting this error while using Autoscaling block...

 

 Microsoft.Practices.ServiceLocation.ActivationException was unhandled

Message=Activation error occured while trying to get instance of type Autoscaler, key ""

Source=Microsoft.Practices.ServiceLocation

StackTrace:

at Microsoft.Practices.ServiceLocation.ServiceLocatorImplBase.GetInstance(Type serviceType, String key) in c:\Home\Chris\Projects\CommonServiceLocator\main\Microsoft.Practices.ServiceLocation\ServiceLocatorImplBase.cs:line 53

at Microsoft.Practices.ServiceLocation.ServiceLocatorImplBase.GetInstance[TService]() in c:\Home\Chris\Projects\CommonServiceLocator\main\Microsoft.Practices.ServiceLocation\ServiceLocatorImplBase.cs:line 90

at AutoScaleOnPrem.Program.Main(String[] args) in c:\users\harid\documents\visual studio 2010\Projects\AutoScaleOnPrem\AutoScaleOnPrem\Program.cs:line 15

at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)

at System.AppDomain.nExecuteAssembly(RuntimeAssembly assembly, String[] args)

at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)

at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()

at System.Threading.ThreadHelper.ThreadStart_Context(Object state)

at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)

at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)

at System.Threading.ThreadHelper.ThreadStart()

InnerException: Microsoft.Practices.Unity.ResolutionFailedException

Message=Resolution of the dependency failed, type = "Microsoft.Practices.EnterpriseLibrary.WindowsAzure.Autoscaling.Autoscaler", name = "(none)".

Exception occurred while: while resolving.

Exception is: InvalidOperationException - The type String cannot be constructed. You must configure the container to supply this value.

-----------------------------------------------

At the time of the exception, the container was:

Resolving Microsoft.Practices.EnterpriseLibrary.WindowsAzure.Autoscaling.Autoscaler,(none)

Resolving parameter "dataPointsStoreStorageAccountConnectionString" of constructor Microsoft.Practices.EnterpriseLibrary.WindowsAzure.Autoscaling.Autoscaler(System.String dataPointsStoreStorageAccountConnectionString, System.String dataPointsTableName, System.TimeSpan ruleEvaluationRate, Microsoft.Practices.EnterpriseLibrary.WindowsAzure.Autoscaling.Rules.IRulesStore rulesStore, Microsoft.Practices.EnterpriseLibrary.WindowsAzure.Autoscaling.ServiceModel.IServiceInformationStore serviceInformationStore, Microsoft.Practices.EnterpriseLibrary.WindowsAzure.Autoscaling.DataPointsCollection.IDataPointsCollectorsProvider dataPointsCollectorsProvider, Microsoft.Practices.EnterpriseLibrary.WindowsAzure.Autoscaling.IExecutionLease executionLease, Microsoft.Practices.EnterpriseLibrary.WindowsAzure.Autoscaling.Logging.ILogger logger, Microsoft.Practices.EnterpriseLibrary.WindowsAzure.Autoscaling.Scaling.IServiceManagementRequestTracker tracker, Microsoft.Practices.EnterpriseLibrary.WindowsAzure.Autoscaling.Scaling.IScaler scaler)

Resolving System.String,(none)

Source=Microsoft.Practices.Unity

TypeRequested=Autoscaler

StackTrace:

at Microsoft.Practices.Unity.UnityContainer.DoBuildUp(Type t, Object existing, String name, IEnumerable`1 resolverOverrides)

at Microsoft.Practices.Unity.UnityContainer.DoBuildUp(Type t, String name, IEnumerable`1 resolverOverrides)

at Microsoft.Practices.Unity.UnityContainer.Resolve(Type t, String name, ResolverOverride[] resolverOverrides)

at Microsoft.Practices.Unity.UnityServiceLocator.DoGetInstance(Type serviceType, String key)

at Microsoft.Practices.ServiceLocation.ServiceLocatorImplBase.GetInstance(Type serviceType, String key) in c:\Home\Chris\Projects\CommonServiceLocator\main\Microsoft.Practices.ServiceLocation\ServiceLocatorImplBase.cs:line 49

InnerException: System.InvalidOperationException

Message=The type String cannot be constructed. You must configure the container to supply this value.

Source=Microsoft.Practices.Unity

StackTrace:

at Microsoft.Practices.ObjectBuilder2.DynamicMethodConstructorStrategy.GuardTypeIsNonPrimitive(IBuilderContext context, SelectedConstructor selectedConstructor)

at Microsoft.Practices.ObjectBuilder2.DynamicMethodConstructorStrategy.PreBuildUp(IBuilderContext context)

at Microsoft.Practices.ObjectBuilder2.StrategyChain.ExecuteBuildUp(IBuilderContext context)

at Microsoft.Practices.ObjectBuilder2.DynamicMethodBuildPlanCreatorPolicy.CreatePlan(IBuilderContext context, NamedTypeBuildKey buildKey)

at Microsoft.Practices.ObjectBuilder2.BuildPlanStrategy.PreBuildUp(IBuilderContext context)

at Microsoft.Practices.ObjectBuilder2.StrategyChain.ExecuteBuildUp(IBuilderContext context)

at Microsoft.Practices.ObjectBuilder2.BuilderContext.NewBuildUp(NamedTypeBuildKey newBuildKey)

at Microsoft.Practices.Unity.ObjectBuilder.NamedTypeDependencyResolverPolicy.Resolve(IBuilderContext context)

at BuildUp_Microsoft.Practices.EnterpriseLibrary.WindowsAzure.Autoscaling.Autoscaler(IBuilderContext )

at Microsoft.Practices.ObjectBuilder2.DynamicMethodBuildPlan.BuildUp(IBuilderContext context)

at Microsoft.Practices.ObjectBuilder2.BuildPlanStrategy.PreBuildUp(IBuilderContext context)

at Microsoft.Practices.ObjectBuilder2.StrategyChain.ExecuteBuildUp(IBuilderContext context)

at Microsoft.Practices.Unity.UnityContainer.DoBuildUp(Type t, Object existing, String name, IEnumerable`1 resolverOverrides)

InnerException:

 

Can someone help me how to resolve this issue? Also, one of my customers is able to run the application, however reactive rules does not seem to function... How to troubleshoot this issue?

May 8, 2012 at 8:45 AM

The message you are seeing usually indicates that there is a configuration issue.  The above error will appear if the app.config is not deployed to the output folder (as AutoscalingHost.exe.config) or if the autoscalingConfiguration section does not exist in the app.config file.  

--
Randy Levy
Enterprise Library support engineer
entlib.support@live.com 

May 13, 2012 at 4:06 AM

I've set "Copy Local" to true for app.config and it started running. However both constraint, reactive rules are not working... Below is the configuration.

Services.xml

<!--l version="1.0" encoding="utf-8"-->
<?xml version="1.0" encoding="utf-8" ?>
<serviceModel xmlns="http://schemas.microsoft.com/practices/2011/entlib/autoscaling/serviceModel">
  <subscriptions>
    <subscription name="DSAzure-harid"
                  certificateThumbprint="AFA149465F28717ACB02302FF7D52D5C45ADDF3B"
                  subscriptionId="000000000000-0000-0000-0000-000000000000"
                  certificateStoreLocation="CurrentUser"
                  certificateStoreName="My">
      <services>
        <service dnsPrefix="mostwanted" slot="Production">
          <roles>
            <role alias="AutoscalingApplicationRole"
                  roleName="WorkerRole2"
                  wadStorageAccountName="targetstorage"/>
          </roles>
        </service>
      </services>
      <storageAccounts>
        <storageAccount alias="targetstorage"
          connectionString="DefaultEndpointsProtocol=https;AccountName=autoscale;AccountKey=<<ActualKey>>">
        </storageAccount>
      </storageAccounts>
    </subscription>
  </subscriptions>
</serviceModel>

Rules.xml

<?xml version="1.0" encoding="utf-8" ?>
<rules xmlns="http://schemas.microsoft.com/practices/2011/entlib/autoscaling/rules">
  <constraintRules>
    <rule name="default" enabled="true" rank="1" description="The default constraint rule">
      <actions>
        <range min="3" max="6" target="AutoscalingApplicationRole"/>
      </actions>
    </rule>
  </constraintRules>
  <reactiveRules>
    <rule name="ScaleUpOnHighUtilization" rank="2" description="Scale up the web role" enabled="true" >
      <when>
        <any>
          <greaterOrEqual operand="WebRoleA_CPU_Avg_5m" than="05"/>
        </any>
      </when>
      <actions>
        <scale target="AutoscalingApplicationRole" by="1"/>
      </actions>
    </rule>
    <rule name="ScaleDownOnLowUtilization" rank="2" description="Scale up the web role" enabled="true" >
      <when>
        <all>
          <less operand="WebRoleA_CPU_Avg_5m" than="60"/>
        </all>
      </when>
      <actions>
        <scale target="AutoscalingApplicationRole" by="-1"/>
      </actions>
    </rule>
  </reactiveRules>
  <operands>
    <performanceCounter alias="WebRoleA_CPU_Avg_5m"
      performanceCounterName="\Processor(_Total)\% Processor Time"
      source ="AutoscalingApplicationRole"
      timespan="00:05:00" aggregate="Average"/>
  </operands>
</rules>

 App.config

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
    <configSections>
        <section name="autoscalingConfiguration" type="Microsoft.Practices.EnterpriseLibrary.WindowsAzure.Autoscaling.Configuration.AutoscalingSettings, Microsoft.Practices.EnterpriseLibrary.WindowsAzure.Autoscaling, Version=5.0.1118.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" requirePermission="true" />
        <section name="typeRegistrationProvidersConfiguration" type="Microsoft.Practices.EnterpriseLibrary.Common.Configuration.TypeRegistrationProvidersConfigurationSection, Microsoft.Practices.EnterpriseLibrary.Common, Version=5.0.505.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
    </configSections>
    <autoscalingConfiguration dataPointsStoreAccount="DefaultEndpointsProtocol=https;AccountName=autoscale;AccountKey=5ReQmNhKMBqxgwDxr5lADh0DftbDJwz0FYyaF04VT0sOrPLmnaqqwngH5aqp0ZKqHMOJJJefOXAm7wJ9VmxRTg=="
        loggerName="System Diagnostics Logger" rulesStoreName="Local File Rules Store"
        serviceInformationStoreName="Local File Service Information Store">
        <loggers>
            <add name="System Diagnostics Logger" type="Microsoft.Practices.EnterpriseLibrary.WindowsAzure.Autoscaling.Logging.SystemDiagnosticsLogger, Microsoft.Practices.EnterpriseLibrary.WindowsAzure.Autoscaling, Version=5.0.1118.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
        </loggers>
        <rulesStores>
            <add name="Local File Rules Store" type="Microsoft.Practices.EnterpriseLibrary.WindowsAzure.Autoscaling.Rules.Configuration.LocalXmlFileRulesStore, Microsoft.Practices.EnterpriseLibrary.WindowsAzure.Autoscaling, Version=5.0.1118.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
                filename="rules.xml" />
        </rulesStores>
        <serviceInformationStores>
            <add name="Local File Service Information Store" type="Microsoft.Practices.EnterpriseLibrary.WindowsAzure.Autoscaling.ServiceModel.Configuration.LocalXmlFileServiceInformationStore, Microsoft.Practices.EnterpriseLibrary.WindowsAzure.Autoscaling, Version=5.0.1118.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
                filename="services.xml" />
        </serviceInformationStores>
        <advancedOptions>
            <serviceManagementRequestTracker trackServiceManagementRequests="false" />
            <executionLease useBlobExecutionLease="false" />
        </advancedOptions>
    </autoscalingConfiguration>
    <typeRegistrationProvidersConfiguration>
        <clear />
        <add name="Caching" sectionName="cachingConfiguration" />
        <add name="Cryptography" sectionName="securityCryptographyConfiguration" />
        <add name="Exception Handling" sectionName="exceptionHandling" />
        <add name="Instrumentation" sectionName="instrumentationConfiguration" />
        <add name="Logging" sectionName="loggingConfiguration" />
        <add name="Policy Injection" sectionName="policyInjection" />
        <add name="Security" sectionName="securityConfiguration" />
        <add name="Data Access" providerType="Microsoft.Practices.EnterpriseLibrary.Data.Configuration.DatabaseSyntheticConfigSettings, Microsoft.Practices.EnterpriseLibrary.Data" />
        <add name="Validation" providerType="Microsoft.Practices.EnterpriseLibrary.Validation.Configuration.ValidationTypeRegistrationProvider, Microsoft.Practices.EnterpriseLibrary.Validation" />
        <add sectionName="autoscalingConfiguration" name="autoscalingConfiguration" />
    </typeRegistrationProvidersConfiguration>
    <!--<system.diagnostics>
        <trace>
            <listeners>
                <add type="Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorTraceListener, Microsoft.WindowsAzure.Diagnostics, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
                    name="AzureDiagnostics">
                    <filter type="" />
                </add>
            </listeners>
        </trace>
    </system.diagnostics>-->

  <system.diagnostics>
    <sources>
      <source name="Autoscaling General" switchName="SourceSwitch" switchType="System.Diagnostics.SourceSwitch" />
      <source name="Autoscaling Updates" switchName="SourceSwitch" switchType="System.Diagnostics.SourceSwitch" />
    </sources>
    <switches>
      <add name="SourceSwitch" value="Verbose, Information, Warning, Error, Critical" />
    </switches>
  </system.diagnostics>

  
  
</configuration>

 

Main Method

 

 

 

 

Autoscaler

scaler = EnterpriseLibraryContainer.Current.GetInstance<Autoscaler

>();

scaler.Start();

 

 

Console.Read();

 


May 13, 2012 at 5:59 AM

Are you running AutoScaling in a Console Application?  I'm confused about setting "Copy Local" on app.config since app.config should be copied to "applicationname".exe.config.  Are the rules.xml and services.xml also copied to the output folder?

Are you getting logging output?  Based on the posted configuration you should be able to see logging output with DebugView.  Autoscaling Application Block Logging has a description of the messages you may see.

For the reactive rules to work ensure that the WorkerRole2 is configured to collect performance counter information.  Also verify that the performance counter information is collected in the WADPerformanceCountersTable.  It's not the issue but the effect of the posted reactive rules is for CPU between 0-5 to decrease instances by 1 and for all other values to increase instances by 1.  See Understanding Rule Ranks and Reconciliation for conflict info.

--
Randy Levy
Enterprise Library support engineer
entlib.support@live.com 

May 14, 2012 at 9:36 AM

Hi Randy,

Yes, I'm running Autoscaling in a Console Application. app.config indeed copied as applicationname.exe.config. But, still resulted in the original error i've posted. After setting the Copy Local to true, it ran with out the Activation error. It is very weird! I agree. Rules.xml, services.xml were  not copied to bin directory before. I have marked "Copy Local" to true for these files too and now they are being copied to bin folder. Still none of the constraint rules, reactive rules work.

I do not see any logging output. I'm checking in Output window of VS. Also downloaded DebugView applicaton from http://technet.microsoft.com/en-us/sysinternals/bb896647 and i do not see anything there either.

Performance logs are being collected. I checked WadPerformanceCounters Table and confirmed the same.

Thanks,

Hari

May 14, 2012 at 6:23 PM

My Autoscaling console application works fine using what you have posted.  Perhaps you can upload a (sanitized) console application that reproduces the issue?

--
Randy Levy
Enterprise Library support engineer
entlib.support@live.com 

May 15, 2012 at 3:29 AM
randylevy wrote:

My Autoscaling console application works fine using what you have posted.  Perhaps you can upload a (sanitized) console application that reproduces the issue?

--
Randy Levy
Enterprise Library support engineer
entlib.support@live.com 


There was no option here to attach a file... Otherwise , i would have attached the whole project... That's the reason i copied all the configuration and the code.

I'm surprised that it works for you. I will find a way to upload a file and refer the path here...

May 15, 2012 at 3:38 AM

Here you go!

http://dl.dropbox.com/u/79503436/AutoScaleOnPrem.zip

 

May 15, 2012 at 6:33 AM

The project seems to work fine for me.

  • The app.config is properly copied to autoscaleonprem.exe.config
  • Log messages appear in Debug View when running the application (not debugging)
  • The role was properly scaled

All I modified was to get it to work was to:

  • Add my storage account connection string as the value for dataPointsStoreAccount in app.config
  • Add my subscription info to services.xml
  • In services.xml set my dnsPrefix, change roleName to my actual roleName and enter my storage account connectionString.

 Are you sure you've added performance counters for WorkerRole2 and not for WebRole1 (since it looks like you are using an ASP.NET web application)?

--
Randy Levy
Enterprise Library support engineer
entlib.support@live.com 

May 16, 2012 at 4:04 AM

Yes , I'm sure i've added perf counters for WorkerRole2 and i looked into storage table WADPerformanceCounters and verified . Below are few sample entries.

ebcec634dbe749a880fcc954221652a0___WorkerRole2___WorkerRole2_IN_0___0000000001038090248___WADPerformanceCountersLocalQuery	07-05-2012 08:58	6.3472E+17	ebcec634dbe749a880fcc954221652a0	WorkerRole2	WorkerRole2_IN_0	\Processor(_Total)\% Processor Time	0.306758
ebcec634dbe749a880fcc954221652a0___WorkerRole2___WorkerRole2_IN_0___0000000001038090249___WADPerformanceCountersLocalQuery	07-05-2012 08:58	6.3472E+17	ebcec634dbe749a880fcc954221652a0	WorkerRole2	WorkerRole2_IN_0	\Processor(_Total)\% Processor Time	0.306758
May 31, 2012 at 9:36 PM

I am getting the same exception.  I am trying to host the Autoscaler in a worker role.  it throws this exception when calling this code...

 

try

{

);

scaler = EnterpriseLibraryContainer.Current.GetInstance<Autoscaler 

>();

 

Trace.TraceInformation("Starting the Autoscalar");

{

Trace.TraceError("Could not get and start autoscaler: "

+ e.Message);

}

I am copying to output my app.config, the role continues to run after the exception, but never does the autoscaling, since the scaler did not start. 



scaler.Start();

}

 

catch (Exception e)

Trace.TraceInformation("Initialize the Autoscaler"

Jun 4, 2012 at 4:40 AM
Edited Jun 4, 2012 at 4:40 AM

@steveloethen: this message usually means that autoscaling is not configured in app.config.  Or it may also mean that IntelliTrace is enabled.  Verify that you are publishing the correct configuration file and that the app is not running IntelliTrace (if applicable).  If you still have issues, I would recommend starting a new discussion with the applicable configuration information.

--
Randy Levy
Enterprise Library support engineer
entlib.support@live.com 

Oct 17, 2012 at 7:41 AM

I have followed all the steps described above. No IntelliTrace. I continue to have the same issue: Activation error occured while trying to get instance of type Autoscaler. I have used project from http://dl.dropbox.com/u/79503436/AutoScaleOnPrem.zip . In services.xml file my parameters are added. Did you try this project on the "clean" PC?

Oct 17, 2012 at 2:15 PM

The exact error appears as an InnerException (or a nested InnerException) to the ActivationException which can give a good hint as to the problem.  

In your case @Viacheslav, the provided dataPointsStoreAccount was not specified so the InnerException was an ArgumentException on "AccountName".  The dataPointsStoreAccount must also be a valid connection string or else you could get an InnerException of type AzureStorageException with the message "The provided Windows Azure Storage connection string is invalid".  Once a proper account was added to the dataPointsStoreAccount the scaler can be resolved.

 Also, it's helpful to start a new thread even for issues that seem related.  Thanks.

--
Randy Levy
Enterprise Library support engineer
entlib.support@live.com 

Oct 19, 2012 at 7:27 AM

In Autoscaler test console application I have added reference to Microsoft.WindowsAzure.StorageClient.dll from Azure SDK v.1.6. No errors. Autoscaler is able to change the number of WebRole instances depending on Requests per second Performance counter value (@"\ASP.NET Applications(__Total__)\Requests/Sec").

If the reference is done to Microsoft.WindowsAzure.StorageClient.dll from Azure SDK v.1.7 then it does not work at all.

If the reference is done to Microsoft.WindowsAzure.StorageClient.dll from Azure SDK v.1.5 then Autoscaler is started, the AutoscalerDatapoints table is created, but the points are not collected in this table.

To deal with issues it is usefull to add the following block to app.config:

  <system.diagnostics>
    <trace autoflush="true"/>
    <sources>
      <source name="Autoscaling General"
              switchName="SourceSwitch"
              switchType="System.Diagnostics.SourceSwitch" >
        <listeners>
          <add name="console" />
          <add name="file" />
          <remove name="Default"  />
        </listeners>
      </source>
      <source name="Autoscaling Updates"
              switchName="SourceSwitch"
              switchType="System.Diagnostics.SourceSwitch" >
        <listeners>
          <add name="console" />
          <add name="file" />
          <remove name="Default" />
        </listeners>
      </source>
    </sources>
    <sharedListeners>
      <add name="console" type="System.Diagnostics.ConsoleTraceListener">
        <filter type="System.Diagnostics.EventTypeFilter" initializeData="Verbose"/>
      </add>
      <add name="file" type="System.Diagnostics.TextWriterTraceListener" initializeData="C:\Logs\ConsoleAutoscaler.log">
        <filter type="System.Diagnostics.EventTypeFilter" initializeData="Verbose"/>
      </add>
    </sharedListeners>
    <switches>
      <add name="SourceSwitch"
          value="Verbose, Information, Warning, Error, Critical" />
    </switches>
  </system.diagnostics>

Jan 23, 2013 at 3:05 PM

Why does it have to be so complicated to use a core cloud feature like Autoscaler for your app?? Microsoft loves to create complex things to waste your time..

Why couldn't they create something more user friendly like in Amazon or like 3rd party solutions (e.g., AzureWatch Paraleap, etc.)?

It's really frustrating [sighs]... :\

Jan 27, 2013 at 4:53 PM

Ok so I took the console application that "harid" posted. I did the changes you suggested. The console app runs without errors; however, there are two problems:

1- I couldn't see any scaling action taking place on my application! I looked through everything and everything seems to be right (WAD metrics are collected, service information is correct, etc.)! 

2- I still couldn't see any debug information for the Autoscaler :s

 

Is there a working Console/Worker-role application you can post? many people are facing problems with it although everyone seems to follow the instructions in the article!

Jan 29, 2013 at 4:28 AM

I re-downloaded the project and made the changes I posted earlier (plus a binding redirect to StorageClient 1.7) and it is working.

How are you checking for debug information?  I see it in DebugView (when running) or the Output window (when debugging).

Have you installed the management certificate into the local (on premises) certificate store?  

Start -> MMC -> Add/Remove Snap-in -> Certificates (Current User) -> OK.  Then the management certificate should appear in Certificates - Current User->Personal->Certificates.

--
Randy Levy
Enterprise Library support engineer
entlib.support@live.com 

Jan 29, 2013 at 4:10 PM
randylevy wrote:

I re-downloaded the project and made the changes I posted earlier (plus a binding redirect to StorageClient 1.7) and it is working.

How are you checking for debug information?  I see it in DebugView (when running) or the Output window (when debugging).

Have you installed the management certificate into the local (on premises) certificate store?  

Start -> MMC -> Add/Remove Snap-in -> Certificates (Current User) -> OK.  Then the management certificate should appear in Certificates - Current User->Personal->Certificates.

--
Randy Levy
Enterprise Library support engineer
entlib.support@live.com 

Oh, I can now see the output of the Autoscaler.. thanks to you! :)

Turns out it was a certificate issue-- wish the original article pinpointed that! The Autoscaler works perfectly now :)

Thanks again!

--
Mohammad