Troubleshooting Sitecore on Azure PAAS – Part 3

Scaling

Today we are going to discuss the Scaling feature of Azure and in particular Scale out and how you can troubleshoot your Sitecore instances. If you haven’t gone through earlier posts on troubleshooting Sitecore on Azure PAAS, do visit https://pratiksatikunvar.wordpress.com/category/sitecore/azure-paas/.

One of the reason enterprise level organizations wants their site in Azure PAAS is the ability to scale at anytime with little to no efforts. Azure provides great scaling options. There are two main ways that an application can scale:

  • Vertical scaling, also called scaling up and down, means changing the capacity of a resource. For example, you could move an application to a larger VM size. Vertical scaling often requires making the system temporarily unavailable while it is being redeployed. Therefore, it’s less common to automate vertical scaling.

  • Horizontal scaling, also called scaling out and in, means adding or removing instances of a resource. The application continues running without interruption as new resources are provisioned. When the provisioning process is complete, the solution is deployed on these additional resources. If demand drops, the additional resources can be shut down cleanly and deallocated.

Both kind of scaling can be done automatic or manually. Automatic scaling can be done based on Resource Metrics, Custom Metrics, Time, Metric/time based Rules, Default Scaling.

With the project i am working currently, have XP scaled on Azure PAAS. We have one CD App Service, additional Web App for Staging Slot, and CD Web App is by default scaled out to two instances. These terminologies is a bit confusing and it is necessary to understand that well to understand your infrastructure.

  • Web App: It’s an application you wanted to develop and deploy. It can be .Net application, Mobile application etc. So, if you have one CD, you can say there is one Web App for CD inside CD Web service.

  • Web Service: It’s a service for hosting web applications or also known as Web App. Web Service can have multiple different kind of Web Apps. App Service not only adds the power of Microsoft Azure to your application, such as security, load balancing, autoscaling, and automated management. You can also take advantage of its DevOps capabilities, such as continuous deployment from Azure DevOps, GitHub, Docker Hub, and other sources, package management, staging environments, custom domain, and SSL certificates. he compute resources you use is determined by the App Service plan that you run your apps on.

  • Web App Instance: In a simple definition we can say number of VMs (on cloud) in which your application is running. So, if you have one CD, it means at least once Instance is running on some VM somewhere in specified data-center. In Scale out option, you can increase the number of instances based on what plan you’ve. This can be increased either based on some rule or by default. So, if someone says I have two instance running for my CD Web App, you should get that there is only one CD in front but in back-end there are two separate VMs on which your CD is running.

You have clear idea now on above different terminologies. So, i will explain my CD infrastructure. And you should be able to relate. We have one CD App Service. We are using Slots for blue/green deployments with the help of production/staging Slots. So, each slot itself is a different Web App. So, we have two Web App in one CD App Service. And each Web App is by default scaled out to two instances. This mean, for one Slot/Web App there are two VMs running behind the scene.

You only deploy your code on one Slot/Web App and not to two individual instances. That being managed by Azure in back-end. All the instances of a Web App will be sharing same web root of that particular Web App.

How to Troubleshoot issues specific to Instances?

If some of the users are experiencing weird issues while for some it works fine including you, how you’re going to troubleshoot such issues. As it’s possible that due to multiple instances any of instance could have some issue.

In IaaS we used to have load-balancer where we can setup Sticky algorithms which will transfer all requests of one user to same VM. But in azure PaaS this is done by Traffic Manager. So, Can we achieve same above sticky behavior with Azure PaaS without any extra efforts? Yes. There comes the ARRAffinity into picture.

Setting up multiple instances of a website in Windows Azure Websites is a terrific way to scale out your website, and Azure makes great use of the Application Request Routing IIS Extension to distribute your connecting users between your active instances. ARR cleverly keeps track of connecting users by giving them a special cookie (known as an affinity cookie), which allows it to know, upon subsequent requests, to which server instance they were talking to. This way, we can be sure that once a client establishes a session with a specific server instance, it will keep talking to the same server as long as his session is active.

Go to your App Service/Web App => Settings => Configuration => General Settings to see setting related to Affinity as shown below. By default it’s value is on.
2019-06-01 14_48_58-cspisango-dev-rg-single - Configuration - Microsoft Azure

When you access your site, you can see affinity cookie being created as shown:
2019-06-01 14_58_27-DevTools - cspdev.isango.com_

Where, value you are seeing for ARRAffinity cookie is the ID of your instance.

I mentioned earlier that we have scaled out CD to two instances by default. So, whenever our CD Web App will be up it will have two instances in background. Now, if one of the instance has any issue, How do i make sure my request is being served through which instance? How do i forcefully connect to one of the instance? You can do this both by following below steps:

      • Get to know IDs of instances of your Web App. To get the IDs, go to App Service/Web App => Development Tools => Resource Explorer as shown:
        2019-06-01 15_19_04-Resource Explorer
        Where you can find everything from your resource group to particular Web App, and it’s certificates, instances, configs, deployments, diagnostics, functions, processes, slots, webjobs, siteextensions etc.

      • When you click on instances as shown in above image, it will list down all the instances available for the Web App at that time. We can consider name or siteInstanceName as ID of instance. So, now you should be able to note the ID and identify individual instances of your Web App.

      • Now when you access your site, you will be able to see the ARRAffinity cookie value which will be ID of one of the instance of your Web App. From this value you can identify which instance has an issue.

      • If you are stick to one instance and whenever you open browser, it’s sending request to same instance and you wanted to connect to another instance, just copy that instance ID from Resource Explorer and update the value of ARRAffinity cookie value and your next request will go to instance you have specified ID in cookie value.

 

What if ARRAffinity is disabled?

If your site is using distributed session state management like SQL or Redis, in this case you can disable ARRAffinity to take advantage of sharing load even more proper way. With this disabled, it’s not fixed which of the instances will serve your next request.

In this case you won’t find the ARRAffinity cookie being created in your browser. Meaning, you don’t have now the control if you want to check specific instance.

Sometimes, this is the best solutions and we might not required to know instances unless and until one of the instance has any issue. In this case to know from which instance this request is served, you can add custom header using environment variable WEBSITE_INSTANCE_ID. To know what are different kind of environment variable available in Azure Web App, visit: http://whatazurewebsiteenvironmentvariablesareavailable.azurewebsites.net/
screencapture-whatazurewebsiteenvironmentvariablesareavailable-azurewebsites-net-2019-06-01-15_46_48

To check all these configurations on your site, login to your azure account and access https://myapp.scm.azurewebsites.net/Env where myapp will be name of your app.

Note: For some reason you want to toggle ARRAffinity setting, Your App Service/Web App will get restart so as all your instances. So, make sure you only switch setting when you have some downtime window available.

How Sitecore’s Remote Events works with multiple instances?

If you’re not sure how remote event works, please see this video tutorial where i am explaining it in detail: https://www.youtube.com/watch?v=mg7V5c-tTzQ

For remote events to work properly on each CD instances/servers, they should have unique InstanceName. And if InstanceName is not specified, it uses [machinename]-[sitename] as a default value. As long as all of the CD’s are using a different machine name and/or sitename, then the EventQueue should be correct.

We left InstanceName empty and each of instances will be itself a separate VMs. All these instances will have different MachineName. SiteName will be same. So, value could look like:
Machine Name:    RD171548A9DCGC
SiteName:              myapp-uat-rg-cd

As far as one of above values are different remote event will work fine. And this is important as all your instances are separate entities and have their own CPU, RAM etc.

So, when you will see logs, you will find remote event entries with multiplication to number of instances on your CD Web App which tells that remote events are getting executed successfully on all instances.

Look this space for more on Sitecore with Azure PAAS.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s