Debugging Sitecore Commerce Engine

While working on Sitecore commerce project, at many point of time we would like to debug the commerce part of the application similar to as we used to debug the Sitecore solution. To be able to debug the Sitecore commerce engine, there are couple of key concepts to understand. Which will help you understand the connection and debug commerce engine effectively.

If you are not much familiar with Commerce Engine, please see Getting started with Sitecore Commerce

I have used Sitecore Commerce 9.2 version to perform all these debugging steps.

Authentication

To debug commerce engine, you fist need to know how authentication works for different systems like Business Tools, Storefront, Content Editor, Postman etc.

Continue reading “Debugging Sitecore Commerce Engine”

Reporting database rebuild issue in Sitecore 9.2 scaled environment

We all must have gone through issue where analytics dashboard not showing latest reports. Recently, we were facing the same issue on one of our environment which is on-prem scaled instance of Sitecore 9.2 version.

We checked all different possible issues like 1) xConnect models not deployed to xConnect role 2) Marketing Automation service getting stopped due to credentials or certificate issue 3) Processing role getting stopped after ideal timeout.

We fixed all these issues and we were able to see the latest data in analytics dashboard. But we were not able to see past data as they were collected but not processed.

So, we wanted to do a reporting database rebuild as mentioned in https://doc.sitecore.com/developers/90/sitecore-experience-platform/en/rebuild-the-reporting-database.html/ We performed all steps mentioned in this document. Now it was time to launch below URL on ContentManagement role and execute the rebuild by providing date range if required.

/sitecore/admin/RebuildReportingDB.aspx

We faced below error when launching above URL:

Reporting

As per https://community.sitecore.net/developers/f/9/t/5594, if you’re having scaled environment where you have different ContentManagement and different Reporting role, you need to have Sitecore.Xdb.Remote.Client.config enabled and configured reporting server URL & schema properly.

We double checked and everything was specified for attributes serviceUrl, requireHttps, connectionStringName, headerName etc. And all patching was getting applied properly as well. As per issue we tried to search serviceHostUrl in config files but we couldn’t found single instance.

xdb-remote-client

We then realized that somehow parameter for reporting URL is misspelled in Sitecore.Xdb.Remote.Client.config. And in scaled version, it uses the API to call reporting rebuild on Reporting role. By changing param desc from serviceUrl to serviceHostUrl on ContentManagement role fixed our issue and we were able to browse rebuildreportingDB.aspx page.

Deploying Sitecore Commerce Engine using TeamCity and Octopus

We are seeing a lot of evolvement around DevOps for Sitecore. As part of that i have been working on many projects where we have implemented CI/CD along with some complexity like xConnect deployment, Deployment to Marketing Automation jobs, Blue-Green deployment. Currently, i am working on a Sitecore Commerce project. We are using 9.2 version of both Sitecore and Commerce. Where we had already implemented CI/CD for Sitecore solution. Now, we wanted to do it for the Commerce as well.

If you are not aware about Commerce basics like Role, Databases, Indexes etc. Please look at my previous blog here.

Continue reading “Deploying Sitecore Commerce Engine using TeamCity and Octopus”

Getting Started with Sitecore Commerce

Recently, I got a chance to look at Sitecore Commerce. So, i will be sharing my understanding after self learning and looking into real world project of commerce for few weeks. I hope this will be helpful for those wanted to get started on commerce.

Getting Started with Commerce

Roles

Along with standard XM/XP roles, there are few other roles created with commerce installation. They are:

  • SitecoreBizFx: Sitecore Business Tools – In simple words, admin panel for managing all features like Customers, Prices and Promotions, Inventory, Orders etc. It’s data driven applications built on Angular and Knockout. You will find config.json in SitecoreBizFx role which will have important configurations to connect to identity server and commerce engine.
    2019-08-11 14_25_13-NEW-LIXIL-VM (Lixil - Sc9.0.2 installed) [Running] - Oracle VM VirtualBox _ 1
  • Engine Roles
    There are different kind of commerce engine roles being created when you install commerce. They share many similarities thus it’s important to understand what each role is responsible for.

    • Ops: The Ops role is an instance of the Commerce Engine that is internal and only available to DevOps personnel. This role can have an identity with higher privileges allowing DevOps personnel to perform maintenance tasks that are not permitted to other roles (e.g., bootstrapping and environment initialization functions).
      Untitled
      For more information on setting up postman for Sitecore commerce API calls:

    • Authoring: The Authoring role is the instance of the Commerce Engine that serves traffic from the Commerce business tools. For ex: Merchandising experience for managing Pricing and Promotions, CSR Experience for managing  Customers and Orders. Since this role serves lighter traffic (i.e., eCommerce solutions have relatively few business users compared to the number of shoppers), scale requirements are normally relatively low. As this role is being used by business tools means SitecoreBizFx role as mentioned above, you will find “EngineUri”: “https://localhost:5000” configuration in config.json of SitecoreBizFx role.
      2019-08-11 14_47_32-NEW-LIXIL-VM (Lixil - Sc9.0.2 installed) [Running] - Oracle VM VirtualBox _ 1
    • Shops: The Shops role is the instance of the Commerce Engine that serves traffic from one or more storefronts. This role is intended to scale to support demand, and is usually installed in close proximity to the Sitecore Experience Platform instances that generate the traffic. To scale the solution, the Sitecore XP instances and/or the Commerce Engine instances can be scaled depending on traffic mix and the location of any bottlenecks. Shops role which is connected to CD role is generally responsible for services like Shopping cart services, Order services, Customer Self Service etc.
      2019-08-11 14_58_11-NEW-LIXIL-VM (Lixil - Sc9.0.2 installed) [Running] - Oracle VM VirtualBox _ 1
      You can see the configuration in your web role from config \App_Config\Include\Y.Commerce.Engine\Sitecore.Commerce.Engine.Connect.config.
      2019-08-11 15_30_02-NEW-LIXIL-VM (Lixil - Sc9.0.2 installed) [Running] - Oracle VM VirtualBox _ 1
      But we couldn’t find https://localhost:5005/ which is shop role in any of configuration in standard installation (also mentioned in https://hachweb.wordpress.com/2018/08/06/sitecore-xc-9-0-2-commerce-environments-behind-the-scenes/).
      Scaled: For scaled XP environment where you will have separate CM and CD, shopServiceUrl will point to authoring role in CM while in CD it will be pointed to shops role.
      You can find scripts written by our community members for scaled XC installation here.
      Deploy-Sitecore-Commerce-SitecorePackagesCD.ps1
      2019-08-11 15_36_34-Sitecore-Commerce-v902-Scaled-Installation_Deploy-Sitecore-Commerce-SitecorePack
    • Minions: The Minions role is an instance of the Commerce Engine that runs independently and supports asynchronous processing. This includes any post-order capture processing as well as any cleanup or pruning.
      2019-08-11 15_44_30-Microsoft Edge
      Minion role is not connected directly to any of role like Web, Authoring, Shops etc. It is just connected to commerce database directly.

Storefront

SXA Storefront is a framework that enables you to create a shop experience easily and quickly. SXA Storefront is built on top of Sitecore Experience Accelerator along with many commerce specific components.

It uses all core feature of SXA like page designs, partial designs, renderings, renderings variants etc.

When you install commerce, by default it installs a storefront site for you. And if you want to create it yourself the you can create it same as normal SXA site by clicking on tenant and adding new site through create site dialog.

Modules
2019-08-11 17_05_15-NEW-LIXIL-VM (Lixil - Sc9.0.2 installed) [Running] - Oracle VM VirtualBox _ 1
Here, you can find three modules specific to commerce.
Commerce Storefront Components:  provides the SXA Toolbox renderings and Sitecore Commerce Control Panel. With this checked, you will see additional commerce specific renderings in toolbox:
15d35bc616ed73
To understand more about each sections and renderings into each sections visit Commerce Account renderings, Commerce Cart renderings, Commerce Catalog renderings, Commerce Checkout renderings, Commerce Orders renderings, Commerce Shared renderings, and Commerce meta renderings
Commerce Storefront Template: provides storefront templates and pages. For ex: Shopping cart, Checkout, Account pages etc. And also it comes with default partial designs and page designs for different kind of pages.
page designs

Page design and partial design relation
page-partial designs
Commerce Storefront Habitat Catalog: provides a sample catalog with categories and sellable items.

Note: There are few changes related to catalog since Commerce 9.0.2. They are:

  • The default Catalog and Start Navigation Category are now configured from under “{site}/settings/commerce/catalog configuration”. This is in order to avoid references from the Global Control Panel settings to specific sites after the catalog content has been moved under the specific sites where it belongs. The old location under Commerce Control Panel is still supported to provide backward compatibility.
  • Catalogs are now incorporated into the content area of the storefront sites and reside under a new “{tenant}/{site}/{home}/catalogs” item where it’s used and belongs. This fixes multiple issues including:
    1. Limited access to the catalog for non-admin Sitecore users.
    2. The same products appearing in multiple locations in content with the same item ID and path, causing problems in different places.
    3. Not being able to set and store Sitecore specific settings and data like personalization, presentation and workflow on catalog items and have it persisted correctly and uniquely.

Theme
2019-08-11 17_29_47-NEW-LIXIL-VM (Lixil - Sc9.0.2 installed) [Running] - Oracle VM VirtualBox _ 1

While creating a new site, we can create our own theme for commerce storefront. Storefront Branded is a sample storefront theme which you can use as an example or base when creating theme for your newly storefront.

Commerce Base Themes: Prior to the 9.0.3 release of Sitecore, JavaScript for commerce functionality was loaded using the AssetService pipeline (processor Sitecore.Comerce.XA.Foundation.Common.Pipelines.AssetService.AddJavascript).

To improve performance and multi-site extensibility, all JavaScripts supporting commerce business functionality have been moved from files on disk to the sitecore/Media Library/Base Themes folder.
15d4d5317092e9

  • Commerce Core Libraries: This theme contains the third-party libraries used by Storefront components (such as KnockoutJS, jQuery, Bootstrap, and so on) Knockout is the main framework used in SXA Storefront for data binding.
  • Commerce Services Theme: This theme contains the shared services used by all Storefront components (such as the Ajax service used to handle the Ajax calls to the API). This theme also contains the communication and messaging logic between the components that follow the publisher-subscriber pattern.
  • Commerce Core Theme: This is the Foundation layer that contains the common JS logic used by all the components. It contains the application logic, component definition, and initialization.
  • Commerce Main Theme: This theme contains the main.js file that represents the entry point to the Storefront front-end application initialization.
  • Commerce Components Theme: This theme represents the Feature layer and it contains all the component-specific logic. Each Storefront component has its own JavaScript file named after the component that handles its initialization, data bindings, and some of the interactions between the user and the component.

Plugin

The Commerce Engine delivers commerce functionality through an extensible, pluggable framework. A plugin is an independently publishable extension to the Commerce Engine.

Your commerce solution with plugin may look something like:
2019-08-11 23_50_20-Training - Microsoft Visual Studio

Sitecore.Commerce.Engine: This is the main project of your commerce solution built on .Net Core framework. All the Plugins being created and added into solution will be referenced into this project. All projects from Feature.SDK folder are plugins. Sitecore.Commerce.Engine project also contains different JSON configurations and Policy files. Sitecore.Commerce.Engine project gets deployed to all commerce engine roles mentioned earlier in this post.

There are already few good posts written by community members. You can go through those to understand more about plugin architecture.

Click here to know more about Plugin

Click here for instructions on creating your first Plugin

Click here for getting the customer sample solution up and running

Click here to understand all entities which comes together and forms a complete Plugin

Click here to understand how to remove sample plugins from customer sample solutions and start with fresh solution for your project

Service Proxy: As we know now, data is being exchanged using OData protocol. And as per Helix guidelines and Commerce architecture, we created plugin and it’s endpoints with implementation in .Net Core project while we creates normal MVC project for our feature. Now, if we need to reference such endpoints and models which we created into plugin in feature project, we cannot reference .Net Core project into feature. That is where Proxy comes into the picture.

Click here to know more about what is service proxy

Click here to understand how service proxy works

Click here to know how to generate service proxy with latest OData package

 

 

 

Enabling reverse proxy in Sitecore on Azure PaaS and gotchas

In Sitecore or any .Net application, you can easily achieve Reverse Proxy with the help of Url Rewrite and Application Request Routing (ARR) as mentioned in https://docs.microsoft.com/en-us/iis/extensions/url-rewrite-module/reverse-proxy-with-url-rewrite-v2-and-application-request-routing.

There are some nice blogs and answers available to achieve same in Sitecore.

Recently, we were required to achieve exact same thing. But the tweak was our instance is on Azure PaaS. So, It’s clear from the configuration in Sitecore’s perspective. But one can have questions like Do we need to install ARR on WebApp? How do we enable Reverse Proxy setting on ARR without IIS access? We were having the same access. Let’s answer these questions.

Continue reading “Enabling reverse proxy in Sitecore on Azure PaaS and gotchas”

Troubleshooting Sitecore on Azure PAAS – Part 3

Scaling

Today we are going to discuss the Scaling feature of Azure and in particular Scale out and how you can troubleshoot your Sitecore instances. If you haven’t gone through earlier posts on troubleshooting Sitecore on Azure PAAS, do visit https://pratiksatikunvar.wordpress.com/category/sitecore/azure-paas/.

One of the reason enterprise level organizations wants their site in Azure PAAS is the ability to scale at anytime with little to no efforts. Azure provides great scaling options. There are two main ways that an application can scale:

  • Vertical scaling, also called scaling up and down, means changing the capacity of a resource. For example, you could move an application to a larger VM size. Vertical scaling often requires making the system temporarily unavailable while it is being redeployed. Therefore, it’s less common to automate vertical scaling.

  • Horizontal scaling, also called scaling out and in, means adding or removing instances of a resource. The application continues running without interruption as new resources are provisioned. When the provisioning process is complete, the solution is deployed on these additional resources. If demand drops, the additional resources can be shut down cleanly and deallocated.

Both kind of scaling can be done automatic or manually. Automatic scaling can be done based on Resource Metrics, Custom Metrics, Time, Metric/time based Rules, Default Scaling.

Continue reading “Troubleshooting Sitecore on Azure PAAS – Part 3”

Solr, SolrCloud, and Solr as a Service

Currently i am working on a project where we started with setting up Solr on VM on premise. Then we moved to Solr as a Service provider which was using Solr in background, and at the end we moved to another Solr as a Service provider which was using SolrCloud in background. All of these terminologies were difficult to understand and it even becomes more complex when you introduce SwitchOnRebuild and Blue/Green indexes.

In this post, we will understand basics of Solr, SolrCloud, and Solr as a Service along with SwitchOnRebuild configurations.

Solr

Solr is an open-source enterprise-search platform, written in Java, from the Apache Lucene project. It provides both a RESTful XML interface, and a JSON API with which search applications can be built.

Solr comes with rich feature sets like:

  • Advanced Full-Text Search Capabilities
  • Optimized for High Volume Traffic
  • Standards Based Open Interfaces – XML, JSON and HTTP
  • Comprehensive Administration Interfaces
  • Easy Monitoring
  • Flexible and Adaptable with easy configuration
  • Near Real-Time Indexing
  • Extensible Plugin Architecture

Sitecore has come up with native support of Solr along with Lucene as search provider. Please see compatibility table below:

2019-05-24 13_20_29-Solr compatibility table - Sitecore Knowledge Base

You can setup the Solr by following https://doc.sitecore.com/developers/82/sitecore-experience-platform/en/walkthrough–setting-up-solr.html. With Sitecore 9, Solr comes as default search provider so you do not need to worry about configuring Sitecore to work with Solr.

You can also use the script by @jermdavis to download and setup Solr as a windows service – https://gist.github.com/jermdavis/8d8a79f680505f1074153f02f70b9105.

SwitchOnRebuild with Solr

You can set up Solr to rebuild an index in a separate core so that the rebuilding does not affect the search index that is currently used. Once the rebuilding and the optimization of the index completes, Sitecore switches the two cores, and the rebuilt and optimized index is used.

We mostly uses master/web/core indexes in our application to index and fetch data and use that in component to show content on page.

To configure SwitchOnRebuild with Solr, follow below steps:

  • From the Solr server, copy the existing sitecore_master_index folder. Call the copy sitecore_master_index_rebuild
  • Update the sitecore_master_index_rebuild/core.properties file and set name of the new core:name=sitecore_master_index_rebuild
  • Restart Solr and check both primary and secondary cores.
  • To use this implementation, change the type reference on a particular search index to Sitecore.ContentSearch.SolrProvider.SwitchOnRebuildSolrSearchIndex, and add the rebuildcore parameter. You can use below patch file for the same:

Note: After you have changed the configuration file, your website uses indexes from the primary core. Each time you initiate a full index rebuild, Sitecore does this in the secondary core. The secondary core then becomes the primary one after the rebuild.

You can copy above patch file on any role without any change. You will have to disable all index strategies except manual on CD server. Please see: https://sitecore.stackexchange.com/questions/3918/patching-to-remove-index-update-strategies.

Though Solr provides great features, it’s sometimes difficult to achieve high availability for the production environment using Solr. SolrCoud comes into the picture as a feature of scaling Solr.

SolrCloud

So, it does not mean if we have Solr configured on cloud it means SolrCloud.

Apache Solr includes the ability to set up a cluster of Solr servers that combines fault tolerance and high availability. Called SolrCloud, SolrCloud is designed to provide a highly available, fault tolerant environment for distributing your indexed content and query requests across multiple servers. These capabilities provide distributed indexing and search capabilities, supporting the following features:

  • Central configuration for the entire cluster
  • Automatic load balancing and fail-over for queries
  • ZooKeeper integration for cluster coordination and configuration.

Key SolrCloud Concepts

A SolrCloud cluster consists of some “logical” concepts layered on top of some “physical” concepts.

Logical Concepts
  • A Cluster can host multiple Collections of Solr Documents.
  • A collection can be partitioned into multiple Shards, which contain a subset of the Documents in the Collection.
  • The number of Shards that a Collection has determines:
    • The theoretical limit to the number of Documents that Collection can reasonably contain.
    • The amount of parallelization that is possible for an individual search request.
Physical Concepts
  • A Cluster is made up of one or more Solr Nodes, which are running instances of the Solr server process.
  • Each Node can host multiple Cores.
  • Each Core in a Cluster is a physical Replica for a logical Shard.
  • Every Replica uses the same configuration specified for the Collection that it is a part of.
  • The number of Replicas that each Shard has determines:
    • The level of redundancy built into the Collection and how fault tolerant the Cluster can be in the event that some Nodes become unavailable.
    • The theoretical limit in the number concurrent search requests that can be processed under heavy load.

Please find below compatibility table of SolrCloud with Sitecore.

2019-05-24 14_35_39-Solr compatibility table - Sitecore Knowledge Base

You can configure SolrCloud by following https://doc.sitecore.com/developers/90/platform-administration-and-architecture/en/walkthrough–setting-up-solrcloud.html or https://sitecore-community.github.io/docs/search/solr/Install-and-configure-SolrCloud/.

Even though SolrCloud provides fault tolerance and high availability, we will still require to setup SwitchOnRebuild to make your site up and running during index rebuild.

SwitchOnRebuild with SolrCloud

SwitchOnRebuild configurations seems simple as we saw with Solr above. But when it comes to SolrCloud, it’s somewhat confusing. This implementation uses Solr aliases instead of collection names.

The mechanism for maintaining and switching two indexes is different when use SolrCloud. The implementation in the SwitchOnRebuildSolrCloudSearchIndex class uses collection aliases: it uses the active alias for search and update operations and the rebuild alias for rebuild operations. When a rebuild operation finishes, the CREATEALIAS command swaps the collections the aliases reference.

You need to create two collections, one is primary and another is Secondary. And also you need to create alias for both collections respectively.

We need to follow these guidelines when we configure Sitecore to use SolrCloud:

  • Only use the SwitchOnRebuildSolrCloudSearchIndex index type in combination with an active index update strategy (index has at least one index update strategy other than manual).
  • Only one Sitecore instance can use the SwitchOnRebuildSolrCloudSearchIndex index type for a particular search index. All other Sitecore instances must use the SolrSearchIndex type, and also use the main alias as the core parameter:
    <index id="sitecore_web_index" type="Sitecore.ContentSearch.SolrProvider.SolrSearchIndex, Sitecore.ContentSearch.SolrProvider">
      <param desc="core">$(id)MainAlias</param>

It is likely to miss these notes which will cause SwitchOnRebuild to not work properly with SolrCloud.

Sitecore also provides an example config file for SwitchOnRebuild with SolrCloud. If you haven’t checked it. Please see \App_Config\Include\Examples
\Sitecore.ContentSearch.SolrCloud.SwitchOnRebuild.config.example
file.

Also, It’s not mentioned what to do with other indexes where SwitchOnRebuild is not required like sitecore_marketingdefinitions_web. Weather we need to use alias name or not. Please see below configuration file which takes care of configuration changes required per role and other required considerations.

Solr as a Service

So, Solr as a service does not mean it offers SolrCloud only, it can be Solr too.

If you do not have in-house Solr expertise and you don’t want your developers to focus on managing, maintaining, and monitoring search infrastructure, you can use Solr as a service. There are few providers available who provides solr as a service. Few of them are:

Opensolr:

https://opensolr.com – It provides GUI from where you need to choose region and Solr version and with few clicks, your node will be up and running.

Note: Opensolr uses Solr as back-end implementation and not SolrCloud. So, if you are setting up solr using Opensolr, you need to keep this in mind and implement all configurations mentioned in Solr section.

What types of Solr Cloud Hosting Opensolr provide?
https://opensolr.freshdesk.com/support/solutions/articles/33000202124-what-types-of-solr-cloud-hosting-do-you-provide

SearchStax:

https://www.searchstax.com – It is fully managed Solr as a Service that allows you to spend more time building your search application and less time on managing search infrastructure. It also provides full GUI and complete control which we generally have for Solr in developer machine.

Note: SearchStax uses SolrCloud as back-end implementation mostly though it also supports Solr. So, if you are setting up solr using SearchStax, you need to keep in mind and implement configurations mentioned in SolrCloud section. Especially, SwitchOnRebuild.

What types of Solr Cloud Hosting SearchStax provide?
SearchStax’s founder @maggon took part in SUGCON 2019 recently held in Bengaluru, India. You can go through slide World class Solr power in 30 minutes which may answer all your questions.