Deploying Sitecore Commerce Engine using TeamCity and Octopus

We are seeing a lot of evolvement around DevOps for Sitecore. As part of that i have been working on many projects where we have implemented CI/CD along with some complexity like xConnect deployment, Deployment to Marketing Automation jobs, Blue-Green deployment. Currently, i am working on a Sitecore Commerce project. We are using 9.2 version of both Sitecore and Commerce. Where we had already implemented CI/CD for Sitecore solution. Now, we wanted to do it for the Commerce as well.

If you are not aware about Commerce basics like Role, Databases, Indexes etc. Please look at my previous blog here.

What’s the difference in Sitecore Commerce?

As i am saying we have already implemented CI/CD for Sitecore part. Then one can have a valid question that What’s the difference in commerce then. What are the changes in commerce? so that we need to implement it in different way. So, here are those mjor changes:

  • Roles:
    • Sitecore Commerce comes with by default 4 roles. Authoring, Shops, Ops, and Minions. Any change we made related to commerce should be deployed to all these roles.
  • .Net Core:
    • Sitecore commerce architecture is based on latest Microsoft technology which is .Net Core. While XP is traditional .Net with which we are used to.
    • In simple Sitecore solution, we generally follow Helix architecture in our most of the project. With Helix we end up with many web projects (Feature/Foundation/Project). We uses Octopack to generate packages and combining all into single package for octopus deployment. But Octopack does not work with .Net Core. So, either we need to use Octo.exe or simple Nuget Pack commands using .nuspec file to generate package.
    • One of the biggest change is we used to have all configurations in form of XML config files in Sitecore. While Commerce based on .Net Core is utilizing JSON to store the configurations. It do not have any XML config files. So, major drawback here is, we could not use Sitecore’s role based patching here. Same Commerce JSON configurations will be deployed to all 4 roles mentioned above. And we need role (Authoring/Ops/Shops/Minions) and environment (Dev/QA/UAT/Prod) specific values to get deployed.

We will see in this post, how we can overcome above issues and changes and still achieve CI/CD for Sitecore Commerce as well.

Visual Studio Structure

We could have two ways here:

  • Once solution with all things Sitecore and Commerce. And then at the build differentiate what’s for Sitecore and what’s for Commerce.
    VS solution - 1.png
  • Another way which we have taken here is to have separate solutions for Sitecore (which will contain all feature/foundation/project layer modules with Controllers/Repositories/Views/CSS/JS which makes the front end site like Storefront) and separate for Commerce (Which will contain Commerce Engine and all the required plugins).

You can find a Commerce Solution inside SDK folder which comes with Commerce installation packages. In real world, we will have actual plugins instead of sample plugins. See below:

SDK package
VS solution

What we are going to deploy?

  • We are going to deploy Sitecore.Commerce.Engine to engine roles.
  • Sitecore.Commerce.Engine will have references of all plugins which is required to get deployed.
  • All plugins won’t have any view files
  • We will deploy below files from wwwroot folder of engine project. Note: We will be deploying all files to all roles. It may be of no use for few roles but to keep everything in sync and how Sitecore installs in plain Sitecore.
    • config.json
    • bootstrap\Global.json
    • data\Environments\*

Deployment Approach

  • Standalone – In this approach, we will create a separate build (in teamcity) and deploy (in octopus) process. And we will add relevant steps required to deploy engine. This way we can control if we want to deploy the web or engine separately. We need to trigger two separate deployment from octopus for web and engine. With the VS solution structure defined above which is separate for Sitecore and Separate for Commerce, this becomes quite easy to achieve.
  • Mix – In this approach we can add steps to existing build (in teamcity) and deploy (in octopus) process. In this case you won’t have choice if you want to deploy only web or commerce. Both will get deployed in single life cycle.

We went ahead with Standalone approach due to isolation it gives. Where we won’t be touching CI/CD which is already implemented for Sitecore. We’ll setup it for Commerce.

TeamCity

These are the steps which will require to build Commerce solution:

  • Dotnet Restore: .NET Core (dotnet) – restores NuGet packages
  • Dotnet Publish: .NET Core (dotnet) – It will build and publishes project to specified output directory
  • Octo Pack – Command Line: executes ‘octo.exe’ command line tool with the following command octo pack --id %PackId% --version %PackVersion% --basepath %PackDirectoryPath% --format zip --outfolder %ArtifactsDirectoryPath%
    Or you can also use simple Nuget Pack command. We used here Octo pack to make it consistent with Sitecore solution as we are using Octopack there to generate package.
  • Octo Push – OctopusDeploy: Push packages – pushes packaged files to Octopus built-in package repository
  • Octo Create Release – OctopusDeploy: Create release – creates release in Octopus with desired packages included in it.

Now, we are good with build and we have generated package and release in Octopus which is ready for deployment to commerce engine roles. Let’s see what all steps we will have to add to Octopus.

Octopus

  • Stop Site/App Pool for Authoring role.
  • Deploy Sitecore.Commerce.Engine package which we generated using TeamCity to the Authoring role.
  • Start Site/App Pool for Authoring role.
  • Check https status for Authoring role if site is up and running before moving ahead. Check for URL https://localhost:5000/api/$metadata which should return 200 status code. URL might defer based on environment and hosts you used.
  •  Repeat all steps above for remaining roles Shops, Minions, and Ops.
  • At the end of all above steps we can add optional step to call bootstrap the environment. For that

We are all done with CI and CD. Wait, biggest challenge is pending. To transform values of JSON configuration files as per role + environment. And it should also work in developer box + higher environments when deployed.

Transforming JSON Configurations

This was the biggest challenge in setting up CI/CD for Sitecore Commerce. We had to make sure after including all JSON files into solution, when we deploy that to local instances in developer machines, it should work. We went through few of the options for this.

Approach 1: Environment specific JSON config files

You probably know web.config transformations available in ASP.NET. Using a transform XML file, you can replace, insert, remove settings from any XML file. Transformations is primarily used to have sets of variables for different environments like localhost, staging and production. Since Core no longer use web.config for application settings, web.config transformations no longer apply. Luckily, Core introduces a similar concept for its JSON configuration files.

  • You might have seen config.json and a nested file config.Development.json in your solution. Here, whatever values you need to overwrite, you will specify in environment specific JSON files.
    config-json
    This is the content in config.Development.json
    config-dev-json
  • How this works? In .Net Core, there is launchsettings.json config file available. You can define project specific settings associated with each profile Visual Studio is configured to launch the application, including any environment variables that should be used.
    If you expand Properties folder in your Sitecore.Commerce.Engine project, you will see below content in launchsettings.json file.
    launchsettings
  • When we debug commerce engine, we generally uses Engine profile. Which you can see in above image. And also notice ASPNETCORE_ENVIRONMENT value defined as Development
    profile
  • From Program.cs file, you will get to see code which loads config.json confiurations and then overriding values by loading environment specific JSON file. And while running with engine, environment will be Development (Note: Default value when hosted to IIS is Production for environment which you can override using ASPNETCORE_ENVIRONMENT variable in web.config). So, it will override values from config.Development.json file.
    program-cs

So, it looked like we can create environment specific files and can manage transform values. But it looks like this is not working for any other JSON files then config.json. And we have many policy files inside Environments folder which needs transform. So, this approach didn’t worked for us. So, we went ahead with approach mentioned below.

Approach 2: JSON transform feature of Octopus

For XML config,
Transformaion happens through Octopus (or Azure DevOps), we just specify the variable in our configs. And value of those variable according to roles/environments are specified in Octopus. Which will perform transformation when deploying package to role.

For JSON config,
Octopus also supports JSON transformation. One need to enable JSON variable substitution feature in package deployment step. One advantage over XML trnsform here is, instead of specifying variables in JSON files. We can use JSON transform feature of Octopus. Where we will have to specify variable in octopus based on hierarchy and name of property in JSON. With this we are not replacing variable with value but specific element’s value with actual value. In this case we won’t need to manage two copies. As config will have values which will work for local and for higher environments those values can be managed through Octopus.

You can find the syntax Octopus supports for JSON transformation https://octopus.com/docs/deployment-process/configuration-features/json-configuration-variables-feature. Including hierarchical and array based replacement.

Now we know how transformation works. Next step is

  • To identify all such values from all the JSON config files
  • Compare JSON files from all roles in each environments with the copy you’re going to commit and which will get deployed.
  • And based on comparison, you need to prepare a list of variables and values according to role/environment and specify same in Octopus.
  • That’s it. Deploy and verify the values are proper in each environment.

You can find below few JSON config files identified which values you might have to change according to your setup.

config.json – \wwwroot\

Variable Dev
Authoring Shops Minions Ops
AppSettings:EnvironmentName HabitatAuthoring HabitatShops HabitatMinions HabitatOps
AppSettings:SiteTitle dev-ce-authoring.HI-us.com dev-ce-shops.HI-us.com dev-ce-minions.HI-us.com dev-ce-ops.HI-us.com
AppSettings:AllowedOrigins [
https://dev-ce-bizfx.HI-us.com:4200“,
https://dev-sc-admin.HI-us.com
]
AppSettings:CommerceServicesHostPostfix HI-us.com
AppSettings:SitecoreIdentityServerUrl https://dev-identity.HI-us.com
AppSettings:UseHttpsInKestrel False
AppSettings:AntiForgeryEnabled False true true true
Certificates:Certificates [
{
“Thumbprint”: “BSE1SB5870341991E581F954FCD0F638A57F3990”,
“DefaultRoles”: [
“sitecore\\QA”,
“sitecore\\Commerce Business User”,
“sitecore\\Commerce DevOps User”
]
}
]
Caching:Memory:Enabled True
Caching:Redis:Enabled False
Caching:Redis:Options:Configuration dev.redis.SC.local
Logging:LogLevel:Default Warning Warning Warning Warning
Serilog:MinimumLevel:Default Warning Warning Warning Warning
Logging:PipelineTraceLoggingEnabled False false false false

Global.json – \wwwroot\bootstrap\

Variable Dev
Authoring Shops Minions Ops
Policies:$values:2 {
“$type”: “Sitecore.Commerce.Plugin.SQL.EntityStoreSqlPolicy, Sitecore.Commerce.Plugin.SQL”,
“PolicyId”: “Global”,
“AllowAdmin”: true,
“Database”: “SC-ce_Global”,
“EffectiveDate”: “0001-01-01T00:00:00”,
“IsVisible”: false,
“Password”: “b”,
“Server”: “.\”,
“TrustedConnection”: true,
“UserName”: “SQL\\devuser”,
“View”: “EntityStoreSqlPolicy”,
“Version”: “9.2.0”
}
Policies:$values:6 {
“$type”: “Sitecore.Commerce.Plugin.Management.SitecoreConnectionPolicy, Sitecore.Commerce.Plugin.Management”,
“Host”: “dev-sc-admin.HI-us.com“,
“SitecoreDatabase”: “master”,
“UserName”: “admin”,
“Domain”: “sitecore”,
“Password”: “b”,
“ConnectionLeaseTimeout”: 60000,
“AuthRequestUrl”: “sitecore/api/ssc/auth/login”
}

PlugIn.Content.PolicySet-1.0.0.json – \wwwroot\data\Environments\

Variable Dev
Authoring Shops Minions Ops
Policies:$values:0 {
“$type”: “Sitecore.Commerce.Plugin.Management.SitecoreConnectionPolicy, Sitecore.Commerce.Plugin.Management”,
“Host”: “dev-sc-admin.HI-us.com“,
“SitecoreDatabase”: “master”,
“UserName”: “admin”,
“Domain”: “sitecore”,
“Password”: “b”,
“ConnectionLeaseTimeout”: 60000,
“AuthRequestUrl”: “sitecore/api/ssc/auth/login”
}

Plugin.Search.PolicySet-1.0.0.json – \wwwroot\data\Environments\

Variable Dev
Authoring Shops Minions Ops
Policies:$values:0:SearchScopeName sitecore_OrdersScope
Policies:$values:1:SearchScopeName sitecore_CustomersScope
Policies:$values:2:SearchScopeName sitecore_CatalogItemsScope
Policies:$values:3:Name sitecore_CatalogItemsScope
Policies:$values:4:Name sitecore_CustomersScope
Policies:$values:5:Name sitecore_OrdersScope
Policies:$values:6:SearchScopeName sitecore_CatalogItemsScope
Policies:$values:7:SearchScopeName sitecore_CustomersScope
Policies:$values:8:SearchScopeName sitecore_OrdersScope

Plugin.SQL.PolicySet-1.0.0.json – \wwwroot\data\Environments\

Variable Dev
Authoring Shops Minions Ops
Policies:$values:0:Server Dev SQL Server
Policies:$values:0:TrustedConnection True
Policies:$values:0:UserName SQL\\cecmdevuser
Policies:$values:0:Password b
Policies:$values:0:Database SC-ce_SharedEnvironments

PlugIn.Search.Solr.PolicySet-1.0.0.json – \wwwroot\data\Environments\

Variable Dev
Authoring Shops Minions Ops
Policies:$values:0:SolrUrl https://solr:8983/solr

As per example above, you can use hierarchy to replace any property value, you can also replace whole array, you can also replace a specific element of array, and further you can also replace a property based on hierarchy of specific element in array.

Note: All these exercise has been done on Sitecore and Commerce 9.2. For other versions, JSON configs and hierarchy may vary.

Keep Automating!

Thanks,

2 thoughts on “Deploying Sitecore Commerce Engine using TeamCity and Octopus

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s