Sitecore xDB Cloud 2.0 – Using RestAPI for xDB Cloud Service to solve many purpose

In previous posts, we discussed what is xDB Cloud, Advantages/Disadvantages, Useful Terminologies, How to configure etc. In disadvantages, we seen that we could not connect xDB Cloud directly using tool like Robomongo and MongoVUE. Limiting the action we can perform on MongoDB over On-Premise setup. To overcome this, Sitecore has come up with xDB Cloud RestAPI, not strong enough but maturing steadily. Using which you can perform different operations.

Let’s understand each methods with an example. I use Postman for such purpose. It is a tool build on top of Chrome with lots of cool feature.

You must use a valid authentication token whenever you make a call to the xDB Cloud API. Which can be generated by using a valid Sitecore license file to call the SSO Encode Sitecore License endpoint. You must include the generated token as a HTTP header in all other requests called X-ScS-Nexus-Auth

https://gateway-sso-scs.cloud.sitecore.net/api/License/Encode/

E2k5QvJ

This is the first step you need to perform as all other endpoint requires X-ScS-Nexus-Auth for authentication.

Once you have obtained the token, first thing you wish want to get is list of xDB sets. In response you will get all xDB set listed and DeploymentId for each. Once you have noted LicenseId and DeploymentId, you will be able to make API calls to get information specific to xDB set (DeploymentId). This endpoint becomes very handy when you haven’t received DeploymentId(s) from Sitecore Support.

https://gateway-xdb-scs.cloud.sitecore.net/api/xdb/{licenseId}

uXEw7IJ

Once you have obtained list of xDB sets, you will wish to get the Connection Strings for one of the xDB set (DeploymentId) as it is a vital part in configuring the xDB Cloud. This endpoint becomes very useful when you haven’t received any information about xDB Cloud like Connection Strings, ReportingService from Sitecore Support. With this method in place, you will be able to get all the information required for your setup.

https://gateway-xdb-scs.cloud.sitecore.net/api/xdb/connectionstrings/{licenseId}/{deploymentId}

FXVAO8G

This is also one of the important thing to successfully complete the xDB Cloud setup. Even though you have done all configuration right, but your infrastructure not configured properly, it won’t allow application to connect to MongoDB. Hence, no data will be written to xDB Cloud. Use this endpoint to get all URLs and ports specific to URL. And you have to enable the outbound connections over given ports in Firewall.

https://gateway-xdb-scs.cloud.sitecore.net/api/xdb/firewallsettings/{licenseId}/{deploymentId}?v=2

BUjwk2w

You can utilize this endpoint as a tool to verify weather data is getting stored in xDB Cloud or not. By providing year and month in endpoint URL, you will get informations like Total Interactions, Interactions Added, Total Contacts, Contacts Added etc. per day basis.

https://gateway-xdb-scs.cloud.sitecore.net/api/xdb/Consumption/{licenseId}/{deploymentId}/{year}/{month}

WFn5eJ8

This endpoint also helps like above one to verify weather data is getting stored to xDB Cloud or not by retrieving collections. Collections includes Interactions as well Contacts.

https://gateway-xdb-scs.cloud.sitecore.net/api/xdb/collection/{licenseId}/{deploymentId}

J3p47VZ

This endpoint is useful to get information about xDB set (DeploymentId) such as Sitecore Version (Not sure why it is there), xDB Cloud Version (1.0/2.0), And Deployment Type (Prod/Non Prod).

https://gateway-xdb-scs.cloud.sitecore.net/api/xdb/version/{licenseId}/{deploymentId}

5hVNcsw

This endpoint will be used to ensure xDB Set (DeploymentId) working well or not. This will return Status, Message, and IssueIdOnError. You will find the message if there is any issue with xDB Set. Make a note of IssueIdOnError if you get that in response because you need to provide it to Sitecore Support team when you contact them about the issue with your xDB Set.

https://gateway-xdb-scs.cloud.sitecore.net/api/xdb/{licenseId}/{deploymentId}/status

cVGxtqE

This is a very useful endpoint which will be used to rebuild the Reporting database. Make note that you won’t be able to rebuild the reporting database using admin page in case of xDB Cloud. This is the method which will do the work for you. When calling this method, it should return “In Process” as response. That means rebuild of reporting database for that specific xDB Set has been started.

https://gateway-xdb-scs.cloud.sitecore.net/api/xdb/historyProcessing/{licenseId}/{deploymentId}

DP1Z09E

Get history processing status

Once you have triggered the rebuild Reporting database with help of above method and received response as “In Progress”, you will want to know the status of that task to determine weather it got completed successfully or encountered with an error. By triggering this same API as above again, you will get status of that operation in response. Possible values are Done, In Process, Error, and None.

https://gateway-xdb-scs.cloud.sitecore.net/api/xdb/historyProcessing/{licenseId}/{deploymentId}

L94HZxR

So, that’s it from the xDB Cloud 2.0 series. Aim was to provide everything at a single point from understanding to Configuration, troubleshooting etc. so that it can save a couple of days for someone.

Enjoy xDB Cloud.

Sitecore xDB Cloud 2.0 – Configuration, Connection, and Troubleshooting

In earlier, blog posts we discussed the basics of xDB Cloud, some of the advantages and disadvantages of using xDB Cloud as well some useful terminologies. Now we will see how to configure xDB Cloud for various environments. Consider below versions of the resources for our exercise:

  • Sitecore Experience Platform 8.1 rev. 160519 (8.1 Update-3)
  • xDB Cloud 2.o
  • Sitecore xDB Cloud 8.1 rev. 160721

Configuration

  • Get xDB subscription by getting in touch with Sitecore. Make sure it is attached with your license id. If you make any change to your licensing than reassure it is still attached with the xDB Cloud. Consider the usage and requirement of business subscribe a number of xDB Cloud database set required.

  • Once you have subscribed for xDB Cloud, To request an xDB Cloud 2.0 customer set, You required below things:

    • Your License ID (11111111111111). Get it from the license file. Sitecore 8.1 now requires a license with the “Sitecore.xDB.base” key to enable all features of the Experience Platform. If your license file does not contain this key, Sitecore will default to Experience Management (CMS-only) mode. Any customers or partners with a license to Experience Platform should contact their account manager or login to SPN if they are missing this key.
    • Your Deployment Id (COMPANYPROD81). Get it based on a role of the instance like Dev, Stag, Prod etc. If you do not receive it from Sitecore support, get it using Rest API.
    • The version of Sitecore that you are running (8.1 rev. 160519).
    • The preferred location for your xDB Cloud environment. See the xDB Cloud Service compatibility tables for compatible data center locations. Though, you don’t required it to specify anywhere.
    • MongoDB connection strings:
      analytics, live, tracking.history, tracking.contact. If you do not receive it from Sitecore support, get it using Rest API.
    • A Search index connection string (applicable to Sitecore 8.2.1 and later): cloud.search.
    • Reporting service settings, including the address of the service and the thumbprint of the SSL certificate. If you do not receive it from Sitecore support, get it using Rest API.
  • Download xDB Cloud Client based on Sitecore version and xDB Cloud version. In our case, we needed to download Sitecore xDB Cloud 8.1 rev. 160721. Check https://kb.sitecore.net/articles/966080 for xDB Cloud Service compatibility tables.

  • Now, we need to Enable/Disable the config files as per the role we are configuring.

    • CM: Enable/Disable the files as described in https://doc.sitecore.net/sitecore_experience_platform/81/setting_up_and_maintaining/xdb/configuring_servers/configure_a_content_management_server and Config Enable Disable spreadsheet. Note: based on the version of Sitecore, this will be very.
    • CD: Enable/Disable the configs as described in https://doc.sitecore.net/sitecore_experience_platform/81/setting_up_and_maintaining/xdb/configuring_servers/configure_a_content_delivery_server and Config Enable Disable spreadsheet. Note: based on the version of Sitecore, this will be very.
    • CM + CD (Development): Ensure that the following configuration files are disabled or removed from your local installation by adding .disabled to the end of the file name.
      File path (relative to the website root) Configuration file name
      /App_Config/Include Sitecore.Analytics.Processing
      .Aggregation.Services.config
      /App_Config/Include Sitecore.Analytics.Processing
      .Services.config
      /App_Config/Include Sitecore.Analytics.Tracking
      .Database.ScaledCM.config
      /App_Config/Include Sitecore.MarketingProcessing
      Role.config
      /App_Config/Include Sitecore.PathAnalyzer
      .Processing.config
      /App_Config/Include/CES Sitecore.CES.DeviceDetection
      .CheckInitialization.config
      /App_Config/Include/ContentTesting Sitecore.ContentTesting
      .Processing.Aggregation.config
      /App_Config/Include/ExperienceAnalytics Sitecore.ExperienceAnalytics
      .Aggregation.config
      /App_Config/Include/ExperienceAnalytics Sitecore.ExperienceAnalytics
      .ReAggregation.config
      /App_Config/Include/ExperienceAnalytics Sitecore.ExperienceAnalytics
      .StorageProviders.config
      /App_Config/Include/ExperienceAnalytics Sitecore.ExperienceAnalytics
      .Reduce.config
  • Configure a content management server to use a remote Reporting Service Server as per https://doc.sitecore.net/sitecore_experience_platform/setting_up_and_maintaining/xdb/configuring_servers/configure_a_content_management_server_to_use_a_remote_reporting_service_server Specify ReportingServiceUrl given by Sitecore Support. If you do not receive it from Sitecore support, get it using Rest API.

  • For xDB Cloud 2.0, Update the Path Analyzer configurations. If you connect to a dedicated remote reporting instance, then the content management server no longer has direct access to the Sitecore.Analytics SQL Server reporting database. Instead, the Path Analyzer client retrieves data from a remote reporting server with direct access to the SQL database using Web API services. To enable the Path Analyzer client to communicate with the remote reporting server, make the following configuration file changes:

    Configuration file Folder Enable Disable
    Sitecore.PathAnalyzer
    .RemoteClient.config
    App_Config/Include Picture 154
    Sitecore.PathAnalyzer
    .Processing.config
    App_Config/Include Picture 155
    Sitecore.PathAnalyzer
    .Services.RemoteServer.config
    App_Config/Include Picture 156
  • In the XdbCloud folder (Website/App_Config/Include/xDBCloud), delete the following files. If they already have the extension.disabled it is a good idea to delete them at this point.

    • Sitecore.Cloud.Xdb.config
    • Sitecore.ContentSearch.Cloud.Index.Analytics.config
    • Sitecore.ContentSearch.Cloud.Default.IndexConfiguration.config

 

  • Use the Sitecore Installation Wizard to install the Sitecore xDB Cloud 8.1 rev. 160721 package.

  • To configure instance to communicate with dedicated Azure services of Sitecore xDB Cloud

    • In the fileConnectionStrings.config(Website/App_Config), configure the MongoDB database connection strings by using the connection strings from the response that you received from Sitecore Support. If you do not receive it from Sitecore support, get it using Rest API.The MongoDB connection strings:
      • analytics
      • tracking.live
      • tracking.history
      • tracking.contact

      Note: For a CD instance, you do not require the tracking.history connection string.

    • In the ConnectionStrings.config file (Website/App_Config), add the analytics.cloud.index Connectionstring. Which you generally do not find in ConnectionStrings.config
    • In the ConnectionStrings.config file (Website/App_Config), ensure the following connection string is removed:<add name="reporting" connectionString="Data Source=…"/>
    • In the Xdb Cloud folder (Website/App_Config/Include/xDBCloud), enable the following configuration file:
      • Sitecore.Cloud.Xdb.config

      For versions prior to 8.2 Update-1, also enable below config files:

      • Sitecore.ContentSearch.Cloud.DefaultIndexConfiguration.config
      • Sitecore.ContentSearch.Azure.Index.Analytics.config
        (There isn't such config file available. Enable below config file instead.)
        Sitecore.ContentSearch.Cloud.Index.Analytics.config
    • In the folder (Website/App_Config/Include/), disable the following configuration files:
      • Sitecore.ContentSearch.Lucene.Index.Analytics.config
      • Social\Sitecore.Social.Lucene.Index.Analytics.Facebook.config
    • In the Sitecore.Cloud.Xdb.config file (Website/App_Config/Include/Xdb Cloud), configure the reporting service by using the actual Service URL and SSL certificate thumbprint from the response that you received from Sitecore Support, for example:
      <httpTransportFactory patch:instead="httpTransportFactory" type="Sitecore.Cloud.Xdb.CloudBasedTransportFactory, Sitecore.Cloud.Xdb" singleInstance="true">
      <param desc="serviceUrl">[reporting service URL]</param>
      <param desc="certificateThumbprint">[SSL certificate thumbprint]</param>
      </httpTransportFactory>

      By setting this reporting service properly, you will able to see experience analytics reports working on your Sitecore instance. Still you need to confirm if analytics data are getting stored to MongoDB or not.

  • To complete the xDB Cloud client configuration and connection to the xDB Cloud service, you must deploy marketing definitions.

    • On the Sitecore Launchpad, click Control Panel, Analytics, Deploy marketing definitions.
    • In the Deploy marketing definitions dialog box, select all of the definitions and taxonomies and click Deploy.

Connection

After successfully configuring xDB Cloud for your website, you will be very eager to test it by connecting to MongoDB. But unfortunately there isn’t a straight forward of doing same as you cannot connect to xDB Cloud using Robomongo or MongoVUE tool.

These are the best possible ways doing same:

  • Using Rest API like
  • By generating some random traffic on the site, and by manually or technically doing Session Abandon to reflect the Contact details in Experience Profile.
  • Give a unique name to the site definition rather than website, and generate the random traffic to get the unique site name reflected in Analytics Reports when Session ends.
  • If none of the above trick work than there is maximum chance of connectivity issue, means your data is not getting saved to MongoDB. But still as last option, you can contact Sitecore Support and request full database set collections as backup. By examining the backup collections, you can confirm weather data is getting stored or not.

Troubleshooting

While configuring/connecting xDB Cloud, I ran into several issues. Below are some note on that if you face the same, can save your day by applying these tweaks:

  • While deploying Marketing Definitions to complete the configuration/setup, i was getting following error:
    MarketingDefinitions
    Kiran Patil – Maiden Indian Sitecore MVP, found following error in log file Could not find configuration node: marketingDefinitions/asset/repositories/remote.
    And with his analytical ability he was able to solve this by enabling following config file Sitecore.Xdb.Remote.Client.MarketingAssets.config in \Website\App_Config\Include\
  • While data was getting stored to xDB Cloud at one infrastructure without touching Firewall settings, at another infrastructure it was not getting saved by performing all same steps mentioned above. By checking log file I found error ERROR unable to connect to a member of the replica set matching the read preference primary.
    One of the thing i tried to get rid of this error is removing replica set and secondary servers from ConnectionString which didn’t solved the issue but ended up changing error to ERROR Unable to connect to server *.mongolab.com:[port]: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
    I ended up reverting the change. And looking into Firewall related settings. Will share my learning once i solve the issue.

 

Sitecore xDB Cloud 2.0 – Useful Terminologies

In earlier, blog post we discussed the basics of xDB Cloud, some of the advantages and disadvantages of using xDB Cloud. Now it’s time to jump on the battle field to configure the xDB Cloud on the local environment. But before moving ahead to configuration and connection, let’s walk through few terminologies which will help you understand each and every configuration very well throughout the setup.

  • The xDB Cloud Set – a set of cloud services and connected databases that implement on-premise xDB functionality off-premise in the cloud. This functionality includes the ability to collect and process experience data, as well as contact search and segmentation. In a simple word, It is the set of databases like analytics, tracking.live, tracking.history, tracking.contact, analytics.cloud.index which you generally found in your connectionstring.config.

  • LicenseId – It refers to the Id of your Sitecore license. It will be required while doing Rest API calls which you will see in future blog posts.

  • DeploymentId – Every xDB Cloud set assigned a DeploymentId. Id using which you identify the different set/role. For ex: If we have purchased two xDB Cloud set one for Stag and one for Prod than we will have to correspond DeploymentId. For ex: XYZSTAG01, XYZPROD01. This will be very helpful along with LicenseId while making Rest API call for various purposes.

  • DeploymentType – Type of the xDB Cloud set, either Prod or Non-Prod.  Generally, you don’t need to specify the Deployment Type anywhere. Prod type won’t have reporting related to dataset.

  • SitecoreVersion – The Sitecore version, on which a site is running. There will be little change in configuration based on Sitecore version. We will consider Sitecore Experience Platform 8.1 rev. 160519 (8.1 Update-3) for rest of the blog series.

  • XdbCloudVersion – Version of the Xdb Cloud. Xdb Cloud 1.0/2.0 are available. Based on which Cloud set you have, it requires different configuration.

  • xDB Cloud Client – A package responsible for copying required xDB Cloud DLLs and Config files when installing. It will be different based on Sitecore Version and XdbCloudVersion.

  • XdbConnectionStrings – Connectionstring for all analytics databases like analytics, tracking.contact, tracking.history, tracking.live, analytics.cloud.index etc.

  • ReportingServiceUri – You cannot access reporting SQL database directly in Cloud. You have to access it using a service. So, service for reporting called ReportingServiceUri needed to be updated in one of the config files for xDB Cloud.

  • ReportingServiceCertificate – It is a certificate id provided by Sitecore Support or Rest API, which is required to securely talk with reporting service. It also needed to update in config along with ServiceUri.

Knowing these terms will give you a better understanding of overall xDB Cloud set. And you are now eligible for the configuration and connection which will take place in next blog post.

Go Cloud 🙂

featured-enterprise-cloud-strategy-eBook

Sitecore xDB Cloud 2.0 – What it is? Advantages and disadvantages over On-Premise setup

Sitecore and Cloud

Sitecore extensively looking to offer the cloud offering after joining the party with Microsoft in 2016. In an effort to that, Sitecore now available as an App Service on Azure. Have a look at it https://azuremarketplace.microsoft.com/en-us/marketplace/apps/Microsoft.AppSvc_SiteCore_xp?tab=Overview/ Along with this, Sitecore also supports xDB service completely on the cloud (xDB Cloud) if you are running Sitecore 8.0 or higher.

What is included in Sitecore xDB Cloud?

Sitecore xDB Cloud Edition is a managed service that enables you to run Sitecore xDB entirely in the Cloud. This service includes dedicated Sitecore application servers for processing, aggregation, and reporting.

  • Microsoft SQL Server reporting database
  • MongoDB collection database
  • Contact segmentation index

xDB Cloud overview-Picture 1-rId10-1173281914

That means, When using xDB Cloud offerings, you do not need to take care of reporting, aggregation, and processing server. Your CM and CD environments will be limited to their job of serving site only rather than acting as processing server too which will make them healthy.

Why it is important?

The Sitecore xDB Cloud service provides the following benefits over an on-premise installation:

  • A simpler infrastructure
  • Reduced upfront costs
  • Reduced maintenance costs
  • Increased reliability, provided by the Microsoft Azure cloud platform and the MongoLab (mlab) service for MongoDB databases.
  • A fully managed service that includes Sitecore updates and monitoring. Sitecore continuously monitors the xDB Cloud Sets to minimize system downtime.

Also refer to the fine blog post, which helps you define what solution fits your requirements, either on-premise or on the cloud.

xDB_DecisionTree
Source: http://www.nonlinearcreations.com/Digital/how-we-think/articles/2015/07/Sitecore-xDB-to-Cloud-or-not-to-Cloud.aspx

Let’s also understand some of the things which are not supported by xDB cloud:

  • The downside is that you don’t have direct access to the various collections. So, you can’t connect using a tool like Robomongo or MongoVUE. However, you can request a full export of the xDB Collection database through support.sitecore.net
  • It’s important to note that the reporting SQL databases are hosted in the cloud. This includes both the primary and secondary. There is currently no way to successfully rebuild the reporting databases from the /sitecore/admin/rebuildreportingdb.aspx page. It looks like we have a Rest API for this purpose.
  • You cannot add custom facets to contacts.
  • You cannot create reports based on custom aggregations.
  • You cannot create custom database and collections, hence you have to opt for stand alone database set for different purpose (For ex: Development, QA, Stag, Prod etc.).
  • For more information on what is possible and what not, refer: https://kb.sitecore.net/articles/042722/

Though there are several limitations using xDB Cloud listed above still it’s winning the game. While Sitecore continued to expand its wings on the cloud with the help of Azure, Sitecore is saying that it’s just a beginning and you can expect a lot of things in that direction.

We will see what are the prerequisite and how to configure it for local, CM, and CD in upcoming blog posts.