How does Hub change detection work?

Introduction

Hub can poll for data in each external repository using three different methods: change detection, high performance full scan, and high fidelity full scan. Within Hub, you can set global change detection and full scan intervals on the Settings screen. The full scan style (either high fidelity or high performance) is chosen by Hub on an integration-by-integration basis (you will see this referred to as "Tasktop-Optimized" in the UI), but can also be manually configured for each integration. You can also set integration-specific change detection and full scan intervals and repository-specific change detection and full scan intervals that will override the global settings.

Note: When configuring change detection settings, integration-level change detection has the highest precedence, followed by repository-level change detection, and then global change detection. This means that if integration-level change detection is configured, the integration-level setting will always be used (even if repository-level or global settings are configured). If no overrides are set, change detection will default to the global settings.

Here’s a metaphor to help understand the differences between the change detection, high performance full scan, and high fidelity full scan:

• Change Detection is the daily maintenance you do on your home: putting clothes away, wiping down the counter, taking the trash out. It picks up the vast majority of artifact changes without putting high server load on your external repositories, but it can miss some updates due to third party limitations. 
• High Performance Full Scan is your monthly deep cleaning: mopping the floors, wiping down windows, really trying to clean out some of the dirt you don’t always get. It has only a slightly higher performance impact than normal change detection and is able to pick up some changes which may have been missed as well as trigger certain Hub features like the Twinless Artifact Update.  
• High Fidelity Full Scan is your monthly deep cleaning on steroids — really dusting those cobwebs off of those hard-to-reach spots. This style should pick up all artifact changes, but the trade off is that it has a high performance impact due to the amount of work Hub must do to query for all artifact changes.

Below is a graph showing the relative performance impacts of each process. Note that Change Detection and High Performance Full Scan have relatively similar, low performance impacts, while High Fidelity Full Scan has a higher and more sustained performance impact.  You can read below to learn more about how each process works.

Also note that when an integration is first started, you may see a high performance impact, especially if the scope is very large (e.g., ITSM tools or a system like Jira which may have many users).

Change Detection

Change Detection is necessary for all customers in all integration scenarios. It is the default means by which Hub recognizes updates to artifacts in external repositories and is able to propagate those changes. The change detection interval should be frequent, for example running once every 1-5 minutes.

High Performance Full Scan

There may be times when a more robust means of determining artifact changes is necessary — either to pick up rare missed artifact changes or to take advantage of a Hub features such as Twinless Artifact Update.

The High Performance Full Scan is sufficient for most customer use cases and has only a slightly larger impact on repository server load than a normal Change Detection query.

Instead of retrieving a short list of artifacts that Hub believes may have changed, the High Performance Full scan retrieves a list containing essential information (ID and Last Modified field) for all the artifacts that have synchronized in the integration, but only fully retrieves and synchronizes those artifacts that have been marked as updated.

A typical high performance full scan is configured to run once every 1-24 hours.

High Fidelity Full Scan

The High Fidelity Full Scan is only needed if you are flowing certain fields in certain repositories whose changes cannot be detected via the other two methods due to third party limitations.

This scan retrieves every artifact that has synchronized in the integration and then inspects each artifact to determine if a change needs to be propagated.

Retrieving every artifact (rather than a simple list of 'ID' and 'Last Modified' for each artifact) generates a sizable repository load.

A typical high fidelity full scan is once every 24 hours or more.

How does the “Process All Artifacts” Button fit into all this? 

On the Field Flow screen of each integration, you will see a ‘Process All Artifacts’ button. This button allows you to force through updates from one collection in your integration. This can be useful if you add a new field mapping to your configuration or if you change your artifact routing or artifact filtering criteria to add new artifacts to your integration.

When Process All Artifacts is clicked, Hub forces a special High Fidelity Full Scan to run at the next change detection interval. Unlike a typical High Fidelity Full Scan, this will scan ALL artifacts within the collection (regardless of whether they have synchronized or not). This means that it will pick up artifacts that are newly eligible for the integration based on updated routing or filtering. As such, users should expect that this feature can lead to high server load on the external repositories.