Skip to main content
Data synchronization processes overview
Search
Search Indirect Tax Support Help and Support.

This article applies to:

  • Product: ONESOURCE Determination,ONESOURCE Indirect Tax

Bulk import

You can use a full synchronization method that replaces SDI by deleting and recreating configuration data in bulk.

About bulk import

Bulk Import replaces SDI for full synchronizations. Its different approach avoids the most common SDI problems, like slow performance and data constraint errors.

Use bulk import

Bulk Import doesn’t perform detailed validation. Instead, it uses a delete-and-recreate process.
  1. Trigger:
     This process is triggered for full sync events, such as a configuration import/export. It can be used in parallel with Streaming for smaller “delta changes”.
  2. Preparation:
     The system identifies the set of configuration data that needs to be synced.
  3. Delete and Recreate:
     Bulk Import doesn’t update each entity individually. It deletes the existing configuration set from the client database tables. Then it inserts the new set in bulk.
  4. ID Preservation:
     This method keeps configuration IDs consistent. The internal database IDs are the same in the Mothership and the client environment. This creates a “canonical” (perfectly matched) copy, which eliminates a whole class of data-related issues.
  5. Cache Validation:
     After the
    Prefetch
     service generates the cache, the Cache Validation application automatically validates it before clients can use it. This validation step typically takes approximately 5 to 10 minutes to complete.
The cache validation process ensures that:
  1. The generated cache is structurally complete and consistent.
  2. All expected entities and relationships are correctly populated.
  3. Any potential cache corruption or partial generation issues are detected early.
  4. This validation step is built into Bulk Sync. It helps prevent downstream client-side issues. Only verified and reliable cache data is served to client environments.

Key characteristics and benefits

  • Extremely Fast:
     By using bulk database operations and skipping individual validations, it reduces sync time from hours to minutes. A
    Bulk Import
     sync completed 26 million entities in about 17 minutes. This assumes good storage I/O throughput. The legacy SDI process took over 8 hours in benchmarking. The benchmark used a smaller configuration of 7 million entities.
  • Reliable:
     This method uses delete and recreate. It avoids errors caused by violations of data constraints that affected the SDI process. There are no conflicts because the old data is removed before the new data is inserted.
  • Data Consistency:
     It ensures that the client matches the Mothership exactly. This includes internal IDs. It also simplifies troubleshooting.
  • Handles All Update Types:
     It works with any type of data update. This includes direct UI changes, Configuration API calls, and full SDI-style export or import. It ensures complete and consistent synchronization.
  • Bulk Import handles both initial full sync and subsequent change detection.
Configuration Options:
Bulk Import
 is the recommended solution for full synchronization but isn't turned on by default. To turn on
Bulk Import Mode
, administrators will need to turn on the
Bulk Import
 profile in the customer overlay. Once active, it replaces SDI as the primary method for all full sync events. This includes events triggered by import or export. It also serves as a more efficient fallback process for Streaming.

System and storage requirements

These specifications represent the minimum requirements for optimal performance during the
Bulk Import
 sync process.
  • CPU:
     2+ cores recommended (for transformation operations)
  • Memory:
     16 GB RAM minimum (32 GB recommended for larger tenants)
  • Storage (recommended):
     50 GiB capacity, ReadWriteOnce (RWO), minimum 3,000 IOPS
    • Example storage classes:
       OCI
      oci-bv
       (vpusPerGB: 10, Balanced tier); AWS
      gp3
       (default settings include 3,000 IOPS)
Scheduling Options:
 The following are turned on in the Overlay.
  • Default: Every 30 minutes
  • Adjust the interval based on data change frequency and performance requirements
    • Recommended for production environments: 30 minutes to 2 hours
    • For environments with infrequent configuration changes, longer intervals (12 hours, daily) work well

Understand sync time for SDI and bulk import

The scheduling intervals listed previous are client-side settings only. They control how frequently your client environment checks for and imports a new configuration file. Before your client can import anything, a server-side background process called Prefetch will need to be completed on the Mothership side. Prefetch is responsible for generating and validating the export file that your client will consume.
This means the actual time for a configuration change to be fully reflected in your client environment is the combined total of 3 sequential steps:
Step
Process
Example duration
1
Server-side Prefetch generates the export file
30 minutes (configurable per tenant)
2
Cache Validation completes on the generated file
5 to 10 minutes
3
Client import interval picks up and applies the file
5 to 30 minutes (based on how it's set up)
Total time
n/a
40 to 70 minutes
note
Any Prefetch process is paused while a client data import is actively running. Once the client import completes, the Prefetch process automatically resumes.
Example:
In this example, Prefetch runs every 30 minutes. Cache validation takes 10 minutes. Client import runs every 5 minutes. If a configuration changes immediately after Prefetch finishes, the change can take up to 45 minutes to become visible in the client environment.
When planning your sync strategy, consider both the
Prefetch
 schedule and the client import interval. A shorter interval for the import won’t help if
Prefetch
 runs less often. The client won’t view a new file until
Prefetch
 completes.
Your
Prefetch
 schedule should match the amount of data you have. For customers with millions of certificates, a longer interval may be needed to make sure all data is prefetched. Thomson Reuters can help you determine how to set up for your environment and data volume.

Related Content