When to Use Burst vs. Continuous Integration

Question

Should I use burst integration or continuous integration?

Environment

DSS 6

Answer

DSS provides two integration methods:

• Continuous: Changes are processed row-by-row in the order in which they were committed in the source.
• Burst: Changes are processed in the following ways:
  • Coalesced down to a single net change per row for a given set of changes.
  • Bulk-loaded into staging ("burst") tables.
  • Applied using set-based changes to the target tables.

By default, both the continuous and the burst integration methods maintain transactional consistency. The burst method generally achieves a higher throughput (row changes/s) at the cost of higher latency.

DSS chooses the default method for burst or continuous processing based on the target type. However, you can change the integration method by selecting a different Method value in the Integrate action of the channel definition.

When to use continuous integration

Use continuous integration if:

• The destination is a database suitable for online transaction processing (for example, Oracle, SQL Server, PostgreSQL, etc.).
• Tables receiving regular updates or deletes are indexed on the key columns.
• Referential integrity constraints are enabled and immediately enforced on the destination. While this is not a requirement, the burst integration method does not work with referential integrity constraints that are enabled and immediately enforced.

When to use burst integration

Use burst integration if:

• The destination is an analytical database (for example, Snowflake, BigQuery, Redshift, etc.).
• You want to achieve higher throughput (row changes per second).
• Continuous integration does not work well for your use case (for example, a table has no key, gets a lot of inserts, or has resilient processing enabled).