How Can I Create Multiple Integrate Jobs to Run Simultaneously?

Question

Occasionally, a large number of records are written or updated on the source database, impacting multiple tables in a channel served by one integrate job. When this happens, the changes are integrated in the target sequentially, resulting in integrate latency.

How can I set up multiple integrate jobs to run simultaneously and speed up data replication when this happens?

Environment

DSS 6

Answer

The following procedure involves creating new table groups, duplicating the existing target location, creating a new Integrate action, and modifying the existing Integrate action.

To create multiple/separate integrate jobs for different groups of tables, do the following:

1. Suspend the capture job.
2. Let the integrate job run until all changes are processed. Ensure at least two empty cycles complete.
3. Suspend the integrate job.
4. Create a new table group.
5. Assign tables to the new group, ensuring they are no longer assigned to any other groups. For more information, see our Changing Table Groups documentation.
6. Duplicate the existing target location in the channel and save it with a different name. Do not select the "Include channel membership" option unless this location is not part of any other channels. For more information about duplicating a location, see sectionDuplicating Location.
7. Add the new location to the channel in the target location group. For more information, see our Adding Existing Location to Channel documentation.
8. Add action Integrate for the new target location with the table scope set to the new table group.
9. Alter existing Integrate actions to ensure that they replicate only the correct table group and not "all tables".
10. Activate replication with the following options selected: Jobs, Table Enrollment, and State Tables. For more information about activating replication/channel, see section Activate Replication.

Once you have completed the above steps, the replication will continue from where it last stopped.