Staging for Aurora MySQL - DSS 6 | Data Source Solutions Documentation

Documentation: Staging for Aurora MySQL - DSS 6 | Data Source Solutions Documentation

Staging for Aurora MySQL

While running Burst Integrate and Bulk Refresh, Data Source Solutions DSS can stream data into a target database straight over the network into a bulk loading interface that is specific for each DBMS (e.g. direct-path-load in Oracle), or else DSS puts data into a temporary directory (‘staging file') before loading data into a target database.

For best performance, DSS performs Burst Integrate and Bulk Refresh into an Aurora MySQL location using staging files. DSS implements the file staging in Aurora MySQL by using the Client Based Staging method.

Aurora MySQL Client directly reads the staging files and passes them to the Aurora MySQL Server, which then ingests them into the target tables.

This use case requires Aurora MySQL Client to be installed on the same machine where DSS is installed.

DSS first writes data into the local staging directory on the machine where DSS is installed.
DSS then uses the MySQL command <b>load data local</b> to ingest the data into Aurora MySQL target tables. Here, the Aurora MySQL Client will directly read the staging files from the staging folder and copy them into the machine where the Aurora MySQL Server is installed. The Aurora MySQL Server will read the staging files received from the Aurora MySQL Client and load them into the target tables.

Configuring Staging

DSS must be configured to stage data on a local directory before loading it into Aurora MySQL. DSS requires the following to stage data on the local directory and perform Burst Integrate and Bulk Refresh into Aurora MySQL:

Enable the local_infile system variable (of the Aurora MySQL Client and Server).
Define the following location properties (while creating a location or by editing the existing location's source and target properties) for the Aurora MySQL location:

The Location Property equivalent to the UI field is shown inside the brackets below.

1.  **Staging Directory Is Local** ([**Staging_Directory_Is_Local**](/docs/dss6-property-reference-location-properties#stagingdirectoryislocal)): Indicates that staging files are created in the local directory. This field is automatically selected by default and cannot be unselected.

2.  **STAGING DIRECTORY** ([**Staging_Directory**](/docs/dss6-property-reference-location-properties#stagingdirectory)): Local directory path where DSS will create the temporary staging files (e.g. **/my_staging_dir**).

3.  **STAGING DIRECTORY AS VISIBLE FROM THE DATABASE** ([**Staging_Directory_Database**](/docs/dss6-property-reference-location-properties#stagingdirectorydatabase)): This field must be left blank.