Capture from SAP NetWeaver on Oracle - DSS 6 | Data Source Solutions Documentation

Documentation: Capture from SAP NetWeaver on Oracle - DSS 6 | Data Source Solutions Documentation

Capture from SAP NetWeaver on Oracle

This section describes the requirements for capturing changes from SAP NetWeaver on Oracle.

Table Types

DSS supports capture from the following table types in Oracle:

  • Ordinary (heap-organized) tables
  • Partitioned tables
  • Index-organized tables

Capture Methods

DSS allows the following methods for capturing (Capture) changes from SAP NetWeaver on Oracle:

  • Direct Redo Access
    Capture changes directly from Oracle's redo and archive file.

    Click here for the configuration required when using this capture method.

    In this capture method (Capture_Method=DIRECT), DSS reads transaction log records directly from the DBMS log file using the file I/O. This method is very fast in capturing changes from the Oracle database. The DIRECT log read method requires that the DSS Agent is installed on the source database machine.

Oracle parameter DB_BLOCK_CHECKSUM=OFF is not supported by log-based capture. Values TYPICAL (the default) and FULL (unnecessarily high) are supported by DSS.

Native Access to Redo Files

DSS's capture job needs permission to read Oracle's redo and archive files at the Operating System level. There are three different ways that this can be done;

  1. Install DSS so it runs as Oracle's user (e.g. oracle).
  2. Install DSS under a username (e.g. User) which is a member of Oracle's default Operating System group (typically either oinstall or dba).
    • On Linux the default group of user oracle can be seen in the 4th field of its line in /etc/passwd. The DSS user be made a member of that group by adding its name to file /etc/group (e.g. line oinstall:x:101:oracle).
    • On Windows, right-click My Computer and select Manage ▶ Local Users and Groups ▶ Groups ▶ ora_dba ▶ Add to Group ▶ Add.
      Note that adding DSS's user to group dba will also give DSS sysdba privilege.
  3. Create special Access Control Lists (ACLs) on these files so that DSS's user can read them.
    • On Linux, the following commands can be run as user oracle to allow user User to see redo files in $ORACLE_HOME/oradata/SID and archive files in $ORACLE_HOME/ora_arch. Note that an extra "default ACL" is needed for the archive directory, so that future archive files will also inherit the directory's permissions.

          $ setfacl -R -m u:dss:rx $ORACLE_HOME/oradata
          $ setfacl -R -m u:dss:rx,d:u:dss:rx $ORACLE_HOME/ora_arch
      

Extra Grants For Accessing Redo Files Over TNS

For certain Oracle versions (11.2.0.2 and 11.2.0.3), DSS reads the redo and archive files directly through its SQL connection, provided those files are on ASM storage or the connection to the source database is over TNS.

The DSS database user must be granted the privileges mentioned in section Grants for Log-Based Capture and additionally <b>select any transaction</b> privilege for DSS to read the redo and archive files directly through its SQL connection.

  • Archive Only
    Capture changes from Oracle's transaction log backup files.

    Click here for the configuration required when using this capture method.

    In this capture method (Capture_Method=ARCHIVE_ONLY), DSS reads changes from redo archive files available in the directory specified in ARCHIVE WATCH DIRECTORY (Archive_Log_Path). Also, DSS can be configured to consider only the files that match the format defined in LOG ARCHIVE FORMAT (Archive_Log_Format).

    DSS does not read anything from online redo files or the 'primary' archive destination. This allows the DSS process to reside on a different machine than the Oracle DBMS and read changes from files that are sent to it by some remote file copy mechanism (e.g. FTP). The capture job still needs an SQL connection to the database for accessing dictionary tables, but this can be a regular connection.

    Using the Archive Only capture method may result in longer replication delays compared to the 'online' mode. To minimize these delays, it is possible to force Oracle to issue an archive once per a predefined period.

    DSS supports cross-platform capture of archived redo files. For example, on a Linux machine where Capture_Method=ARCHIVE_ONLY is defined, DSS can capture archived redo files coming from an AIX machine.

    On RAC systems, the delays are defined by the slowest or the least busy node because archives from all threads must be merged by SCNs to generate replicated data flow.

    Archive Watch Directory

    DSS will search for the log archives in the directory specified in this field.

    Any process such as the Oracle archiver (if another LOG_ARCHIVE_DEST_N is defined), RMAN, or a simple shell script can copy log archive files to this directory. Therefore, it is important to ensure that the files in this directory are purged periodically to prevent the directory from filling up.

    Log Archive Format

    If the names of the compressed archive log files differ from the original names of the archive log files, you may describe the filename format (template) of the transaction log archive files stored in the ARCHIVE WATCH DIRECTORY.

    This field accepts the following format variables:

    • %d - match numbers (zero or more decimal digits). Numbers matched using this variable are ignored by DSS.
    • %r or %R - resetlogs ID
    • %s or %S - log sequence number
    • %t or %T - thread number
    • %z or %Z - match alphanumeric characters. Characters matched using this variable are ignored by DSS.

Wildcard character * is not supported.

For more information about the format variables, refer to [LOG_ARCHIVE_FORMAT](https://docs.oracle.com/cd/B19306_01/server.102/b14237/initparams103.htm#REFRN10089) in [Oracle documentation](https://docs.oracle.com).

When this location field is not defined, then by `default` DSS will query the database for Oracle's initialization field **LOG_ARCHIVE_FORMAT**.

**Example 1:** <br />
Suppose an archive log file is named '**o1_mf_1\_41744_234324343.arc**' according to a certain Oracle archive log format pattern '**o1_mf\_\<thread\>\_\<sequence\>\_\<some_number\>.arc**'. The archive file is copied to some custom directory and compressed to '**o1_mf_1\_41744_234324343.arc.gz**' with the **.gz** extension added to its name. In such a case, specify the following pattern '**o1_mf\_%t\_%s\_%z.arc.gz**' in **LOG ARCHIVE FORMAT** field.

**Example 2:** <br />
Suppose the compressed archive log file is named **CQ1arch1_142657_1019376160.dbf.Z** with the **.Z** extension added to its name. In such a case, specify the following pattern '**CQ1arch%t\_%s\_%r.dbf.Z'** in **LOG ARCHIVE FORMAT** field.

The following are used by DSS if **LOG ARCHIVE FORMAT** field is not defined,

- For Oracle ASM system, the default name pattern used is '**thread\_%t_seq\_%s.%d.%d**'.

- Non-ASM system,
    
  - if Fast Recovery Area (FRA) is used, then the default name pattern used is '**o1_mf\_%t\_%s\_%z\_.arc**'
    
  - if FRA is not used, then DSS uses the following SQL query:
    
      ```shell
      SELECT value
      FROM v$parameter
      WHERE name = 'log_archive_format';
      ```
    
      DSS picks up the first valid archive destination and then finds the format as described above.
    
    <div class="callout callout-note">

To determine whether the destination uses FRA, DSS uses the following query:

    ```shell
    SELECT destination
    FROM v$archive_dest
    WHERE dest_name='LOG_ARCHIVE_DEST_[n]';
    ```

    For example, for destination 1, the query is as follows:

    ```shell
    SELECT destination
    FROM v$archive_dest
    WHERE dest_name='LOG_ARCHIVE_DEST_1';
    ```

    If the query returns **USE_DB_RECOVERY_FILE_DEST**, it indicates the destination uses FRA.
#### Capturing from Compressed Archive Log Files

DSS supports capturing changes from compressed archive log files that are moved from a 'primary' archive log directory to a custom directory. DSS automatically detects the compressed files, decompresses them, and reads data from them.

This feature is activated when the directory defined in **ARCHIVE WATCH DIRECTORY** is set to the custom directory.

Only **gzip** compressed files are supported.

#### Compressed Archive Log File Format

If the names of the compressed archive log files differ from the original names of the archive log files, then the relevant naming format should be defined in [**LOG ARCHIVE FORMAT**](#logarchiveformat) field.

Grants for Log-Based Capture

This section lists the grants required for capturing changes from Oracle database.

DSS can either connect to the database as the owner of the replicated tables, or it can connect as a special user (e.g. username).

If you do not wish to provide <b>create table</b> privilege to the DSS database user, the alternative method is to manually create this temporary table using the SQL statement:

```shell
create global temporary table dss_sys_table (table_name varchar(128), table_owner varchar(128));
```

This temporary table is not used when capturing from a Data Guard standby database.

An alternative to all of the above grants is to provide the sysdba privilege to the DSS database user (e.g. username):

Supplemental Logging

DSS needs the Oracle supplemental logging feature enabled on replicate tables that it replicates. Otherwise, when an <b>update</b> is done, Oracle will only log the columns which are changed. But DSS also needs other data (e.g. the key columns) so that it can generate a full <b>update</b> statement on the target database. The Oracle supplemental logging can be set at the database level and on specific tables. In certain cases, this requirement can be dropped by defining action ColumnProperties with parameter CaptureFromRowId, for more information, see section Capturing from Oracle ROWID.

The very first time that Activate Replication (dssactivate) runs, it will check if the database allows any supplemental logging at all. If it is not, then Activate Replication will attempt statement <b>alter database add supplemental log data</b> (see Extra Grants for Supplemental Logging to execute this statement). Note that this statement will hang if other users are changing tables. This is called 'minimal supplemental logging'; it does not actually cause extra logging; that only happens once supplemental logging is also enabled on a specific table.

Activate Replication will normally only enable supplemental logging for the key columns of each replicated table, using statement <b>alter table</b> <em>table1</em> <b>add supplemental log data (primary key) columns</b>. But in some cases, Activate Replication will instead perform <b>alter table</b> <em>table1</em> <b>add supplemental log data (all) columns</b>, this will happen if either of the following condition is met:

Supplemental Log Data Subset Database Replication

DSS does not support the 'Supplemental Log Data Subset Database Replication' option on Oracle 19c and higher versions. This feature must be disabled for your Oracle database when using DSS for replication.

To verify whether the database is enabled for subset database replication ('YES' or 'NO'), use the following query:

select supplemental_log_data_sr from v$database;

To disable this option, use the following query:

alter database drop supplemental log data subset database replication;

Capturing from Oracle ROWID

If none of the above requirements force DSS to enable supplemental logging on all columns, the requirement for supplemental logging on key columns can be removed by defining action ColumnProperties with parameters CaptureFromRowId and SurrogateKey to the channel. By doing so, DSS will treat the Oracle rowid column as part of the table and use it as the key column during replication, seamlessly integrating it into the target database. To ensure the capture of rowid values and their use as surrogate replication keys, the following two actions must be added to the channel prior to adding tables:

Location Action Parameter Annotation
Source ColumnProperties Name=dss_rowid
CaptureFromRowId
This action should be defined for capture location(s) only.
* ColumnProperties Name=dss_rowid
SurrogateKey
This action should be defined for both capture and integrate locations.

Grants for Supplemental Logging

The DSS database user must be granted the privileges mentioned in section Grants for Log-Based Capture and additionally the following grants for using supplemental logging:

Accessing Redo and Archive

The Oracle instance must have archiving enabled. If archiving is not enabled, DSS will lose changes if it falls behind the redo logs or it is suspended for a time.

DSS supports capturing changes made by Oracle's Direct-Load INSERT feature (e.g. using <b>insert</b> statements with 'append hints' (<b>insert /*+ append */ into</b>)). For DSS to capture these changes:

Archiving of redo log files can be enabled by running the following statement as sysdba against a mounted but unopened database:

alter database archivelog

The current state of archiving can be checked using query:

select log_mode from v$database

The current archive destination can be checked with the query <b>select destination, status from v$archive_dest</b>. By default, this query will return values USE_DB_RECOVERY_FILE_DEST, VALID, which means that DSS will read changes from within the flashback recovery area. Alternatively, an archive destination can be defined with the following statement: <b>alter system set log_archive_dest_1='location=/disk1/arch'</b> and then restart the instance.

Often Oracle's RMAN will be configured to delete archive files after a certain time. But if they are deleted too quickly then DSS may fail if it falls behind or it is suspended for a time. This can be resolved by re-configuring RMAN so that archive files are guaranteed to be available for a specific longer period (e.g. 2 days).

If DSS is restarted it will need to go back to the start oldest transaction that was still open, so if the system has long running transactions then archive files will need to be retained for longer period.

Managing Archive or Redo Log files

If log-based capture is configured for an Oracle database then DSS may need to revert to reading the Oracle archive/redo files. However, it is important to note that each site has an existing backup/recovery regime (normal RMAN) that periodically deletes these Oracle archive files. To ensure that these archive files remain available for the DSS capture job(s), it is necessary to configure RMAN in such a way that the archive files are retained for a sufficient amount of time. The length of this time will vary depending on the replication environment, and factors such as how long could replication be interrupted, and after what period of time would the system be reset using a Refresh.