Location Connection for Azure Data Lake Storage - DSS 6 | Data Source Solutions Documentation
Documentation: Location Connection for Azure Data Lake Storage - DSS 6 | Data Source Solutions Documentation
Location Connection for Azure Data Lake Storage
This section lists and describes the connection details required for creating Azure Data Lake Storage (DLS) location in Data Source Solutions DSS. For information about the pre-requisites, access privileges, and other configuration requirements for creating Azure DLS location, see Azure Data Lake Storage Requirements.
If the managed secrets feature is enabled, option USE TOKEN INSTEAD is displayed in the fields designated for entering secrets.
| Field | Description | Equivalent Location Property |
|---|---|---|
| PROTOCOL | Protocol for connecting DSS to the Azure DLS server.
|
File_Scheme |
| HOST | Hostname or IP-address of the server on which the Azure DLS is running. This field is enabled only if PROTOCOL is set to Azure Data Lake Storage Gen1. |
File_Host |
| ACCOUNT | Name of the Azure DLS Gen2 storage account. This field is not available if PROTOCOL is set to Azure Data Lake Storage Gen1. |
ABFS_Account |
| CONTAINER | Name of the container available within the Azure DLS Gen2 storage ACCOUNT . This field is not available if PROTOCOL is set to Azure Data Lake Storage Gen1. |
ABFS_Container |
| DIRECTORY | For Azure DLS Gen1, this is the directory path where the files are replicated to or captured from. For Azure DLS Gen2, this is the directory path in CONTAINER where the files are replicated to or captured from. |
File_Path |
| AUTHENTICATION METHOD | Authentication method for connecting DSS to Azure Data Lake Storage (ADLS) Gen2 server.
For more information about these authentication types, see section Authentication Methods in Azure Data Lake Storage Requirements.
Available options for ADLS Gen2 are:
When PROTOCOL is set to Azure Data Lake Storage Gen2 (HTTP), only Shared Key is available as the authentication method. Available options for ADLS Gen1 are:
|
ABFS_Authentication_Method |
| SHARED KEY | Access key of the Azure storage account. This field is enabled only if the AUTHENTICATION METHOD is set to Shared Key. |
Azure_Shared_Secret_Key |
| OAUTH2 ENDPOINT | URL used for obtaining the bearer token with credential token. This field is enabled only if the AUTHENTICATION METHOD is set to Service-to-service. |
Azure_OAuth2_Endpoint |
| OAUTH2 USER | Username for the OAuth 2.0 authentication. This field is enabled only if the AUTHENTICATION METHOD is set to User Name and Password. |
Azure_OAuth2_User |
| PASSWORD | Password of the OAUTH2 USER. | Database_Password |
| CLIENT ID | Client ID used to obtain Microsoft Entra ID (formerly Azure Active Directory) access token. This field is enabled only if the AUTHENTICATION METHOD is set to Service-to-service or Refresh token. |
Azure_OAuth2_Client_Id |
| CLIENT SECRET KEY | Secret key of the CLIENT ID. This field is enabled only if the AUTHENTICATION METHOD is set to Service-to-service. |
Azure_OAuth2_Client_Secret |
| TOKEN | Directory path to the text file containing the refresh token. This field is enabled only if the AUTHENTICATION METHOD is set to Refresh token. |
Azure_OAuth2_Refresh_Token |
| TENANT ID | URL for the REST endpoint of the token service exposed to localhost by the identity extension in the Azure VM. This field is enabled only if the PROTOCOL is set to Azure Data Lake Storage Gen2 (HTTPS) and the AUTHENTICATION METHOD is set to MSI. |
Azure_OAuth2_MSI_Tenant |
| PORT | Port number for the REST endpoint of the token service exposed to localhost by the identity extension in the Azure VM. The |
Azure_OAuth2_MSI_Port |
| Hive External Tables | Enable/Disable Hive ODBC connection configuration for Hive external tables created above Azure DLS. |
Configuration for Hive External Tables
This section lists and describes the connection parameters required for connecting to Hive External Tables created above Azure DLS.
DSS allows you to create Hive External Tables above Azure DLS which are only used during compare. You can enable/disable the Hive configuration for Azure DLS in location creation screen using the field Hive External Tables . For more information about configuring Hive external tables, refer to Hadoop Azure Data Lake Storage Support documentation.
| Field | Description | Equivalent Location Property |
|---|---|---|
| HIVE SERVER TYPE |
Type of the Hive server. Available options are:
|
Hive_Server_Type |
| HOST(S) | Hostname or IP-address of the server on which the database is running. | Database_Host |
| PORT | Port number on which the Hive server is expecting connections. | Database_Port |
| DATABASE | Name of the database. | Database_Name |
| SERVICE DISCOVERY MODE | Mode for connecting DSS to Hive Server 2. This field is enabled only if HIVE SERVER TYPE is set to 2. Available options are:
|
Hive_Service_Discovery_Mode |
| ZOOKEEPER NAMESPACE | Namespace on ZooKeeper under which Hive Server 2 nodes are added. This field is enabled only if SERVICE DISCOVERY MODE is set to ZooKeeper. |
Hive_Zookeeper_Namespace |
| AUTHENTICATION METHOD |
Authentication method for connecting DSS to Hive Server 2. This field is enabled only if HIVE SERVER TYPE is set to 2. Available options are:
|
Hive_Authentication_Method |
| USER | Username for connecting DSS to the database (defined in Database_Name). This field is enabled only when the AUTHENTICATION METHOD is set to User Name or User Name and Password. |
Database_User |
| PASSWORD | Password for the USER. | Database_Password |
| SERVICE | Kerberos service principal name of the Hive server. This is the service name part of Kerberos principal of the Hive server. For example, if the principal is hive/example.host@EXAMPLE.REALM then "hive" should be specified here. This field is enabled only if AUTHENTICATION METHOD is set to Kerberos. |
Hive_Kerberos_Service |
| HOST | Fully Qualified Domain Name (FQDN) of the Hive server host. This is the host part of Kerberos principal of the Hive server. For example, if the principal is "hive/example.host@EXAMPLE.REALM" then "example.host" should be specified here. The value for this field may be set to _HOST to use the Hive server hostname as the domain name for Kerberos authentication. If SERVICE DISCOVERY MODE is set to None, then the driver uses the value specified in the Host connection attribute. |
Hive_Kerberos_Host |
| REALM | Realm of the Hive Server 2 host. It is not required to specify any value in this field if the realm of the Hive Server 2 host is defined as the default realm in Kerberos configuration. This field is enabled only if AUTHENTICATION METHOD is set to Kerberos. |
Hive_Kerberos_Realm |
THRIFT TRANSPORT |
Transport protocol to use in the Thrift layer. This field is enabled only if HIVE SERVER TYPE is set to Hive Server 2. Available options are:
For information about determining which Thrift transport protocols your Hive server supports, refer to HiveServer2 Overview and Setting Up HiveServer2 sections in Hive documentation. |
Hive_Thrift_Transport |
| HTTP PATH | The partial URL corresponding to the Hive server. This field is required only if THRIFT TRANSPORT is set to HTTP. |
Hive_HTTP_Path |
| Enable SSL | Enable/disable (one way) SSL. If enabled, DSS authenticates the Hive server by validating the SSL certificate shared by the Hive server. | |
| TRUSTED CA CERTIFICATE | Directory path where the .pem file containing the server's public SSL certificate signed by a trusted CA is located. Defining this property will enable (one way) SSL, which means, DSS will authenticate the Hive server by validating the SSL certificate shared by the Hive server. This property is also required for enabling two way SSL. |
Database_Public_Certificate |
| Two-way SSL | Enable/disable two way SSL. If enabled, both DSS and Hive server authenticate each other by validating each others SSL certificate. This field is enabled only if Enable SSL is selected. | |
| CLIENT PUBLIC CERTIFICATE | Directory path where the .pem file containing the client's SSL public certificate is located. This field is enabled only if Two-way SSL is selected. |
Database_Client_Public_Certificate |
| CLIENT PRIVATE KEY | Directory path where the .pem file containing the client's SSL private key is located. This field is enabled only if Two-way SSL is selected. |
Database_Client_Private_Key |
| CLIENT PRIVATE KEY PASSWORD | Password of the client's SSL private key specified in CLIENT PRIVATE KEY. This field is enabled only if Two-way SSL is selected. |
Database_Client_Private_Key_Password |
Hive Advanced Settings
| Field | Description | Equivalent Location Property |
|---|---|---|
| LINUX / UNIX ODBC DRIVER MANAGER LIBRARY PATH | Directory path where the ODBC Driver Manager Library is installed. This field is applicable only for Linux/Unix operating system. For a default installation, the ODBC Driver Manager Library is available at /usr/lib64 and does not need to be specified. However, when UnixODBC is installed in for example /opt/unixodbc the value for this field would be /opt/unixodbc/lib. |
ODBC_DM_Lib_Path |
| LINUX / UNIX ODBCSYSINI | Directory path where the odbc.ini and odbcinst.ini files are located. This field is applicable only for Linux/Unix operating system. For a default installation, these files are available at /etc directory and do not need to be specified using this field. However, when UnixODBC is installed in for example /opt/unixodbc the value for this field would be /opt/unixodbc/etc. |
ODBC_Sysini |
| ODBC DRIVER | Name of the user defined (installed) ODBC driver used for connecting DSS to the Hive server. | ODBC_Driver |
Advanced Settings
<b>Since</b> v6.1.0/45 <b>Since</b> v6.1.5/4
This section lists and describes the connection details required when connecting DSS to the Azure DLS server via a proxy server.
DSS provides enhanced control over network routing for authentication and data traffic to Azure Data Lake Storage (DLS). By utilizing separate proxy servers, DSS can route the traffic through distinct network paths. There are two options available - Data Proxy and Authentication Proxy.
If only the Data Proxy connection parameters are defined without the Authentication Proxy, then the data proxy server will handle both authentication and data traffic.
Data Proxy
| Field | Description | Equivalent Location Property |
|---|---|---|
| PROXY PROTOCOL | Protocol for the proxy server host used for connecting DSS to the Azure DLS server. Available option:
|
File_Proxy_Scheme |
| PROXY HOST | Host name of the proxy server used for connecting DSS to the Azure DLS server. | File_Proxy_Host |
| PROXY PORT | Port number of the proxy server host used for connecting DSS to the Azure DLS server. | File_Proxy_Port |
| PROXY USER | Username for the proxy server host used for connecting DSS to the Azure DLS server. | File_Proxy_User |
| PROXY PASSWORD | Password for the PROXY USER. | File_Proxy_Password |
Authentication Proxy
This option is not available if AUTHENTICATION METHOD is set to Shared Key.
| Field | Description | Equivalent Location Property |
|---|---|---|
| AUTHENTICATION PROXY PROTOCOL | Protocol for the authentication proxy server host used for connecting DSS to the Azure DLS server. Available option:
|
Azure_Auth_Proxy_Scheme |
| AUTHENTICATION PROXY HOST | Host name of the authentication proxy server used for connecting DSS to the Azure DLS server. | Azure_Auth_Proxy_Host |
| AUTHENTICATION PROXY PORT | Port number of the authentication proxy server host used for connecting DSS to the Azure DLS server. | Azure_Auth_Proxy_Port |
| AUTHENTICATION PROXY USER | Username for the authentication proxy server host used for connecting DSS to the Azure DLS server. | Azure_Auth_Proxy_User |
| AUTHENTICATION PROXY PASSWORD | Password for the AUTHENTICATION PROXY USER. | Azure_Auth_Proxy_Password |