Connector Configuration

SMB File Share Configuration

SMB File Share Connection Settings

Configuration Options related to establish connection to the target SMB File Share.

Share Connection

The connection details for a share. Multiple Shares can be configured.

Name Description

User Domain

The domain of the user used to access the file share.

Username

The username used to access the file share.

User Password

The password of the user used to access the file share.

Host Name

The host that provides the file share.

Share Name

The name of the file share.

Start Folders

A list of folders to crawl.

Enable Filters

Enable Filter options for the SMB file share.

Share Filter Settings

Multiple filters can be configured for a Share.

Folder Filter Settings

Name Description

Regular Expression

The regular expression the path will be matched against.

File Filter Settings

Name Description

Action

When the Filter Rule matches, this action is performed.

Regular Expression

The regular expression the path will be matched against

File Size Filter Settings

Name Description

Action

When the Filter Rule matches, this action is performed.

Rule

The applied rule.

File Size

The applied File Size.

Path Length Filter Settings

Name Description

Maximum Path Length

Maximum path length allowed.

Date Index Filter Settings

Name Description

Date Field

Date field of the Item/Folder.

Mode

Choose if the filter will be applied on a period or on a specific date.

Unit

To calculate the relative date take the current date and go back N time.

Quantity

The amount of units which shall be used to calculate the cut-off date.

Format

A date format string.E.g.’yyyy-MM-dd’for year-month-day.

Date

A fixed date of the specified format.

SMB File Share Global Index Settings

Index Options related to all configured SMB File Shares.

Name Description

Index Folders

If Setting is enabled then the information of folders will be indexed too.

SMB File Share Global Connection Settings

Configuration Options related to all configured SMB File Shares.

Name Description

Disable Security

If you experience low download speed, changing the download buffer size may help (default: 1MB).

Throttle

If you have to decrease load on the file server you can configure a crawl throttle. Each crawled item will get delayed for the configured amount of milliseconds (default: 0 = unthrottled).

Retry Pause

The pause between retrying to connect to a file share after a failed connection.

Enable DFS

If set to true, the connector connects to both, DFS and non-DFS SMB fileshares. If you know that you don’t have DFS set up then setting this option to false generates less network traffic, but the connector won’t connect to a DFS file share. (default: true).

Kerberos Realm

To authenticate via Kerberos, the Kerberos realm needs to be configured. If you authenticate via NTLM you can leave this field blank.

Kerberos KDC

Please enter the KDC (Key Distribution Center) hostname or the Active Domain were the KDC can be looked up from. If you authenticate via NTLM you can leave this field blank.

Auto close idle period

The number of seconds a connection can be idle before being automatically closed.

Auto close check period

The number of seconds between checking for idle connections to closed.

SMB File Share ACL Cache Settings

Configuration Options related the ACL cache.

Name Description

Maximum ACL Cache Size

Maximum number of ACL entries that can be cached before the cache attempts to remove entries due to the cache size.

Expiration Period

The cache entry expiration period in seconds, the expiration period is calculated from the last time the cache entry was accessed.

LDAP SID Well Known Groups Settings

Configuration options related to bypass LDAP lookup for groups that are not part of LDAP but are known to be groups.

SIDs which are known to be groups

Some SIDs that are used on the fileshares are not part of LDAP, e.g. local groups. These can be added here to prevent warnings during the traversal.

Name Description

sid

A well known SID

label

A human readable debug label for the SID

Amazon Kendra Configuration

Instance Configuration

Configuration options related to specifying the target Kendra Index and Data Source including authentication/authorization settings.

Name Property Key Description

Index ID

raytion.connector.backend.amazon.kendra
.instance.indexId

ID of the target index. It can be retrieved in your AWS management console under Services → Amazon-Kendra → Indexes → <your_index>.

Region ID

raytion.connector.backend.amazon.kendra
.instance.regionId

ID of the region where the index is deployed. One of us-east-1(N. Virginia), us-east-2(Ohio), us-west-2(Oregon), eu-west-1(Ireland), ca-central-1(Canada), ap-southeast-1(Singapore) or ap-southeast-2(Sydney) is available.

Amazon Resource Name

raytion.connector.backend.amazon.kendra
.instance.roleArn

ARN of the IAM Service Role assigned to the index. It can be retrieved in your AWS management console under Services → Amazon-Kendra → Indexes → <your_index>. If the option Use S3 is enabled under Advanced Configuration → Content Processing Settings, make sure that the policy attached to the role contains the permission S3:GetObject for all objects inside the target bucket.

Data Source ID

raytion.connector.backend.amazon.kendra
.instance.datasourceId

ID of the Custom Data Source Connector added to target index. All documents and groups processed by the connector will be attached to this data source. It can be retrieved in your AWS management console under Services → Amazon-Kendra → Indexes → <your_index> → Data management → Data sources → <your_data_source>.

Use System Credentials

raytion.connector.backend.amazon.kendra
.instance.useSystemCredentials

To authenticate against Amazon Kendra, you must provide your AWS Access Key and AWS Secret Access Key. If Use System Credentials is set to true, these keys will be automatically discovered from following locations:

- Java System Properties aws.accessKeyId and aws.secretAccessKey

- Environment Variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

- Web Identity Token credentials from System or Environment Variables

- Credentials Profile File at location ~/.aws/credentials

- Credentials delivered through the Amazon EC2 container

- Instance profile credentials delivered through the Amazon EC2 metadata service

Access Key

raytion.connector.backend.amazon.kendra
.instance.accessKey

If Use System Credentials is set to false, access keys need to be specified explicitly in the configuration. The specified account requires the Managed Policy AmazonKendraFullAccess.

Secret Access Key

raytion.connector.backend.amazon.kendra
.instance.secretAccessKey

Secret Key of the specified AWS account. The value will be stored encrypted by the connector.

Assume Role

raytion.connector.backend.amazon.kendra
.instance.assumeRole

Enable this option to fetch the security token from STS using the provided role.

STS Assume Role Region

raytion.connector.backend.amazon.kendra
.instance.stsAssumeRole.regionId

Region ID for invoking the regional STS endpoint when requesting the service.

STS Assume Role Amazon Resource Name

raytion.connector.backend.amazon.kendra
.instance.stsAssumeRole.roleArn

ARN of the role which should be assumed by the configured role or account in the instance settings.

STS Assume Role Session Name

raytion.connector.backend.amazon.kendra
.instance.stsAssumeRole.sessionName

Arbitrary session name attached to the session established by the connector and STS for tracking the session.

STS Assume Role Session Duration

raytion.connector.backend.amazon.kendra
.instance.stsAssumeRole.sessionTimeToLive

Time to live duration for a single session.

Use Proxy

raytion.connector.backend.amazon.kendra
.instance.usepProxy

If enabled, the connection to AWS and Kendra Service will be established through a HTTP/HTTPS proxy.

Proxy Endpoint

raytion.connector.backend.amazon.kendra
.instance.proxy.endpoint

Target proxy URL including protocol, host and port.

Proxy Authentication

raytion.connector.backend.amazon.kendra
.instance.proxy.authenticate

If enabled, the connector uses the specified credentials to authenticate towards proxy.

Proxy Username

raytion.connector.backend.amazon.kendra
.instance.proxy.username

Proxy authentication username.

Proxy Password

raytion.connector.backend.amazon.kendra
.instance.proxy.password

Proxy authentication password. The value will be stored encrypted by the connector.

Content Processing Configuration (Optional)

Documents with empty content or large content can be rejected by Kendra. In order to fine-tune the behaviour for the processing of these documents, consider to set one of the properties below.

Name Property Key Description

Empty Content Token

raytion.connector.backend.amazon.kendra
.content.emptyContentToken

Items with unsupported mime types (supported are: application/pdf, text/html, application/xhtml+xml, application/msword, application/mspowerpoint and text/plain) or empty content are rejected by Kendra. To make those items available in the search, the connector allows you to configure a token which will be appended to the content of those items.

Use S3

raytion.connector.backend.amazon.kendra
.content.useS3Content

If enabled, binary content of documents exceeding the content size limit will be processed to a S3 bucket.

Content Size Limit

raytion.connector.backend.amazon.kendra
.content.s3Content.contentSizeLimit

All documents with content size exceeding this value will be processed to the configured S3 bucket. Else, documents are processed as inline documents including their content directly to Kendra index. If the S3 option is enabled, it is recommended to set the value below 5MB, as this is the limit defined by Kendra for inline documents.

Bucket ID

raytion.connector.backend.amazon.kendra
.content.s3Content.bucketId

ID of the bucket.

Region ID

raytion.connector.backend.amazon.kendra
.content.s3Content.regionId

ID of the region where the bucket is deployed. One of us-east-1(N. Virginia), us-east-2(Ohio), us-west-2(Oregon), eu-west-1(Ireland), ca-central-1(Canada), ap-southeast-1(Singapore) or ap-southeast-2(Sydney) is available.

Use System Credentials

raytion.connector.backend.amazon.kendra
.content.s3Content.useSystemCredentials

To authenticate against Amazon S3, you must provide your AWS Access Key and AWS Secret Access Key. If Use System Credentials is set to true, these keys will be automatically discovered from following locations:

- Java System Properties aws.accessKeyId and aws.secretAccessKey

- Environment Variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

- Web Identity Token credentials from System or Environment Variables

- Credentials Profile File at location ~/.aws/credentials

- Credentials delivered through the Amazon EC2 container

- Instance profile credentials delivered through the Amazon EC2 metadata service

Access Key

raytion.connector.backend.amazon.kendra
.content.s3Content.accessKey

If Use System Credentials is set to false, access keys need to be specified explicitly in the configuration. The specified account requires at least write access to the bucket.

Secret Access Key

raytion.connector.backend.amazon.kendra
.content.s3Content.secretAccessKey

Secret Key of the specified AWS account. The value will be stored encrypted by the connector.

Assume Role

raytion.connector.backend.amazon.kendra
.content.s3Content.assumeRole

Enable this option to fetch the security token from STS using the provided role.

STS Assume Role Region

raytion.connector.backend.amazon.kendra
.content.s3Content.stsAssumeRole.regionId

Region ID for invoking the regional STS endpoint when requesting the service.

STS Assume Role Amazon Resource Name

raytion.connector.backend.amazon.kendra
.content.s3Content.stsAssumeRole.roleArn

ARN of the role which should be assumed by the configured role or account in the instance settings.

STS Assume Role Session Name

raytion.connector.backend.amazon.kendra
.content.s3Content.stsAssumeRole.sessionName

Arbitrary session name attached to the session established by the connector and STS for tracking the session.

STS Assume Role Session Duration

raytion.connector.backend.amazon.kendra
.content.s3Content.stsAssumeRole.sessionTimeToLive

Time to live duration for a single session.

Use Proxy

raytion.connector.backend.amazon.kendra
.content.s3Content.useProxy

If enabled, the connection to AWS and S3 Service will be established through a HTTP/HTTPS proxy.

Proxy Endpoint

raytion.connector.backend.amazon.kendra
.content.s3Content.proxy.endpoint

Target proxy URL including protocol, host and port.

Proxy Authentication

raytion.connector.backend.amazon.kendra
.content.s3Content.proxy.authenticate

If enabled, the connector uses the specified credentials to authenticate towards proxy.

Proxy Username

raytion.connector.backend.amazon.kendra
.content.s3Content.proxy.username

Proxy authentication username.

Proxy Password

raytion.connector.backend.amazon.kendra
.content.s3Content.proxy.password

Proxy authentication password. The value will be stored encrypted by the connector.

Content Batching Configuration (Optional)

Documents are processed in a batch to Kendra. This configuration section includes all batch related properties including the callback behavior.

Name Property Key Description

Max. Size

raytion.connector.backend.amazon.kendra
.batch.batchSize

Max. batch size. All batch put requests will be restricted to this value. The max. allowed value is 10.

Ignore Processing State

raytion.connector.backend.amazon.kendra
.batch.async

If enabled, the connector submits all documents asynchronously without polling the processing state from Kendra. Documents failed during processing are not recognized by the connector. Unless you would like to monitor the indexing process using Amazon CloudWatch only, it is recommended to disable this option.

Flush Timeout

raytion.connector.backend.amazon.kendra
.batch.flushTimeout

Periodic delay between flushing the batch. Within this period, it is guaranteed that the batch is flushed. If the current batch size exceeds the configured max. batch size, only the max. number of items will be flushed in a single cycle.

Callback Timeout

raytion.connector.backend.amazon.kendra
.batch.callbackTimeout

The Batch API used to index or delete items is asynchronous. The connector is polling the state of the submitted requests to track the state of the items. This property defines the timeout until the connector is expecting the requests to be completed in the asynchronous processing in the search engine.

HTTP Connection Configuration (Optional)

Configuration options for fine-tuning the Http connection parameters.

Name Property Key Description

Connection Acquire Timeout

raytion.connector.backend.amazon.kendra
.http.connection.connectionAcquireTimeout

Timeout value for acquiring an already established connection from the connector’s connection manager.

Connection Timeout

raytion.connector.backend.amazon.kendra
.http.connection.connectionTimeout

Timeout value for establishing a connection to AWS.

Connection Idle Timeout

raytion.connector.backend.amazon.kendra
.http.connection.maxConnectionIdleTimeout

Timeout value after an idle connection should be closed.

Connection Time to Live

raytion.connector.backend.amazon.kendra
.http.connection.maxConnectionTimeToLive

Timeout value after the connection should be closed regardless of its current state.

Max. Number of Connections

raytion.connector.backend.amazon.kendra
.http.connection.maxConnections

Max. number of allowed connections maintained by the connection manager.

Max. Number of acquired connections

raytion.connector.backend.amazon.kendra
.http.connection.maxConnectionAcquires

Max. number of requests allowed to wait for a connection.

General Configuration

Database Configuration

Name Property Key Description

URL

spring.datasource.url

JDBC URL for the target database. Out of the box, the connector will use H2 file database. For productive usage, use PostgreSQL specifying the URL in format: jdbc:postgresql:<host>:<port>/<database>

Username

spring.datasource.username

Database Username to read and write to database.

Password

spring.datasource.password

Database Password for the specified user

Traversal Configuration

Name Property Key Description

Traversal History Length

raytion.connector.agent.traversal
.store.historyLength

Max. number of traversals to store in the history. Once the limit is exceeded, the connector will automatically remove oldest entries in the history. (default: 100)

Number of Traversal Workers

raytion.connector.agent.traversal
.workers.worker

Number of workers to execute the traversal in parallel. Increasing this value might improve the performance, but will footprint higher memory consumption. It is recommended to keep the default value. (default: 10)

Traversal Job Poll Interval

raytion.connector.agent.traversal
.workers.jobPollInterval

Interval between the workers to be triggered to fetch and process the next tasks. (default: 10ms)

Completion Timeout

raytion.connector.agent.traversal
.workers.completionTimeout

If the search engine indexes the items asynchronously, there might be some processing still in-flight during the completion process of a traversal. This value specifies the timeout value until all asynchronous callbacks are expected to return before completing the traversal. (default: 10m)

Principal Aliaser Configuration

Principal Aliasing is applied on user information as part of Content ACL processing during Content Synchronization and Principal processing during Principal Synchronization. It’s purpose is to map external source system user to the corresponding user in search engines domain. You can configure a list of aliasers in the connector which will be applied in sequence and in order on user ACEs and user principals. The Connector supports following custom aliasing mechanism.

Custom Aliaser Disabled

If the Custom Aliaser checkbox is not selected, the connector will process user information on ACE and user principals unchanged to Search Engine. If all relevant users in the source system can be found with the same identifier in the search engine, this setup is sufficient to reflect the same secure search experience in the search engine as defined by the policy in the source system. The connector uses this option as default to process user information.

Custom Aliaser Enabled

If custom aliasing is enable then there are four types of aliaser avaialble:

Simple XML Table Aliaser

Static mapping table which can be uploaded as XML file. The connector uses the uploaded file as lookup table to map a user in the source system to a user in the search engine. Users missing a record in the file will be dropped from the ACE and during Principal Synchronization. This option is only recommended for environment with a manageable amount of users as for each user the corresponding mapping entry needs to be specified in the file.

Name Description

XML Mapping File

Browse and upload or drag and drop.

Sample XML mapping file:

<?xml version="1.0" encoding="UTF-8"?>
<storeddata>
    <entry keyValue="user1">user1@raytion.com</entry>
    <entry keyValue="user2">user2@raytion.com</entry>
    <entry keyValue="user3">user3@raytion.com</entry>
</storeddata>
Regex Replacer Aliaser

Regex Replacer Aliaser computes aliases based on a regular expression. Principals that match the regular expression are replaced by the Substitution String.

Name Property Key Description

Pattern

raytion.connector.aliaser.aliasers[*]
.replacer.pattern

The regular expression to match, this is the part that will be replaced. If braces (…​) are used in the pattern then the matched value can be retrieved using $1

Substitute String

raytion.connector.aliaser.aliasers[*]
.replacer.substituteString

String to replace the matching part of the find string. Matched value is accessed by employing $1
Regex Extractor Aliaser

Regex Extractor Aliaser computes aliases based on a regular expression. Principals that match the regular expression are inserted into the Insert-Into String.

Name PropertyKey Description

Pattern

raytion.connector.aliaser.aliasers[*]
.extractor.pattern

The regular expression to match, this is the part that will be inserted into the new value. If braces (…​) are used in the pattern then the matched value can be retrieved using $$

Insert-Into String

raytion.connector.aliaser.aliasers[*]
.extractor.insertIntoString

String to replace the matching part of the pattern. Matched value is accessed by employing $$
LDAP Aliaser

Ldap Aliaser searches for an LDAP entry with the requested name in the input value and returns the specified output attribute.

Name Property Key Description

Host

raytion.connector.aliaser.aliasers[*]
.ldap.host

Fully Qualified Domain Name of an LDAP server

Port

raytion.connector.aliaser.aliasers[*]
.ldap.port

Port to use for LDAP connection, defaults are 389/636 or (recommended) 3268/3269 for simple/SSL

AccountDN

raytion.connector.aliaser.aliasers[*]
.ldap.bindAccountDN

AccountDN for bind to LDAP

Password

raytion.connector.aliaser.aliasers[*]
.ldap.password

Password part of credentials

Input Field

raytion.connector.aliaser.aliasers[*]
.ldap.inputField

The Active Directory attribute name for this equality filter

Search Root DN

raytion.connector.aliaser.aliasers[*]
.ldap.baseDN

Distinguished Name of the subtree which is searched. The smaller the subtree the better the performance but the higher the chance of encountering principals which are not part of this subtree

Output Field

raytion.connector.aliaser.aliasers[*]
.ldap.outputField

Attribute that should be returned in result entries