Hadoop Connector 24.2

# Store Action Queues in an External Database

Hadoop Connector provides asynchronous actions. Each asynchronous action has a queue to store requests until threads become available to process them. You can configure Hadoop Connector to store these queues either in an internal database file, or in an external database hosted on a database server.

The default configuration stores queues in an internal database. Using this type of database does not require any additional configuration.

You might want to store the action queues in an external database so that several servers can share the same queues. In this configuration, sending a request to any of the servers adds the request to the shared queue. Whenever a server is ready to start processing a new request, it takes the next request from the shared queue, runs the action, and adds the results of the action back to the shared database so that they can be retrieved by any of the servers. You can therefore distribute requests between components without configuring a Distributed Action Handler (DAH).

NOTE: You cannot use multiple servers to process a single request. Each request is processed by one server.

NOTE: Although you can configure several connectors to share the same action queues, the connectors do not share fetch task data. If you share action queues between several connectors and distribute synchronize actions, the connector that processes a synchronize action cannot determine which items the other connectors have retrieved. This might result in some documents being ingested several times.

## Prerequisites

- Supported databases:

  - PostgreSQL 9.0 or later.

  - MySQL 5.0 or later.

- On each machine that hosts Hadoop Connector, you must install an ODBC driver for your chosen database. On Linux you must also install the unixODBC driver manager and configure the name and path of the ODBC driver in the unixODBC `odbcinst.ini` configuration file.
- If you use PostgreSQL, you must set the PostgreSQL ODBC driver setting `MaxVarChar` to `0` (zero). If you use a DSN, you can configure this parameter when you create the DSN. Otherwise, you can set the `MaxVarcharSize` parameter in the connection string.

## Configure Hadoop Connector

To configure Hadoop Connector to use a shared action queue, follow these steps.

To store action queues in an external database

1. Stop Hadoop Connector, if it is running.
2. Open the Hadoop Connector configuration file.

3. Find the relevant section in the configuration file:

   - To store queues for all asynchronous actions in the external database, find the `[Actions]` section.
   - To store the queue for a single asynchronous action in the external database, find the section that configures that action.

4. Set the following configuration parameters.

   `AsyncStoreLibraryDirectory` | The path of the directory that contains the library to use to connect to the database. Specify either an absolute path, or a path relative to the server executable file.
   `AsyncStoreLibraryName` | The name of the library to use to connect to the database. You can omit the file extension. The following libraries are available: `postgresAsyncStoreLibrary` - for connecting to a PostgreSQL database. `mysqlAsyncStoreLibrary` - for connecting to a MySQL database.
   `ConnectionString` | The connection string to use to connect to the database. The user that you specify must have permission to create tables in the database. For example: `ConnectionString=DSN=ActionStore` or `ConnectionString=Driver={PostgreSQL}; Server=10.0.0.1; Port=9876; Database=SharedActions; Uid=user; Pwd=password; MaxVarcharSize=0;` If your connection string includes a password, OpenText recommends encrypting the value of the parameter before entering it into the configuration file. Encrypt the entire connection string. For information about how to encrypt parameter values, see Encrypt Passwords.

   For example:

       [Actions]
       AsyncStoreLibraryDirectory=acidlls
       AsyncStoreLibraryName=postgresAsyncStoreLibrary
       ConnectionString=DSN=ActionStore

5. You can use the same database to store action queues for more than one type of IDOL component (for example, a group of File System Connectors and a group of Media Servers). To use a database for more than one type of component, set the following parameter in the `[Actions]` section of the configuration file.

   `DatastoreSharingGroupName` | The group of components to share actions with. You can set this parameter to any string, but the value must be the same for each server in the group. For example, to configure several Hadoop Connectors to share their action queues, set this parameter to the same value in every Hadoop Connector configuration. OpenText recommends setting this parameter to the name of the component. CAUTION: Do not configure different components (for example, two different types of connector) to share the same action queues. This will result in unexpected behavior.

       For example:

       [Actions]
       ...
       DatastoreSharingGroupName=MediaServer

6. Save and close the configuration file.

   When you start Hadoop Connector it connects to the shared database.

Copyright 2024 Open Text
Version 24.2 | Last updated March 2024

Send documentation feedback

Close

Send documentation feedback

To send feedback using your default email client, open an email window.

Otherwise, send your feedback and the information below to MFI-swpdl.idoldocsfeedback@opentext.com.

We appreciate your feedback!

Product: IDOL Hadoop Connector 24.2

Topic Title: Store Action Queues in an External Database