IDOL 12.13

# Regenerate Data Indexes

Regenerating an IDOL data indexes rebuilds the index of stored information for optimized fields (MatchType fields, NumericType fields, and so on) from the textual document field values held in the server. You can do this to reconcile the index or indexes with the document data if:

- validation of one or more indexes has failed, indicating that the data is no longer in a consistent state

- you have changed the server `[FieldProcessing]` or `[Properties]` configuration to:

- make an existing field an optimized type.

- remove an optimized field.

- set or clear the `NumericIntegerOnly` setting for a numeric type field.

In addition, you might need to regenerate fields if you have indexed `ReferenceMemoryMappedType` fields out of order, with some child fields indexed before their parents.

In these cases, the alternative to regenerating is to reindex all documents. Regenerating is generally significantly faster, as only the requested indexes regenerate.

You can regenerate a data index by using one of the following methods:

- The Regenerate*Index configuration parameters. These configuration options rebuild the index when you restart the server.
- The `DREREGENERATE` index action. This option allows you to rebuild the index while the server is running.
- The Regenerate feature in IDOL Admin (accessible from the Control section, on the Console page, Service Control tab).

The following tables describe the regeneration options for different indexes and field types.

Index Type | Validation Repair | Update Configuration | Configuration Parameter | DREREGENERATE Type
--- | --- | --- | --- | ---
`BitFieldType` |  |  | `RegenerateBitFieldIndex` | `BitField`
`CountType` |  |  | `RegenerateCountIndex` | `Count`
`GeospatialType` |  |  | `RegenerateGeospatialIndex` | `Geospatial`
`MatchType` |  |  | `RegenerateMatchIndex` | `Match`
`NumericDateType` |  |  | `RegenerateNumericDateIndex` | `NumericDate`
`NumericType` |  |  | `RegenerateNumericIndex` | `Numeric`
`ParametricRangeType` |  |  | `RegenerateParametricIndex` | `Parametric`
`ParametricType` |  |  | `RegenerateParametricIndex` | `Parametric`
`ReferenceType` |  |  | `RegenerateRefIndex` | `RefIndex`
`SecurityType` |  |  | `RegenerateSecIndex` | `SecIndex`
`SortType` |  |  | `RegenerateSortFieldIndex` | `SortField`
Unstemmed index Wildcard searches (and some advanced search techniques) use the index of unstemmed terms. This index contains all the exact terms that occur in documents. |  |  | `RegenerateUnstemmedIndex` |

## Regenerate an Index by Using DREREGENERATE

The `DREREGENERATE` index action allows you to change the field configuration and regenerate the index in one action. You can also use it to repair the Reference and Security indexes after a validation failure.

For example:

    DREREGENERATE?FieldProcessingSection=SetMatchFields&AugmentFieldCSVs=*/NewMatchField,*/SpecialMatch&DiminishFieldCSVs=*/ExistingMatchField

By default, `DREREGENERATE` automatically detects the configuration changes that you make and regenerates the indexes accordingly. You can also explicitly select the indexes to regenerate. For example:

    http://12.3.4.56:20001/DREREGENERATE?&Type=Numeric,RefIndex

For more information, refer to the IDOL Server Administration Guide and the IDOL Server Reference.

When the index action runs, IDOL runs through all stored documents, and loads the values of any fields to regenerate from disk. It inserts these values into new index structures.

When the process is complete, IDOL Server deletes any existing indexes and replaces them with the newly built versions.

## Regenerate an Index by Updating the Configuration File

When you regenerate by updating the configuration file, you must restart the server. Regenerating the indexes also results in an increased startup time.

To regenerate a data index

1. Stop IDOL Server.

2. Make any required changes to the `[FieldProcessing]` or `[Properties]` configuration sections.

3. Set the required `Regenerate*Index` settings to `True` in the `[Server]` section of the configuration file.

4. Restart IDOL Server.

At start-up, IDOL runs through all stored documents, and loads the values of any fields to regenerate from disk. It inserts these values into new index structures.

When the process is complete, IDOL Server deletes any existing indexes and replaces them with the newly built versions.

After the server has started, remember to remove or turn off any `Regenerate*Index` settings in the configuration file if you do not want the server to regenerate the indexes every time it starts.

## Performance

Regenerating the index by using `DREREGENERATE` is typically I/O intensive, because Content must reread all the documents in your index. Where it can, Content generates a second data structure in memory, and swaps over the files when it has completed the process. This option might be more memory intensive than using the configuration option, but it means that you can continue to query the existing index while regeneration is in progress.

Regenerating the index by using the `Regenerate*Index` configuration options requires a server restart. In addition, regenerating indexes also results in an increased startup time. The server is not available for further indexing or querying until the regeneration process is complete.

NOTE: Regenerating the unstemmed index is an intensive process and generally takes significantly longer than regenerating other indexes (though it is still faster than reindexing the data).

## Notes

Consider the following points when you use the regeneration options:

- The regeneration affects all indexes of the requested type. For example, if you add a field as NumericType in the configuration file, setting `RegenerateNumericIndex` to `True` regenerates all existing numeric indexes, as well as creating one for the new numeric field.

- The original document content must be stored in IDOL Server.

  If you have set `NodeTableStoreContent` to `False`, you can only regenerate a given index if all the fields in question are also explicitly configured as `StoredType`.

© Copyright 2022 Micro Focus or one of its affiliates
Version 12.13 | Last updated October 2022

Send documentation feedback

Close

Send documentation feedback

To send feedback using your default email client, open an email window.

Otherwise, send your feedback and the information below to swpdl.idoldocsfeedback@microfocus.com.

We appreciate your feedback!

Product: IDOL 12.13

Topic Title: Regenerate Data Indexes