IDOL 12.13

# Reduce Memory Usage

In many situations, the memory usage of an IDOL instance can become the limiting factor in its sizing. In this case, reducing the memory usage by editing the configuration often allows you to index more documents per instance.

The MemoryReport action returns the usage of each memory index, and is the key to determining how to alter your configuration. However, reducing the largest index does not always give the best return, because some changes affect performance more than others. This section lists the primary indexes to help you to determine a strategy for reducing your memory usage.

TIP: You can view information about system and process memory usage on the Memory and Memory Map tabs on the Performance page in the Monitor section of IDOL Admin. You can also view overall real-time performance data on a single graph on the Real Time tab.

NOTE: In general, limiting the memory usage for a particular index forces the server to swap the information in and out of memory when needed, and the resulting disk reads will affect your performance. This is not the case for the index and term caches.

## Index Cache

The index cache holds information on terms in index fields, to minimize the time it takes to flush the cache to disk, which in turn improves the indexing performance.

For many IDOL Servers, you index data in two phases: the first phase indexes a large amount of historic documents, usually before the server reaches production. The second phase slowly increases the index by adding new documents over time. In such a case, Micro Focus recommends that you set the index cache to a high value (perhaps 1 GB or more) during the first phase, and then lower it (to perhaps 100 MB) for the second phase to free up memory.

You can dynamically reduce the size of the index cache by using the `DRERESIZEINDEXCACHE` index action. Alternatively, you can resize the cache on the Caches tab of the Status page in IDOL Admin.

NOTE: The `DRERESIZEINDEXCACHE` index action does not update the value in your configuration file. If you restart IDOL Server, it reverts to using the value in the configuration file.

## Nodetable

The `nodetable` index stores information to optimize the loading of document fields. Its size is proportional to the number of documents in the server. Micro Focus recommends that you only limit this index if you only rarely load fields during queries, or if query performance is not your primary concern.

## Numeric, Match, Sort, Parametric

The `numeric`, `match`, `sort` and `parametric` structures store information about fields that you configure as `NumericType`, `MatchType`, `SortType`, or `ParametricType`. The sizes of these structures are proportional to the number of documents in the server, and the average number of values of each type in a document.

The first consideration in limiting any of these structures is to ensure that you keep the number of fields of each type to a minimum. To view the fields of a given type use the `GetTagNames` action (or use the Field Types page in the Monitor section of IDOL Admin). You can use the output to remove any unnecessary fields from the configuration.

The following action returns a list of all the `NumericType` fields:

    action=GetTagNames&FieldType=numeric

You can limit the usage of each structure further by using the `NumericMemoryMaxSize`, `SortFieldMemoryMaxSize`, and `ParametricMemoryMaxSize` configuration parameters. It is often possible to make these changes without greatly affecting query performance. However, Micro Focus recommends that you test performance for a number of values to determine the impact.

NOTE: Setting `NumericMemoryMaxSize` also reduces memory usage of `MatchType` fields, and also that different numeric fields can have memory reduced by different amounts by using the `NumericNormalMaxMem` configuration parameter.

For parametric fields, Micro Focus recommends that you always set the `ParametricMaxPairsPerDocument` configuration parameter to zero to reduce memory. This parameter affects performance only for the rarely used `GetTagValues` action.

## Unstemmed

The unstemmed index is used for wildcard expansion and spelling correction, and its size is proportional to the number of terms stored in it. The recommended method for reducing its size is to limit the number of unstemmed terms that IDOL Server stores.

You can remove extremely rare terms from the unstemmed index by increasing the `UnstemmedMinDocOccs` configuration parameter.

Often, the majority of terms in the unstemmed index are numeric or mixed alphanumeric terms. You can control which terms to match with wildcard terms by using the `UnstemmedIndexNumbers` configuration parameters, or `IndexNumbersMaxValue` and `IndexNumbersNTruncateLength`. These parameters prevent long numeric terms from taking up space.

The `UnstemmedMemoryMaxSize` parameter configures the amount of memory used to memory map portions of the `dbub.dat` file (part of the unstemmed index). By default there is no memory limit and the entire file can be mapped into memory over time.

NOTE: The `dbun.dat` file is always wholly memory mapped and cannot be memory limited.

## Refindex

The `refindex` controls the mapping of a reference field to its internal `docid`, and its size is proportional to the number of documents in the server and the number of reference fields in each document. Its size is usually not large compared to other indexes, and so Micro Focus recommends that you limit it only if you rarely use reference fields in queries.

## Term Cache

The term cache stores information on terms in index fields to optimize a query containing that term.

The only situation in which you might want to use the term cache is when you have a large amount of memory available, and your queries often use frequently occurring terms. In this case, you can set the amount of memory to use with the `TermCachePersistentKB` configuration parameter.

© Copyright 2022 Micro Focus or one of its affiliates
Version 12.13 | Last updated October 2022

Send documentation feedback

Close

Send documentation feedback

To send feedback using your default email client, open an email window.

Otherwise, send your feedback and the information below to swpdl.idoldocsfeedback@microfocus.com.

We appreciate your feedback!

Product: IDOL 12.13

Topic Title: Reduce Memory Usage