IDOL 12.13

# Remove Duplicates

You can configure IDOL Server to automatically prevent duplicate documents being indexed. This section describes the settings that you can use to decide how to process duplicates. You must also consider deduplication when you are choosing a distribution mode for DIH.

See Also: Deduplicate Indexed Content

## KillDuplicates

The `KillDuplicates` parameter is available as a configuration option, or as an option during indexing. If you want to prevent duplicate documents, you can use one of the following options for deduplication.

Option | Description
--- | ---
`REFERENCE` | A new document is a duplicate if the DREREFERENCE field contains the same value as an existing document.
`REFERENCEMATCHN` | A new document is a duplicate if `N` or more percent of the content is the same as an existing document.
`FieldName` | A new document is a duplicate if the `FieldName` reference field contains the same value as an existing document.

## KeepExisting

By default, when IDOL Server detects a duplicate, it indexes the new document and deletes the original. When your repository content is updated, your connector retrieves the updates, and IDOL Server indexes the updated version.

If you are sure that you do not want to overwrite an existing document, you can use the `KeepExisting` parameter for an index action. In this case, IDOL Server uses the existing version of the document, and discards the new version. This option saves time during indexing.

You can use this option if your content does not change much, or if it does not matter if users search against slightly older versions of the document.

On a blog, a user might modify a post to add a few corrections or clarifications. Most of the content does not change, so it does not matter if IDOL Server searches against an earlier version of the document.

On a News Web site, the content of each article is updated regularly as new information is made available. However, the topic of the story does not change significantly, so that the same searches retrieve the earlier or the later version of the document. In this case, it might not matter if IDOL Server has indexed the latest version.

In a legal search, there is a requirement that the search always matches against the latest version of the document. In this case, you would not use `KeepExisting`.

IDOL Server indexes new documents as normal. You might also run your index schedule such that at peak times, only new documents are indexed, but you update other content at periods of low search activity.

## Other Duplicate Actions

You can use other index action parameters to send the duplicate document to another database. For more information, refer to the IDOL Server Reference.

## Preserve Fields

When IDOL Server detects a duplicate document, you can add the content of some of the fields from one version to the other. IDOL Server overwrites the old document, but copies the specified fields to the new document.

Related Topics

- Choose a Distribution Mode

- Deduplicate Indexed Content

© Copyright 2022 Micro Focus or one of its affiliates
Version 12.13 | Last updated October 2022

Send documentation feedback

Close

Send documentation feedback

To send feedback using your default email client, open an email window.

Otherwise, send your feedback and the information below to swpdl.idoldocsfeedback@microfocus.com.

We appreciate your feedback!

Product: IDOL 12.13

Topic Title: Remove Duplicates