Community Component 25.1
Community Component Help

# Deduplication Options—KillDuplicates

Use the following parameters to specify deduplication options. The Content component uses these parameters to determine whether documents match.

- The `KillDuplicates` parameter specified in either the `[Server]` section of the Content component configuration file or in the `DREADD` or `DREADDDATA` index action.

- The `KillDuplicates` parameter specified in the `[Indexing]` section of the CFS configuration file.

The following options are available for the deduplication parameters.

`NONE` | Allows duplicate documents in Content index. Content does not replace or delete documents.
`REFERENCE` | Replaces an existing document with the new document if the document to index has the same value in its `DREREFERENCE` field.
`REFERENCEMATCHN` | Replaces the existing document with the new document if the content of the document to index is more than `N` percent similar to the existing document. Content determines the similarity by comparing the content of the `SourceType` fields in the document, or the `Index` fields if no `SourceType` fields are configured. NOTE: This method can deduplicate only documents that are already synced in the Content component index. It cannot deduplicate similar documents in the same index job.
`FieldName` | Replaces the existing document with the new document if the document to index contains a `ReferenceType` field named `FieldName` that has the same content as the `FieldName` field in the existing document. You can specify multiple `ReferenceType` fields in this option (separated by a plus symbol or space), in which case Content deletes documents that contain any of the specified fields with identical content. You must percent-encode any punctuation characters in the field name. NOTE: You identify fields as `ReferenceType` fields through field processes in the Content component configuration file. If you list multiple fields in the same `PropertyFieldCSVs` parameter where you list the `FieldName` for deduplication, Content uses all the fields to eliminate duplicate documents. If you want to define multiple `ReferenceType` fields but do not want to use all fields for duplicate elimination, set up multiple field processes.
`ReferenceField,GREATER:VersionField` | Replaces the existing document with the new document if the document to index contains a `ReferenceType` field named `ReferenceField` that has the same content as the `ReferenceField` field in the existing document, and if the `VersionField` field in the document to index has a higher value than the `VersionField` in the existing document. For XML documents, you must fully qualify the path of the XML field that you want to use as the version field (you cannot use wildcard values). `VersionField` must contain a positive integer value, but you do not need to configure it as a numeric field. If only one of the incoming and current documents has a valid value in the `VersionField`, Content keeps the version with a valid `VersionField`. When both documents have the same `VersionField`, Content keeps the existing document. NOTE: When you index IDX documents, for the version comparison to work correctly, the value in the field that you use as the `VersionField` must be listed in quotation marks (`""`). That is, the field must have the following format in the IDX:
    #DREFIELD MyField="N"
Content treats existing documents with a missing or non-numeric value in the `VersionField` as having a version number of negative infinity. It treats a new document with a missing or non-numeric value in the `VersionField` as having a version number of 0.
`NOOP` (`DREADDDATA` only) | Use the `KillDuplicates` parameter in the `[Server]` section of the Content component configuration file to determine how to treat duplicate documents. NOTE: This option is available only for the `DREADDDATA` action.

Related Topics

- ReferenceType Fields

- Enable Deduplication for all Index Jobs

When you specify a deduplication option, note that:

- If you postfix any of these options with `=2`, Content applies the `KillDuplicates` process to all databases, rather than just the database into which the current IDX or XML file indexes. For example:

      KillDuplicates=REFERENCE=2

- The setting in the `KillDuplicates` option in either the `DREADD` or `DREADDDATA` index action overrides the setting in the `KillDuplicates` configuration parameter.

Copyright 2024 Open Text
Version 25.1 | Last updated January 2025

Send documentation feedback

Close

Send documentation feedback

To send feedback using your default email client, open an email window.

Otherwise, send your feedback and the information below to MFI-swpdl.idoldocsfeedback@opentext.com.

We appreciate your feedback!

Product: Knowledge Discovery Community Component 25.1

Topic Title: Deduplication Options—KillDuplicates