How to find and remove duplicate CRM enquiries
Duplicate requests in CRM make it difficult to understand who is already communicating with the client, which request is relevant, and where the request came from. Managers can respond to one person at a time, and the report on new requests can take into account repeated requests. Simply deleting similar cards does not solve the cause and sometimes destroys useful history.
It’s worth starting with the rules: what is considered one contact, a separate request and re-delivery of one event. Then check the data sources, prepare a copy, and only then merge the records. Below is a work procedure that can be adapted to your CRM, website form or Telegram bot.
1. Separate contacts, enquiries and delivery retries
One client can send several different requests: first order a website, then ask for support. A common telephone number in this case indicates one person, but does not prove that the second enquiry is redundant. A contact stores information about the client, a request stores a specific task, and an integration event stores the fact of data transfer.
A technical duplicate occurs, for example, when a handler repeatedly delivers one event after a connection is lost. The client’s repeated request may be meaningful: he clarified the deadlines or changed the requirements. Record these differences in processing rules. For each decision, indicate the result: associate with a contact, add a message to an existing ticket, create a new ticket, or send for manual review.
2. Find the source of duplicate records
First, make a list of channels: site forms, Telegram bot, mail, table import, manual input by the manager and external integrations. For each suspicious pair, compare the creation time, source, external event ID, and associated messages. If the cards appeared almost simultaneously from the same form, check whether the button was pressed again and the network request was repeated. Coincidence of time in itself is not a reason for deletion.
Look to see if two processors of the same form are running in parallel. A common organizational mistake is to enable a new integration while leaving the old one active. It is useful to go through the entire enquiry path from the site to the manager and note which component creates the record and which only sends the notification.
3. Define matching criteria
For technical retries, it is better to use the event or operation ID provided by the source. Normalized phone and email are suitable for searching for similar contacts, but controversial cases require verification. Bring your phone to a consistent international format while keeping the original value separate. Don’t guess the country code if there isn’t enough context. Remove random spaces in your email; do not remove dots and additional parts of the address according to the universal rule for all providers.
First and last names are auxiliary characteristics: they are not unique. A shared corporate phone can also belong to several employees. The HubSpot documentation, for example, describes different keys for contacts, companies, and importing records. This is an example of the behavior of a specific CRM, and not a rule for any channel and any system.
4. Agree record merging rules
Before making changes, decide which card will be the main one and which fields can be transferred automatically. A newer entry is not always better: the old one may contain a confirmed address, agreed terms and correspondence history. If there is a conflict between the person in charge, the transaction status or the details, it is safer to show the differences to the employee than to silently select the last value.
Separately check the connections: tasks, letters, files, invoices and other client requests. Merging contacts should not silently close an active deal or lose documents. Write down a table of decisions: a sign of a match, an acceptable action, transferred fields, and the person responsible for controversial cases. Start with a preliminary report without changing the data and check the proposed matches with your manager.
5. Prepare backups and an audit trail
Before mass processing, prepare a backup copy and specify the recovery method. Exporting a list of contacts is useful, but may not include conversations, files, or connections between objects. Make sure that the method you choose returns exactly the data that the merge will change. The general procedure is discussed in the article about backup copies and recovery verification.
For each operation, save the identifiers of the source and main records, the reason for the match, the transferred fields, time and executor. In technical journals, there is no need to duplicate complete documents and the contents of personal correspondence. First run a matched small sample and check the result. Then expand processing while maintaining the ability to stop and parse errors.
6. Prevent duplicate creation at the entry point
Blocking a button after sending helps the user, but does not protect the server from resending. Integration needs a stable operation key: when repeating the same request, the system must return the result of an already created request. Coordinate the key retention period and reuse rules with the actual delivery scenario, rather than choosing it randomly.
By MDN’s definition of idempotency repeating an operation should maintain the same intended effect. The POST method itself does not provide such a guarantee: the application and storage implement the protection. Key verification and record creation must be able to withstand simultaneous requests. A simple sequence of “find a similar card, then create” can miss two parallel hits. The exact mechanism depends on the CRM API and your integration.
7. Test real workflows
In a test environment, repeat one form submission, repeat one event delivery, and repeat parallel requests with the same key. Then test the opposite case: one client submits two different tasks. They should remain separate requests, even if they are linked to a common contact. Add scenarios with a shared family phone number, changed email and import of existing records.
After the merge, make sure that the manager sees the history, opens the files and can continue working. Check the search for old identifiers, reports and notifications: retry should not trigger unnecessary messages to the client. Record the expected result of each test before running it. This way it will be clear whether the reason for the duplicates has been corrected, and not just whether the number of cards has decreased.
Where to start checking your CRM
Collect several characteristic pairs of records and restore the path of each request: source, integration event, card creation and manager notification. Don’t start with mass deletion. First define the matching criteria and automatic merge boundaries, then prepare the copy and tests. If repeated events continue to arrive, clearing the database without changing the input processing will give a temporary result.
To connect a bot taking into account requests, parsing Telegram bot and CRM is useful. Describe in the brief your CRM, the channels for receiving enquiries and an example of the appearance of a duplicate – this will help determine the scope of integration testing without transferring the client base in open correspondence.