Remove Duplicate Data with a Process Shape

Last updated Mon Jul 25 2022

Set yourself up for success with this solution, which includes:

  • Steps for converting a document into a Flat File
  • How to combine multiple documents into one
  • Steps to configure a Data Process shape to spot duplicates


One of the major challenges with data today is the issue of duplicate data. It can be misleading in so many ways and cause pain-points for any kind of business, small or large. Inaccurate reporting, missed sales opportunities, and lost productivity are only a few of the potentially devastating obstacles duplicate data can cause.

In this recipe, learn how to remove duplicates by using Boomi’s Data Process shape. The recipe starts off by pulling in multiple XML documents that are then converted into a flat files. From there, the documents are combined into one flat file document via the Data Process shape and then they are sent to a document cache that can be referenced at any point. The Document Cache is configured to only pull one line from a flat file, which will eliminate the possibility of introducing duplicate data.

Locating and eliminating duplicate data is a major piece in the ever expanding data integrity puzzle for businesses. Use Boomi Integration to help ease your businesses data duplication issues! Check out another way to remove duplicates by using a Map shape and a Profile Element here.

The recipe qualifies for Boomi’s Recipe Program.

Qualifying customers get access to Boomi’s Recipe Program* with the purchase of a Boomi subscription. As part of the Recipe Program you have direct access to a Boomi specialist to get you up and running with Boomi at no additional cost. Learn more

*Offer applies only to new customers and requires purchase of a qualifying Boomi subscription.

Related Solutions


Get Started with Structuring Complex Objects and Lists in Flow

Learn the importance of Types, Object Values, and List Values in flow and how to structure data in your flow build.
Learn more


Simply Vision: Boomi Monitoring Solution

Monitor your deployed assets and accurately forecast support requirements with Boomi!


Oracle Cloud ERP Integration Accelerator – HCM/HR: Employee by Apps Associates

Automatically load employee data into Oracle Cloud ERP and connect to HCM/HR systems with Boomi.