Hasso-Plattner-Institut
Prof. Dr. Felix Naumann
 

What is SNNDedupe?

The integration of multiple data sources is a common problem in a large variety of applications. Traditionally, handcrafted similarity measures are used to discover, merge, and integrate multiple representations of same entities into large data collections. Often, these similarity measures do not cope well with the heterogeneity of the underlying dataset. In addition, domain experts are needed to manually generate such measures, which is both time-consuming and requires extensive domain expertise.

We propose SNNDedupe, a deep Siamese neural network, capable of learning a similarity measure that is tailored to the characteristics of a particular dataset. With the properties of deep learning methods, we are able to eliminate the manual feature engineering process and thus considerably reduce the effort required for model construction. In addition, we show that it is possible to transfer knowledge acquired during the deduplication of one dataset to another, and thus significantly reduce the amount of data required to train a similarity measure. We test our method on multiple datasets and compare our approach to state of the art methods. Our approach outperforms competitors by up to +26%, depending on task and dataset. In addition, we show that knowledge transfer is not only feasible, but in our experiments led to an improvement in F-measure of up to +4.7%.

Key Contributions

  • We present a siamese neural network for deduplication that facilitates knowledge transfer.
  • We achieve both competitive deduplication performance and knowledge transfer, by combining character embeddings with a special network design.
  • We show how to transfer previously learned knowledge to support the deduplication of new datasets.
  • We analyze the impact of transfer learning strategies on the performance of the neural network.

Performance

Deduplication with increasing data
Record linkage with increasing data
Performance with and without knowledge transfer

Datasets

Dataset All Duplicates Non-Duplicates
CD
Census
Cora
Restaurants

Remarks

Due to copyright restrictions, we regret that we cannot release the Movies dataset.

Contact

For questions or feedback, please contact Michael Loster.