Bringing Back Structure to Free Text Email Conversations with Recurrent Neural Networks

Project Description

Email communication plays an integral part of everybody's life nowadays. Especially for business emails, extracting and analysing these communication networks can reveal interesting patterns of processes and decision making within a company. Fraud detection is another application area where precise detection of communication networks is essential. In this paper we present an approach based on recurrent neural networks to untangle email threads originating from forward and reply behaviour. We further classify parts of emails into 2 or 5 zones to capture not only header and body information but also greetings and signatures.

We use the model presented in our ECIR paper in QuaggaLib. This library parses the raw email body into separate blocks and extracts meta-data from inline-headers. This kind of pre-processing should be used in all applications using email data. The library provides the actual written text content as well as the meta-data that would otherwise be hidden in the unstructured email body.

Reference

If you use our data or find this work related to yours, please cite us as...

Repke, T., Krestel, R.: Visualising Large Document Collections by Jointly Modeling Text and Network Structure.Proceedings of the Joint Conference on Digital Libraries (JCDL). (2020).
[ Abstract ] [ BibTeX ] [ Download ]

Many large text collections exhibit graph structures, either inherent to the content itself or encoded in the metadata of the individual documents. Example graphs extracted from document collections are co-author networks, citation networks, or named-entity-cooccurrence networks. Furthermore, social networks can be extracted from email corpora, tweets, or social media. When it comes to visualising these large corpora, either the textual content or the network graph are used. In this paper, we propose to incorporate both, text and graph, to not only visualise the semantic information encoded in the documents' content but also the relationships expressed by the inherent network structure. To this end, we introduce a novel algorithm based on multi-objective optimisation to jointly position embedded documents and graph nodes in a two-dimensional landscape. We illustrate the effectiveness of our approach with real-world datasets and show that we can capture the semantics of large document collections better than other visualisations based on either the content or the network information.
@article{repke2020visualising, abstract = {Many large text collections exhibit graph structures, either inherent to the content itself or encoded in the metadata of the individual documents. Example graphs extracted from document collections are co-author networks, citation networks, or named-entity-cooccurrence networks. Furthermore, social networks can be extracted from email corpora, tweets, or social media. When it comes to visualising these large corpora, either the textual content or the network graph are used. In this paper, we propose to incorporate both, text and graph, to not only visualise the semantic information encoded in the documents' content but also the relationships expressed by the inherent network structure. To this end, we introduce a novel algorithm based on multi-objective optimisation to jointly position embedded documents and graph nodes in a two-dimensional landscape. We illustrate the effectiveness of our approach with real-world datasets and show that we can capture the semantics of large document collections better than other visualisations based on either the content or the network information.}, author = {Repke, Tim and Krestel, Ralf}, booktitle = {Proceedings of the Joint Conference on Digital Libraries (JCDL)}, keywords = {sys:relevantfor:isg hpi business_communication web_science modir isg}, title = {Visualising Large Document Collections by Jointly Modeling Text and Network Structure}, year = 2020 }
Repke, T., Krestel, R.: Bringing Back Structure to Free Text Email Conversations with Recurrent Neural Networks.40th European Conference on Information Retrieval (ECIR 2018). Springer, Grenoble, France (2018).
[ Abstract ] [ BibTeX ] [ URL ] [ Download ]

Email communication plays an integral part of everybody's life nowadays. Especially for business emails, extracting and analysing these communication networks can reveal interesting patterns of processes and decision making within a company. Fraud detection is another application area where precise detection of communication networks is essential. In this paper we present an approach based on recurrent neural networks to untangle email threads originating from forward and reply behaviour. We further classify parts of emails into 2 or 5 zones to capture not only header and body information but also greetings and signatures. We show that our deep learning approach outperforms state-of-the-art systems based on traditional machine learning and hand-crafted rules. Besides using the well-known Enron email corpus for our experiments, we additionally created a new annotated email benchmark corpus from Apache mailing lists.
@inproceedings{repke-ecir18, abstract = {Email communication plays an integral part of everybody's life nowadays. Especially for business emails, extracting and analysing these communication networks can reveal interesting patterns of processes and decision making within a company. Fraud detection is another application area where precise detection of communication networks is essential. In this paper we present an approach based on recurrent neural networks to untangle email threads originating from forward and reply behaviour. We further classify parts of emails into 2 or 5 zones to capture not only header and body information but also greetings and signatures. We show that our deep learning approach outperforms state-of-the-art systems based on traditional machine learning and hand-crafted rules. Besides using the well-known Enron email corpus for our experiments, we additionally created a new annotated email benchmark corpus from Apache mailing lists.}, author = {Repke, Tim and Krestel, Ralf}, booktitle = {40th European Conference on Information Retrieval (ECIR 2018)}, keywords = {business_communication web_science deep_learning isg}, month = {March 26--29}, publisher = {Springer}, title = {Bringing Back Structure to Free Text Email Conversations with Recurrent Neural Networks}, year = 2018 }

Implementations

Production-ready email parsing:
- https://github.com/HPI-Information-Systems/QuaggaLib
Reference implementation as used in the paper including competitor approaches and data:
- https://github.com/HPI-Information-Systems/Quagga

Datasets

On this page we provide datasets used in our ECIR 2018 paper and a fully parsed Enron corpus. Data was manually annotated using our Enno tool.

newly collected ASF email corpus, annotated by email zones only
selection of Enron corpus, annotated by email zones only
selection of Enron corpus, detailled annotation (including names, aliases, metadata)
automatically split, normalised, and cleaned Enron corpus as graph

Apache Software Foundation Emails (ASF)

Annotated Enron Emails

Fully Parsed Enron Graph

Related Work

Original Code for Jangada, Carvalho, 2004
More infos and data for Jangada (600+ annotated mails in 20 newsgroup dataset)
MinorThird Library used by Jangada
400 annotated emails by Lampert et. al (Enron data)
Zebra System for email zoning
Another implementation of Zebra
Talon is an awesome universal tool for everything that has to do with email structure

Bringing Back Structure to Free Text Email Conversations with Recurrent Neural Networks

Project Description

Reference

Implementations

Datasets

Apache Software Foundation Emails (ASF)

Sampling

Format

Data

Annotated Enron Emails

Sampling

Format

Data

Fully Parsed Enron Graph

Related Work

Chair

News

16.01.2021 | Paper accepted at WWW 2021

06.01.2021 | Congratulations Julian Risch!

07.12.2020 | Paper accepted at CHIIR 2021

25.11.2020 | Congratulations Ioannis Koumarelas!

10.11.2020 | Paper accepted at FAPER workshop in conjunction with ICPR 2020

Project highlights

People and open positions