Sampling

For details on sampling, please look at the scripts and logs here. We samples emails from mailing lists of the Apache Software Foundation via http://mail-archives.apache.org/mod_mbox/, specifically 50 for flink-user for evaluation, 100 from goovy-users for testing, and 250 from hadoop-user for training. Therefore we randomly selected the last mail of a thread within 2017.

Format

Each *.txt file is the original email downloaded as described above. Each *.ann file is created by Enno and is a json file with "text" containing the full original email, an "id", empty "meta" data, and a list of "denotations". These are the zomes of the email thread, whereas each has an "id", a "start" and "end" character offset, the according "text", a "type" and empty "meta" data. 

Types are: Header, Body, Body/Intro, Body/Outro, Body/Signature. Intro and outro are phrases such as "Dear all" and "Thank you, Tim". The Body includes these phrases and signatures.

Data

download here

Sampling

For details on sampling, please look at the scripts and logs here. This is based on the Enron corpus provided by the CMU.

Format

Same as for the ASF dataset. Additionally, the detailled annotation contains denotations for all semi-structured metadata in the email, for example the from and to fields in inline-headers down to the level where email addresses and name ares are individual denotations. Also the names in the Body/Intro and Body/Outro and Signatures are marked. The "meta" field contains the parsed email header. There is also a list of "relations", where each has an "id", an "origin" and "target referring to the denotation id, for example when an email address is an alias for a person's name. There are relation types for "ContactInfo", "Alias", "WorksFor".

This data can potentially be used to create a high detailed parser to extract lots of hidden metadata from email thread text.

Data

• Enron Data with detailled annotation (~400 emails)
• Enron Data with zone annotation (as ASF, ~800 emails)
• Enron Data with rudimentary Header/Body annotation (~170 emails)

Using our QuaggaLib and some additional processing, we create a new Enron corpus. This is a more truthful corpus, because

• each email only appears once, 
• emails are stripped to only contain actual text
• people are disambiguated
• meta data is extracted and can be used directly

Code is also in the repository above and here.

The corpus is available as

• GraphML
• JSON, one email per line (part a, part b; simply concatenate and untar)
• List of people