BlackSwan (*)

Automated annotation of global statistics

Contact

Organizational Info

website: http://www.blackswanevents.org
Monday, 15:15 - 16:45 (exceptions will be announced)
room A-2.2
Wiki (internal)
Dates and Material

Statistical Data

There are numerous sources for statistical data sets: companies, government agencies, international organizations, etc. Statistical data is usually

of numerical type
collected in fixed intervals over a certain time period, and
reveals certain short-term and long-term trends.

Event Data

Event data can be gathered from various (mostly unstructured) sources, such as Wikipedia, Freebase, News Archives, and so on. In the context of this seminar, an event is described by its

type,
location, and
(starting) point in time.

Augmenting Statistical Data

Our goal is to detect certain trends in statistical data and automatically relate them to the historical events that triggered these events based on specific rules previously learned by our system.

Proposed Architecture

The following diagram depicts a possible architecture for the system. The green boxes each represents a specific component a team of students will be working on.

Team structure and tasks

	Event Extractor	Time Series Analyzer	Rule System	Visualizer
Tasks	gather data sources text extraction information integration choose data format	gather data sources information integration regression / time series analysis statistics	association rule mining machine learning time shift evaluation	find visualization framework implement interactive GUI data export

Important Dates and Material

Date	Event	Slides
<strike>18.10.10</strike>	first meeting
<strike>22.10.10,<br>23:59 CEST</strike>	deadline for sending participation request mail to Johannes include your preferred focus (if any): extraction or analysis include your preferred amount of "Leistungspunkte": 3 or 6	n.a.
<strike>24.10.10,<br>20:00 CEST</strike>	participation confirmation mails are sent out	n.a.
<strike>25.10.10</strike>	announcement of team set-up discussion of first steps data model event classes statistical datasets
<strike>01.11.10</strike>	tutorial statistical analysis speaker: Prof. Liero start time: 15:15
<strike>08.11.10</strike>	tutorial information extraction speaker: Prof. Leser start time: 16:15
<strike>15.11.10</strike>	short overview (30 min) by extraction team of proposed datasets proposed extraction algorithms proposed data format proposed frameworks first results/implementations? short overview (30 min) by statistics team of proposed datasets proposed analysis algorithms proposed frameworks first results/implementations?	n.a.
<strike>29.11.10</strike>	informal progress report	n.a.
<strike>10.12.10</strike>	presentation of preliminary results statistics team start time: 15:15 (room A-2.2, as usual) 10 minutes per speaker
<strike>13.12.10</strike>	presentation of preliminary results extraction team start time: 15:15 10 minutes per speaker
<strike>03.01.11</strike>	"naïve" join of statistics and event data on "date" attribute should be possible by now no meeting on this day!	n.a.
<strike>10.01.11</strike>	team set-up for second phase
<strike>17.01.11</strike>	tutorial data mining
<strike>24.01.11</strike>	no meeting on this day!	n.a.
<strike>31.01.11</strike>	short overview of next steps by visualization team + approx. 3 members of rule system team (no slides)	n.a.
<strike>07.02.11</strike>	short overview of next steps by remaining rule system team (no slides)	n.a.
<strike>31.03.11</strike>	hand in source code hand in "BlackSwan advertisement"	n.a.
<strike>11.04.11,<br>17:00</strike>	final presentations (rule mining)
15.04.11, 15:00	final presentations (visualization)

Literature

Data Mining / Association Rules

Agrawal et al.: Mining association rules between sets of items in large databases, SIGMOD 1993
Agrawal et al.: Fast Algorithms for Mining Association Rules, VLDB 1994
Tan et al.: Selecting the right objective measure for association analysis, Information Systems 2004
Knorr et al.: Algorithms for Mining Distance-Based Outliers in Large Datasets, VLDB 1998
Han et al.: Data Mining: Concepts and Techniques, Morgan Kaufmann 2005 (can be found in our group library)
Segaran: Kollektive Intelligenz: analysieren, programmieren und nutzen, O'Reilly 2008 (can be found in our group library)
Witten, Frank: Data Mining. Practical Machine Learning Tools and Techniques, Morgan Kaufmann 2005 (can be found in our group library)

Information extraction

Sarawagi: Information Extraction, Now Publishers Inc. 2008

Statistical analysis

Adler: R in a nutshell, O'Reilly 2010 (can be found in our group library)

Links

(*) http://en.wikipedia.org/wiki/Black_swan_theory