2. Sample of dirty XML data generation
Suppose there is a clean XML file persons.xml containing a set of persons each with a name, an address and some additional data.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
| <?xml version="1.0"?>
<persons>
<person ID="1" age="57" sex="m">
<firstname>Henry</firstname>
<lastname>Evening</lastname>
<address>157 Short St., Sydney NSW 2113, Australia</address>
</person>
<person ID="2" age="21" sex="f">
<firstname>Olga</firstname>
<lastname>Tschernikova</lastname>
<address>101000 Moscow, Russia</address>
</person>
<person ID="17" sex="f">
<firstname>Maria</firstname>
<lastname>Lamprecht</lastname>
<address>Meiereiplatz 27, 16223 Mehrow, Germany</address>
</person>
</persons>
|
In order to test a certain algorithm, we want to create a dirty XML file based on the clean one. For instance, it should contain some duplicates and perhaps misses a specified amount of data values. In addition, the values of some attributes and the text content of a couple of elements should contain misspellings and data errors.
To define how to pollute the data in the way described above, we write an XML file persons_params.xml containing a set of parameters:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
| <?xml version="1.0"?>
<dirtyXMLparameters
xmlns:xs="http://www.w3.org/2001/XMLSchema-instance"
xs:noNamespaceSchemaLocation="../dist/DirtyXMLParameters.xsd"
valid4Desc="true"
errorsInAncestors="false">
<algo name="swap1" baseAlgo="swapChars">
<parameter name="includeFirstChar" value="true"/>
<parameter name="includeLastChar" value="true"/>
<parameter name="minSwaps" value="1"/>
<parameter name="maxSwaps" value="2"/>
</algo>
<algo name="swap2" baseAlgo="swapChars">
<parameter name="includeFirstChar" value="false"/>
<parameter name="includeLastChar" value="true"/>
<parameter name="minSwaps" value="2"/>
<parameter name="maxSwaps" value="4"/>
</algo>
<algo name="del1" baseAlgo="deleteChar">
<parameter name="includeFirstChar" value="false"/>
<parameter name="includeLastChar" value="true"/>
</algo>
<algo name="ins1" baseAlgo="insertChar">
<parameter name="includeFirstChar" value="false"/>
<parameter name="includeLastChar" value="true"/>
<parameter name="includeUpper" value="false"/>
<parameter name="includeLower" value="true"/>
<parameter name="includeDigits" value="false"/>
</algo>
<dupElement name="person" delProb="0" dupProb="100" maxDup="1">
<attribute name="sex">
<chars delProb="30" changeProb="75">
<changeAlgo algoName="swap1" useProb="100"/>
</chars>
</attribute>
<dupElement name="firstname" delProb="25" dupProb="60" maxDup="2">
<chars delProb="30" changeProb="80">
<changeAlgo algoName="del1" useProb="50"/>
<changeAlgo algoName="ins1" useProb="50"/>
</chars>
</dupElement>
<dupElement name="address" delProb="0" dupProb="30" maxDup="1">
<chars delProb="0" changeProb="70">
<changeAlgo algoName="swap2" useProb="80"/>
<changeAlgo algoName="del1" useProb="20"/>
</chars>
</dupElement>
</dupElement>
</dirtyXMLparameters> |
In the first part of the parameter file (lines 8 to 33), four algorithms are defined that rest upon the three base algorithms SwapChars, DeleteChar, and InsertChar. These parameterised algorithms will be applied to character data contained in elements and attribute values.
In line 6 we define by means of errorsInAncestors="false" that the original elements from which the duplicates originate should not be polluted.
The second part of the file (beginning with line 35) constitutes the elements from which duplicates should be created.
Some examples: The element person will never be deleted but duplicated every time it occurs in the source file. The maximum number of duplicates generated is 1. The attribute with the name sex is contained in the person element. Its string values will be deleted with a probability of 0.3 and, if not deleted, will be changed with an 0.75 probability using the swap1 algorithm defined above. As it is the only algorithm used to pollute this attribute, the usage probability is 1. In the other case there is the element address, whose chars will be polluted using two different algorithms ( swap2 and del1) used with a probability of 0.8 and 0.2, respectively. Note that the probabilities must add up to 1 (that means 100%).
For a detailed explanation of the parameters please have a look at the Detailed Documentation.
Executing the Dirty XML Data Generator with the clean XML file, the parameter XML file, and the name of the dirty XML file (here: persons_dirty.xml) as input leads to the following result:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
| <?xml version="1.0" encoding="UTF-8"?>
<persons>
<person ID="1" age="57" sex="m">
<firstname>Henry</firstname>
<lastname>Evening</lastname>
<address>157 Short St., Sydney NSW 2113, Australia</address>
</person>
<person ID="2" age="21" sex="f">
<firstname>Olga</firstname>
<lastname>Tschernikova</lastname>
<address>101000 Moscow, Russia</address>
</person>
<person ID="17" sex="f">
<firstname>Maria</firstname>
<lastname>Lamprecht</lastname>
<address>Meiereiplatz 27, 16223 Mehrow, Germany</address>
</person>
<person ID="1" age="57" sex="m">
<firstname>Henry</firstname>
<lastname>Evening</lastname>
<address>157 Short St., Sydney NSW 2113, Australia</address>
<firstname>Henry</firstname>
</person>
<person ID="2" age="21" sex="f">
<firstname>Olga</firstname>
<lastname>Tschernikova</lastname>
<address>101000 Moscow, Russia</address>
<firstname>Oga</firstname>
<address>101000 oMscow ,Russia</address>
</person>
<person ID="17">
<firstname>Maria</firstname>
<lastname>Lamprecht</lastname>
<address>Meiereiplatz 27, 16223 Mehrow, Germany</address>
</person>
</persons>
|
In the first 17 lines you will see the person elements of the source file that have not been polluted (as requested with the errorsInAncestors attribute in the root elements of the parameter file). The lines 18 to 35 contain the same elements, but polluted (we defined a duplication probability of 1 and that at most one duplicate should be created). They contain the dirty data according to the parameters.
^ zur Inhaltsübersicht
