Managing the Evolution of Information Systems with Intensional Views and Relational Algebra
1 like645 views
This document describes research into managing consistency in evolving information systems. The researchers developed an initial solution to detect inconsistencies when merging data from multiple sources for an IP phone localization case study at a university. This involved scripts to extract data from different sources and merge it daily. Numerous errors and inconsistencies were found in the individual sources and merged data. To address this, the researchers improved the solution by combining intensional views and relational algebra to define and check constraints across data tables via a user interface. This allowed expressing and detecting inconsistencies more easily. The approach was validated on the phone localization data.
![Mul8ple
errors,
inconsistencies
and
lacking
informa8on
• In
each
of
the
sources
individually
• When
merging
the
data
Errors
logged
in
files
• Difficult
to
manipulate
• Difficult
to
understand
• Difficult
to
solve
• Hard
to
see
the
«bigger
picture»
10/20
1.
Plan
2.
Intro
3.
Ini8al
Solu8on
4.
Case
Study
5.
Valida0on
6.
Improved
Solu8on
7.
Intensional
Views
8.
Valida8on
9.
Conclusion
[ERROR]
Le
local
pour
le
numéro
de
téléphone
43325
est
manquant
à
la
ligne
34.
[WARNING]
Un
déploiement
existe
déjà
pour
le
numéro
UCL
TA-‐00803
à
la
ligne
47.
[ERROR]
Iden8fiant
du
switch
non
trouvé
:
SalleOleffe01281098
(172.31.28.137)
pour
l'adresse
MAC
:
00:08:5d:35:32:cc
[ERROR]
Le
bâ8ment
pour
l'adresse
MAC
00:08:5d:
35:3c:d2
est
manquant
[ERROR]
Le
bâ8ment
Logement
456
ne
peut
être
conver8
en
un
code
ba8ment
car
ce
nom
est
inconnu
dans
la
table
buildings
[ERROR]
Le
numéro
de
téléphone
73999
n'existe
pas
dans
le
fichier
SAP
Examples
of
some
errors
and
warnings
David
Colpaert,
Kim
Mens
&
Bernard
Lambeau](https://image.slidesharecdn.com/benevol2014kimmensiv4is-141128030111-conversion-gate02/85/Managing-the-Evolution-of-Information-Systems-with-Intensional-Views-and-Relational-Algebra-11-320.jpg)
![Alf
(www.try-‐alf.org)
A
database
query
language
based
on
rela8onal
algebra
project,
restrict,
join,
union,
intersect,
minus,…
«
join_on(le6_table,
right_table,
[:mac])
»
Vues
intensionnelles
Ini8alement
des8né
à
la
maintenance
logicielle
Vérifier
des
contraintes
sur
un
code
source
Ici,
contraintes
sur
une
base
de
données
1.
Plan
2.
Intro
3.
Ini8al
Solu8on
4.
Case
Study
5.
Valida8on
6.
Improved
Solu8on
7.
Intensional
Views
8.
Valida8on
9.
Conclusion
It is also possible to define some additional filters in
order to consider only a subset of the data, for instance, only
considering one particular building. This can be useful, for
example, when analyzing very large databases with lots of
inconsistencies, and the user wants to inspect the inconsisten-cies
for a particular subset of the data only. In our particular
example, we didn’t apply any such filters.
Finally, when the user clicks on the ‘Check constraint’
button, three different Alf queries are generated. The first
one is a query to find the positive results, i.e. all tuples that
satisfy the declared constraint. A second query will calculate
the mismatches in the source table, i.e. all tuples in the source
table that do not satisfy the declared constraint. A third query
calculates the mismatches in the target table.
The generated Alf query for the positive results looks
somewhat like this:
r e s t r i c t (
join on ( source t a b l e , t a r g e t t a b l e ,
common key ) ,
eq ( : sour c e t a b l e bui l d i ng ,
: target t a b l e b u i l d i n g ) &
eq ( : source t a b le room ,
: target t a b l e r oom ) )
From this query it can be observed that the two concerned
tables are first joined based on their common key, and then
the results are restricted to the tuples satisfying all conditions,
Fig. 2. Inspecting data (in)consistencies with the highlighted view it suffices to click on the button ‘Highlighted the bottom of the intensional view editor.
Figure 2 illustrates what this highlighted view like for our previous example. It displays each of the tables, that is, the source and target tables but intermediate table used for defining the key mapping. of these tables the tuples are coloured either in correspond to an inconsistency, in green if they correspond a positive result, or just appear in white if the concerned by this particular constraint.](https://image.slidesharecdn.com/benevol2014kimmensiv4is-141128030111-conversion-gate02/85/Managing-the-Evolution-of-Information-Systems-with-Intensional-Views-and-Relational-Algebra-17-320.jpg)
Managing the Evolution of Information Systems with Intensional Views and Relational Algebra
- 1. Managing the Consistency of (Evolving) Informa8on Systems with Intensional Views and Rela8onal Algebra applied to a case of IP phone localisa1on David Colpaert, Kim Mens & Bernard Lambeau Presented at BENEVOL 2014, Amsterdam — 28 November 2014 based on David Colpaert’s Master Thesis in Computer Science at UCL, Belgium — 19 June 2014
- 2. Managing the Consistency of (Evolving) Informa8on Systems with MAY CONTAIN TRACES OF FRENCH Intensional Views and Rela8onal Algebra applied to a case of IP phone localisa1on David Colpaert, Kim Mens & Bernard Lambeau Presented at BENEVOL 2014, Amsterdam — 28 November 2014 based on David Colpaert’s Master Thesis in Computer Science at UCL, Belgium — 19 June 2014
- 3. David Colpaert, Kim Mens & Bernard Lambeau 2/20 Intro-‐ duc8on Ini8al Solu8on Case Study Valida8on Improved Solu8on Intensio-‐ nal Views Valida8on Conclusion Ini8al Solu8on Improved Solu8on
- 4. When evolving, migra8ng or merging databases, how to detect poten8al inconsistencies that may exist in the data? • Data coming from mul8ple contradictory or incomplete sources • Preferably via an easy-‐to-‐understand graphical user interface Case study : localisa8on of IP telephones at a university Need a generic tool to describe and detect consistency rules 3/20 1. Plan 2. Intro 3. Ini8al Solu8on 4. Case Study 5. Valida8on 6. Improved Solu8on 7. Intensional Views 8. Valida8on 9. Conclusion David Colpaert, Kim Mens & Bernard Lambeau
- 5. Localise IP telephones at a university in case of emergency calls • By merging data coming from different sources • Via automated scripts • While iden8fying poten8al errors in the data Generic tool for managing IP telephones Web Interface 4/20 1. Plan 2. Intro 3. Ini0al Solu0on 4. Case Study 5. Valida8on 6. Improved Solu8on 7. Intensional Views 8. Valida8on 9. Conclusion David Colpaert, Kim Mens & Bernard Lambeau
- 6. 3 sources for localisa8on data: • Via the « deployment » Excel files • Via the network (IP switches) • Via the telephone exchange system MX1 and SAP system Deployment MX1+SAP Network Merging 1. Plan 2. Intro 3. Ini8al Solu8on 4. Case Study 5. Valida8on 6. Improved Solu8on 7. Intensional Views 8. Valida8on 9. Conclusion Numéro UCL Bâ0ment Local UA-‐00001 SC16 A 001 Extract of the deployment file David Colpaert, Kim Mens & Bernard Lambeau 5/20
- 7. D. Colpaert, K. Mens & B. Lambeau 6/20
- 8. 3 sources for localisa8on data: • Via the « deployment » Excel files • Via the network (IP switches) • Via the telephone exchange system MX1 and SAP system One script per source Merging of the data from these sources Scripts and merges executed daily 20.000 lignes of code Currently used in produc8on at the university Deployment MX1+SAP Network Merging 1. Plan 2. Intro 3. Ini8al Solu8on 4. Case Study 5. Valida8on 6. Improved Solu8on 7. Intensional Views 8. Valida8on 9. Conclusion Numéro UCL Bâ0ment Local UA-‐00001 SC16 A 001 Extract of the deployment file David Colpaert, Kim Mens & Bernard Lambeau 7/20
- 10. D. Colpaert, K. Mens & B. Lambeau 9/20
- 11. Mul8ple errors, inconsistencies and lacking informa8on • In each of the sources individually • When merging the data Errors logged in files • Difficult to manipulate • Difficult to understand • Difficult to solve • Hard to see the «bigger picture» 10/20 1. Plan 2. Intro 3. Ini8al Solu8on 4. Case Study 5. Valida0on 6. Improved Solu8on 7. Intensional Views 8. Valida8on 9. Conclusion [ERROR] Le local pour le numéro de téléphone 43325 est manquant à la ligne 34. [WARNING] Un déploiement existe déjà pour le numéro UCL TA-‐00803 à la ligne 47. [ERROR] Iden8fiant du switch non trouvé : SalleOleffe01281098 (172.31.28.137) pour l'adresse MAC : 00:08:5d:35:32:cc [ERROR] Le bâ8ment pour l'adresse MAC 00:08:5d: 35:3c:d2 est manquant [ERROR] Le bâ8ment Logement 456 ne peut être conver8 en un code ba8ment car ce nom est inconnu dans la table buildings [ERROR] Le numéro de téléphone 73999 n'existe pas dans le fichier SAP Examples of some errors and warnings David Colpaert, Kim Mens & Bernard Lambeau
- 12. Mul8ple errors, inconsistencies and lacking informa8on • In each of the sources individually • When merging the data Errors logged in files • Difficult to manipulate • Difficult to understand • Difficult to solve • Hard to see the «bigger picture» 10/20 1. Plan 2. Intro 3. Ini8al Solu8on 4. Case Study 5. Valida0on 6. Improved Solu8on 7. Intensional Views 8. Valida8on 9. Conclusion MISSING DATA INCONSISTENT DATA MISSING DATA MISSING DATA INCORRECT DATA MISSING DATA Different kinds of inconsistencies David Colpaert, Kim Mens & Bernard Lambeau
- 13. 10/20 1. Plan 2. Intro 3. Ini8al Solu8on 4. Case Study 5. Valida0on 6. Improved Solu8on 7. Intensional Views 8. Valida8on 9. Conclusion Errors in the deployment files Errors in the network data Errors in the MX1+SAP data Errors in the merged data
- 14. 11/20 1. Plan 2. Intro 3. Ini8al Solu8on 4. Case Study 5. Valida8on 6. Improved Solu0on 7. Intensional Views Expressing constraints on the data • Over mul8ple tables and fields • While filtering irrelevant entries Detec8ng and inspec8ng inconsistencies • with respect to these constraints Simplicity of expression without sacrificing expressiveness 8. Valida8on 9. Conclusion David Colpaert, Kim Mens & Bernard Lambeau
- 15. 12/20 1. Plan 2. Intro 3. Ini8al Solu8on 4. Case Study 5. Valida8on Combine 2 exis8ng ideas: • Intensional Views • Rela8onal Algebra Querying with Alf Make a generic tool for defining and checking constraints over the data Via an easy-‐to-‐use user interface Valida8on by applying it to our case 6. Improved Solu8on 7. Intensional Views 8. Valida8on 9. Conclusion David Colpaert, Kim Mens & Bernard Lambeau
- 16. 1. Plan 2. Intro 3. Ini8al Solu8on 4. Case Study 5. Valida8on Intensional Views Originally designed for sovware (code) quality assurance purposes Allows expressing and verifying structural source-‐code regulari8es Reuse this idea for expressing and detec8ng database constraints 6. Improved Solu8on 7. Intensional Views 8. Valida8on 9. Conclusion
- 17. Alf (www.try-‐alf.org) A database query language based on rela8onal algebra project, restrict, join, union, intersect, minus,… « join_on(le6_table, right_table, [:mac]) » Vues intensionnelles Ini8alement des8né à la maintenance logicielle Vérifier des contraintes sur un code source Ici, contraintes sur une base de données 1. Plan 2. Intro 3. Ini8al Solu8on 4. Case Study 5. Valida8on 6. Improved Solu8on 7. Intensional Views 8. Valida8on 9. Conclusion It is also possible to define some additional filters in order to consider only a subset of the data, for instance, only considering one particular building. This can be useful, for example, when analyzing very large databases with lots of inconsistencies, and the user wants to inspect the inconsisten-cies for a particular subset of the data only. In our particular example, we didn’t apply any such filters. Finally, when the user clicks on the ‘Check constraint’ button, three different Alf queries are generated. The first one is a query to find the positive results, i.e. all tuples that satisfy the declared constraint. A second query will calculate the mismatches in the source table, i.e. all tuples in the source table that do not satisfy the declared constraint. A third query calculates the mismatches in the target table. The generated Alf query for the positive results looks somewhat like this: r e s t r i c t ( join on ( source t a b l e , t a r g e t t a b l e , common key ) , eq ( : sour c e t a b l e bui l d i ng , : target t a b l e b u i l d i n g ) & eq ( : source t a b le room , : target t a b l e r oom ) ) From this query it can be observed that the two concerned tables are first joined based on their common key, and then the results are restricted to the tuples satisfying all conditions, Fig. 2. Inspecting data (in)consistencies with the highlighted view it suffices to click on the button ‘Highlighted the bottom of the intensional view editor. Figure 2 illustrates what this highlighted view like for our previous example. It displays each of the tables, that is, the source and target tables but intermediate table used for defining the key mapping. of these tables the tuples are coloured either in correspond to an inconsistency, in green if they correspond a positive result, or just appear in white if the concerned by this particular constraint.
- 18. If 8me (and beamer) permit: video of comparing localisa8on from network and deployment sources 15/20 1. Plan 2. Intro 3. Ini8al Solu8on 4. Case Study 5. Valida8on 6. Improved Solu8on 7. Intensional Views 8. Valida8on 9. Conclusion David Colpaert, Kim Mens & Bernard Lambeau
- 19. 1. Plan 2. Intro 3. Objec8fs ini8aux 4. Cas d’étude 5. Valida8on 6. Nouveaux objec8fs 7. Vues intensionnel les 8. Valida8on 9. Conclusion
- 20. 1. Plan 2. Intro 3. Ini8al Solu8on 4. Case Study 5. Valida8on 6. Improved Solu8on 7. Intensional Views 8. Valida8on 9. Conclusion David Colpaert, Kim Mens & Bernard Lambeau 17/20
- 21. Consistency of localisa8on data (network vs. deployment) • 1322 posi8ve results • 2787/4098 (68%) nega8ves in the network source • 2057/3368 (61%) nega8ves in the deployment source Iden8cal localisa8on data in all three sources • ~1000/7512 (13%) cases Contradictory data in all three sources • 104/7512 (1%) cases Missing ayribu8ons in the network source • 701/4098 (17%) cases 18/20 1. Plan 2. Intro 3. Ini8al Solu8on 4. Case Study 5. Valida8on 6. Improved Solu8on 7. Intensional Views 8. Valida0on 9. Conclusion David Colpaert, Kim Mens & Bernard Lambeau
- 22. Achieved Objec8ves: • New approach to express, verify and visualise data constraints • Combining intensional views and rela8onal algebra Possible Improvements: • Contraints on more than two tables • Increase expressivity (aggrega8on, user-‐def. pred., devia8ons, logic queries) • Ergonomy of the user interface, efficiency improvements, … Cross Fer8lisa8on: • Think out of the box • Apply old ideas to new domains (here we applied code tools to data) • (Could DB tools also be applied to code by seeing it as structured data?) 19/20 1. Plan 2. Intro 3. Ini8al Solu8on 4. Case Study 5. Valida8on 6. Improved Solu8on 7. Intensional Views 8. Valida8on 9. Conclusion David Colpaert, Kim Mens & Bernard Lambeau
- 23. David Colpaert, Kim Mens & Bernard Lambeau 20/20