VO web-services-based astronomy workflows
1 like•989 views
- The document discusses the Virtual Observatory (VO), which aims to make astronomical data and services interoperable. - It describes various VO standards for data discovery, access, and sharing including web services, data models, and ontologies. - It notes that upcoming large surveys will generate huge datasets and require distributed computing architectures like grids and clouds to handle storage and processing.
VO web-services-based astronomy workflows
- 1. VO Web-services-based astronomy workflows! Jose Enrique Ruiz! IAA - CSIC! Manchester 13th July 2011!
- 2. IAA - CSIC!
- 3. Wf4Ever! Curating and preserving collaborative digital experiments 1. Intelligent Software Components (ISOCO, Spain)! 2. University of Manchester (UNIMAN, UK)! 2 7 3. Universidad Politécnica de Madrid (UPM, Spain)! 5! 4! 4. Poznan Supercomputing and Networking Centre (PSNC, Poland)! 5. Universisty of Oxford (OXF, UK)! 6. Instituto Astrofísica Andalucía (IAA-CSIC, Spain)! 1! 3! 7. Leiden University Medical Centre (LUMC, NL)! 6!
- 4. Who are you ?! The AMIGA Group! Analysis of the interstellar Medium of Isolated Galaxies! ! Statistical baseline of isolated galaxies to compare! with the behaviour of galaxies in denser environments! Multi study of ~1000 galaxies! ! Instituto Astrofisica de Andalucia - CSIC! Univ . Granada, Obs. Marseille, Obs. Paris, ! NAOJ, FCRAO, UNAM, Univ. Edinburgh, ! IRAM, ESO, Kapteyn Astronomical Institute.! ! P.I. Lourdes Verdes-Montenegro! http://amiga.iaa.es!
- 5. Who are you ?! VO Virtual Observatory! • International Virtual Observatory Alliance (IVOA)! • Interoperability and Discovery! • Publishing and Accessing Data! • Service Oriented Architecture (SoA)! • Integration of Software and Data! • Distributed Resources! • Panchromatic Astronomy! • Data Models! • Web Services! • Semantics! !
- 6. Who are you ?! VO Virtual Observatory! !
- 7. Who are you ?! The AMIGA VO Catalog! The Data Provider!
- 8. Who are you ?! RADAMS! Radio Astronomy Data Model for Single-Dish telescopes !
- 9. Who are you ?! RADAMS Implementation
- 10. Who are you ?! VO Archives Developments Robledo DSS-63! • Madrid Deep Space Communication Complex (MDSCC)! • 70m single dish in Robledo de Chavela (Madrid)! • 5% operational time for observations! • K band Spectra (18 - 26 GHz)! • H2O Masers, methanol, NH3,..! ! ! TAPAS – IRAM 30m! • Telescope Archive for Public Access System! • Bolometric observations, maps, spectra! • Rotational molecular transitions! • ~200 scientific projects / year, 1TB! Radio Astronomy DAta Model for Single-dish telescopes!
- 11. Who are you ?! The AMIGA Group! Analysis of the interstellar Medium of Isolated Galaxies! ! Statistical baseline of isolated galaxies to compare! with the behaviour of galaxies in denser environments! ! Multi study of ~1000 galaxies! +! Need of intensive and complex analysis of 3D data! 2D spatial + 1 Velocity!
- 12. Who are you ?! Velocity Datacubes! ! M. Krips – ESO 3D2008 Workshop – Garching!
- 13. Who are you ?! GIPSY! Groningen Image Processing SYstem! Connectivity ! • VO Archives ! • VO Software! ! Accessibility! • Usability GUI! • VO Web Services! ! Kapteyn Astronomical Institute! IAA - CSIC!
- 14. Who are you ?! B0DEGA Below 0 DEgrees GAlaxies! P.I. : D. Espada! Legacy project of Submillimiter Array interferometer (SMA)! http://b0dega.iaa.es! ! IAA-CSIC! CfA (Harvard-Smithsonian Center for Astrophysics)! ASIAA (Institute of Academia Sinica Astronomy and Astrophysics) ! ! Molecular gas properties of a survey of nearby galaxies.! 30 processed and reduced datacubes of galaxies!
- 15. Who are you ?! The B0DEGA 3D VO Catalog! The Data and Service provider! Aladin VO Software!
- 16. The Virtual Observatory! The Virtual Observatory! Infrastructure of interoperable data and services. Standards for:! • Providers to share data and services! • Developers to discover the services, find and access the data! Goal: astronomers to use this infrastructure in a seamless way!
- 17. The Virtual Observatory! Standards for Web Services! • Most of the Web Services in Astronomy! • They are registered and curated ! • VO Registry! • WS for Humans! • Data discovery and data access! • Accessed with local software (Europe)! • Integrated in web portals (USA)! • WS for Machines! • Storage, transport, authentication, etc.!
- 18. The Virtual Observatory! The VO Registry! • If you are not registered, you are not in the VO! • Web forms to register services! • Three VO Registries! • Euro-VO! • National Virtual Observatory (USA)! • AstroGrid (UK)! • Harvesting among registries! • A VO Registry register resources! • Organizations! • Authorities! • Data collections! • Services!
- 19. The Virtual Observatory! WS for Humans! • Most WS provide “just” Data Discovery and Access! • Associated to a very specific Archive! • Designed to discover! • VO Services! • Catalogs! • Images! • Spectra! • Parameters-based -> Standards! • Responses are always VOTables! • Characterization of data! • Actual data values ! • List of services ! • Spreadsheets for catalogues! • Links to binaries for images and spectra!
- 20. The Virtual Observatory! WS for Humans! • Sesame name resolver is one of the most used! • Resolves objects names into coordinates! • Provided by Centre de Données de Strasboug (CDS) ! • Data Discovery and Access (RESTful)! • ConeSearch! • Simple Image Access! • Simple Spectra Access! • Parameters: RA, DEC, SIZE ! • Table Access Protocol (TAP), OpenSkyQuery, SkyNodes! • Astronomical Data Query Langage (ADQL) requests! • Sparse complex services (SOAP)! • Mosaicing of images, footprint of regions, spectral building and fitting, principal components analysis in spectra..! • Common Execution Architecture (AstroGrid)- not took off!
- 21. The Virtual Observatory! WS for Machines! • Implementation in progress! • More standards than implemented services! • Universal Worker Service (Grid oriented)! • asynchronous! • stateful! • job oriented services! • VOSpace! • distributed storage! • will be provided for Big Data archives! • Single Sign-On and Credential Delegation! • Registry Interfaces: services acting on the Registry!
- 22. The Virtual Observatory! VOSI! • VO Services Support Interface (REST binding)! • In progress of implementation! • Provides interoperability among services! • Common Contract for all VO services! • Self-descriptive services! "- operations and data! /capabilities /tables! - state of the service ! /availability /upSince /downAt /backAt /note! • XML/VOTable VOSI files! • VOSI files stored in service provider server! • Files are scanned by VO Regrisries! • Provide also state of the service!
- 23. The Virtual Observatory! VOTables! ! XML Format! • Characterization of Data! • Semantics! • UCDs (Universal Content Descriptors)! • Data Models! • UTypes! • Actual Data! • Tabular data! • Links to binary data!
- 24. The Virtual Observatory! Ontologies, SKOS Vocabularies! M16!
- 25. The Virtual Observatory! Ontologies, SKOS Vocabularies!
- 28. VO Software! Aladin Sky Atlas!
- 31. A Cloud of Services! The next generation of archives! ! Much wider FoV and spectral coverage! • Large volumes for an observed datacube! • Subproducts are Virtual Data generated on-the-fly! Automated surveys ! • Huge amounts of tabular data! • Services for Knowledge Discovery in Databases!
- 32. A Cloud of Services! Cube sizes! ! ASKAP Cubes! Prof. Kevin Vinsen !
- 33. A Cloud of Services! The overall picture! ! Distributed, scalable and flexible infrastructure! • Grid + Cloud may solve storage and processing! • Bandwidth is the issue! Big Data Science performance is highly dependent upon I/O data rates (local and transfer)! ! The data is the infrastructure! • Interconnected and interoperable archives! • Distributed, multi-wavelength and multi-facilities! ! Archives speaking Web Services! ALMA, LSST, ASKAP, MeerKAT, LOFAR, Apertif,...!
- 34. A Cloud of Services! The overall picture! ! We are moving into a world where ! • computing and storage are cheap ! • data movement is death! ! Archives should evolve from data providers into virtual data and services providers, where web services may help to solve bandwidth issues.! ! Web Services! • Smaller virtual data subproducts! • Distributed, multi-archive, multi-wavelength astronomy! • Workflows as a disruptive working methodology! !
- 35. A Cloud of Services! 3D Data Services! • Cutout! • Resample! • Spectrum extraction! • 2D slice extraction! • Dimensional reduction! • Filtering/Flagging! • 2D Moments! • Complex transformations! !
- 36. Scientific Use Cases! Exploration services! KDD - Knowledge Discovery in Databases! Understand what information is contained within the data in order to know how we can efficiently extract it ! • Anomaly detection! • Cross-matching data! • Dimensionality reduction! ! Extraction of scientifically ! relevant information from a! multidimensional parameter space.! visIt software !
- 37. Scientific Use Cases! Data Mining! Some key astronomy problems that can be addressed with data mining techniques:! ! • Cross-Match objects from different catalogues! • The distance problem (e.g., Photometric Redshift estimators)! • Star-Galaxy Separation! • Cosmic-Ray Detection in images! • Supernova Detection and Classification! • Morphological Classification (galaxies, AGN, gravitat. lenses, ...)! • Class and Subclass Discovery (brown dwarfs stars, ...)! • Dimension Reduction = Correlation Discovery! • Learning Rules for improved classifiers ! • Classification of massive data streams! • Real-time Classification of Astronomical Events ! • Clustering of massive data collections! • Novelty, Anomaly, Outlier Detection in massive databases! !
- 40. Scientific Use Cases! Multidimensional Clustering! !
- 41. Scientific Use Cases! Clustering! ! Cepheid Variables! Cosmic yardsticks! ! -- One Correlation! -- Two Classes!!
- 42. Scientific Use Cases! Outlier detection! !
- 43. Scientific Use Cases! Self Organizing Map! Organizing information in complex data collections! Find hidden relationships and patterns! Based on links among keywords and metadata ! ! ! !
- 44. Scientific Use Cases! The time domain! • VO Sky Event reporting metadata! • What, Where, Who, How ?! • Stars flares ,GRBs, solar, atmospheric particle bursts,..! ! The Helio-VO Project! ! ! ! ! ! ! ! !
- 45. Scientific Use Cases! The VO-Experiment! • Data Mining Oriented! • VO Services ! • Discovery ! • Access! • Waiting for analysis services! • Local software (also some Web portals)! • Crossmatching! • Inspection! • Visualization! • Web services associated to archives of big facilities! • Hinders cross-boundary science! ! ! !
- 46. Scientific Use Cases! XMM Observations of the AMIGA Sample! ! ! TopCat Hands-On ! Let’s do some science !! ! !
- 47. Scientific Use Cases! XMM Observations of the AMIGA Sample! ! ! Slightly brighter!
- 48. Scientific Use Cases! XMM Observations of the AMIGA Sample! ! ! Slightly brighter! Closer!
- 49. Scientific Use Cases! XMM Observations of the AMIGA Sample! ! ! Slightly brighter! Closer! Brighter in FIR!
- 50. Scientific Use Cases! XMM Observations of the AMIGA Sample! ! ! Slightly brighter! Closer! Brighter in FIR! Excess in longer ! !
- 51. Wf4Ever! Why Workflows ?! Web-services-based vs. Pipelines! ! • Expose the scientific methodology! • Keep the provenance ! • Pack the experiment ! • Enable ! • repeatable results ! • reproducibility! • reuse, repurpose! • cross-boundary science! • preservation!
- 52. Wf4Ever! Workflows Preservation! ! All components related to the! research lifecycle should be available. ! ! Preserved and easily retrievables ! ! • Proposals! • Data! • Processes! • Workflows! • Publications! ! !
- 53. IVOA Wf! Open questions for Web Services! In the Virtual Observatory! ! • Curation and preservation (identifiers)! • Discovery (semantics) of web services! • Characterization: input, outputs, functionality, etc.! • Copies (authenticity) or similar used as alternates ! • Permissions (authentication), licenses, platform, costs,..! • Metrics for quality: popularity, use stats, logs uptime, etc.! • Versioning and authoring (referenced and acknowledged)! ! In a cloud of services and data, Web Services should benefit of the same privileges acquired by Data.!
- 54. IVOA Wf! IVOA Note on Workflows! ! !
- 55. MyExperiment! Astronomy! • No VO services-based Wfs! • Helio Project Wfs! • VOTables parsing! • Internal services! Amiga! • Querying Catalogue!
- 56. Taverna! Working with the v2.3!
- 57. Taverna! Simple AMIGA ConeSearch! • Xpath plugin not a useful for extracting info from VOTable! • Helio-VO beanshell used instead (Thanks !)! • Visualization of results.. (VOTables) !
- 58. Taverna! XMM Multi-ConeSearch! • Lot of previous VOTable parsing ..! • The response is 1051 VOTables !! • VOTable merging tool needed!
- 59. Taverna! AMIGA Multi-ConeSearch! • Lot of beanshells for VOTabl and CSV parsing ..! • Beanshells development needed for splitting lists into values! • STILTS Library needed for VOTable crossmatching!
- 60. Taverna! The VO-experiment! • Discover Services! • Multi-query! • Crossmatching! • Inspection! • Visualization and Comparison! ! Proposed shortcuts for Taverna! • VORegistry Access Perspective! • STILTS VOTable Library ! • SAMP (Connectivity with VO Software)! • Python based beanshells! • Simple standard astronomy functions! !
- 61. Thanks !! ! Wf4Ever @ Manchester! • Carole Goble! • Sean Bechhofer! • Jiten Baghat! • Stian Soiland-Reyes! • Kalid Belhajjame! ! Helio-VO! • John Brooke! • Donal Felows! • Anja Leblanc! ! !
- 62. Thanks !!
- 63. Thanks !!
- 64. Thanks !!
- 65. Thanks !!