Archiver pilot phase kick off Award Ceremony

PILOT PHASE KICK-OFF EVENT AND
AWARD CEREMONY
29 November 2021
Contact: info@archiver-project.eu
Project website: www.archiver-project.eu

Welcome to the Pilot phase!
Started in January 2019, ARCHIVER is a unique initiative currently
running in the EOSC framework that is competitively procuring
R&D services for archiving and digital preservation for data
generated in the petabyte range.
Between December 2020 and August 2021 three consortia worked
on innovative, prototype solutions for Long-term data
preservation, in close collaboration with CERN, EMBL-EBI, DESY
and PIC.
2

The selection process for
proceeding to the next phase
is now over and the consortia
selected to continue with the
pilot phase will be officially
announced on the public
ceremony today!
3

14:30 - 14:40: Welcome from Sergey Yakubov (DESY)
14:40 - 15:00: Project overview / update - João Fernandes (CERN)
15:00 - 15:30: Expected outcomes of the Pilot Phase - Buyers Group representatives (CERN,
DESY, EMBL-EBI, PIC)
15:30 - 15:40: Interactive brainstorming
15:40 - 15:50: Break
Award ceremony
15:50 - 16:20: Presentation from consortium 1
16:50 - 17:00: Closing remarks - Sergey Yakubov (DESY)
Event Outline
4

HOUSE KEEPING
This event is being recorded in its entirety. A link to the full recordings
will be shared with participants afterwards
Two Q&A sessions are foreseen before the break and towards the end:
keep your questions and use the chat!
Please don’t activate your microphone and videos unless the host gives
you permission
5

Welcome!
Sergey Yakubov – DESY
Deutsches Elektronen-Synchrotron - DESY
Ein Forschungszentrum der Helmholtz-Gemeinschaft

Project overview / update
Pilot Phase Kick-Off
João Fernandes (CERN)
Contact: joao.fernandes@cern.ch

Focus: Archiving and Data Preservation Services using cloud services available via the European Open Science Cloud (EOSC)
Procurement R&D budget: 3.4M euro; Total Budget: 4.8M
Starting Date: 1st
of January 2019
Duration: 42 Months
Coordinator: CERN (Lead Procurer)
ARCHIVER Project
Buyers Group (BG) - Public organisations committing funds to
contribute to a joint-R&D-procurement, research data use cases and
R&D testing effort
Experts - Partner organisations bringing expertise in requirement
assessment and promotion activities
Buyers
Experts
Consortium
ARCHIVER has received funding from the European Union’s H2020 Research & Innovation programme under Grant Agreement No 824516.
14

Progress Beyond the State of the Art
15

Data integrity/security; cloud/hybrid deployment
Data volume in the PB range; high, sustained ingest data
rates in Gb/s. ISO certification: 27000, 27040, 19086 and
related standards. Archives connected to the GEANT
network
OAIS conformant services: data readability formats,
normalization, obsolesce monitoring, files fixity,
authenticity checks, etc. ISO 14721/16363, 26324 and
related standards
User services: search, discover, share, indexing, data
removal, etc. Access under Federated IAM
Layer 1
Storage/Basic Archiving/Secure
backup
Layer 2
Preservation
Layer 3
Baseline user services
Layer 4
Advanced
services
High level services: visual representation of data (domain
specific), reproducibility of scientific analyses, etc.
EMBL
1
–
FIRE
PIC
2
–
Mix
File
Storage
DESY
1
–
Individual
Scientist
CERN
1
–
CERN
Open
Data
CERN
2
–
CERN
Digital
Memory
PIC
3
–
Data
Distribution
EMBL
2
–
Cloud
Caching
PIC
1
–
Large
File
Storage
Scientific use cases deployments: https://www.archiver-project.eu/deployment-scenarios
DESY
2
–
Petra
III
Experiment
DESY
3
–
EUXFEL
Experiment
R&D Scope
ARCHIVER “current state of the art” report in the context of the EOSC: https://doi.org/10.5281/zenodo.3618215
16

Prototype Phase Highlights
• Functional objectives of the Prototype phase were successfully met.
• R&D challenge and scientific use cases requirements were globally understood.
• Systematic and structured feedback across the Buyers and Contractors to allow full understanding of the
R&D challenge by the Contractors and its service capabilities by the Buyer organizations.
• Prototype Phase organized into 13 bi-weekly sprints in an Agile-like process with a regular rhythm of
feedback and adjustment between Buyers and Contractors.
• CERN, EMBL-EBI, DESY & PIC allocated significant effort assessing and testing the prototypes, ingesting
several hundreds of TBs of data.
18

From Prototypes to Pilot
• R&D produced by the Contractors
• More than 200 tests executed during the Prototype phase across three platforms
• Technical assessment in two steps:
• R&D produced during the Prototype completed & of sufficient quality
• Mini-competition (Call-off) for Pilot phase services R&D
• Main Criteria considered in the R&D review process:
• Performance to scale correctly in the PB data region
• R&D validation progressing from functional to “Go-To-Market” ready
• Expertise in supporting Data Stewards achieving certification of scientific repositories
• Clear commercialisation plans for the resulting services after the project end
19

Pilot Phase Challenges
• Deployment model: on-premise vs cloud, portability, migration between different public
cloud infrastructure, strategies to avoid vendor lock-ins, disaster recovery
• Advanced AAI & Access management; Network performance: 100TB/day
• Scalability and Elasticity: High Volumes for data ingestion and data recall
• Security: simulated cyberattacks
• Layer-4 reproducibility of scientific analyses independent of infrastructure
• FAIRness measurements: F-UJI tool: https://www.f-uji.net/
• Total Cost of Ownership (TCO): SaaS on cloud vs SaaS on-premise
20

Early Adopters
• Participants:
• Demand side public sector organisations
• Key advantages
• Assess if resulting services address their archiving and preservation needs
• Contribute to shaping the R&D carried out in the project, identify additional use cases
• Have the option to purchase pilot-scale services by the end of the project
https://archiver-project.eu/early-adopters-programme
21

A Minimum Viable EOSC
• EOSC Association established to govern the European
Open Science Cloud
• Ambition to develop “Web of FAIR Data and Services”
for science in Europe for multiple domains
• Provide access to services for storage, computation,
analysis, preservation, etc. based on standards,
combining data and services
“FAIR Forever Report: A study to remember” with an
explicit mention to the ARCHIVER activities :
→https://www.dpconline.org/blog/fair-forever-a-fair-st
udy-to-remember
EOSC Long-term Data Preservation Task Force charter
→https://www.eosc.eu/sites/default/files/tfcharters/e
osca_tflongtermdatapreservation_draftcharter_202106
14.pdf
22

ARCHIVER & EOSC
● Broad pan-European requirement analysis of the research sector
○ Analysis considered in the competitive R&D tender
○ Technical and organisational measures aligned with European legislation in the services being
developed (by default & by design)
● Early Adopters Programme established
○ Additional use cases expanding further the set of supported scientific domains
○ Publicly funded research actors external to the ARCHIVER consortium
● Model for facilitate procurement of sustainable pilot services
○ Consortium members and Early Adopter organisations
○ Beyond the lifetime of the project
ARCHIVER is the only EOSC related H2020 project focusing on Archiving & LTDP services for PB scale datasets
across multiple research domains and countries.
23

Summary
● The R&D challenge of digital archiving goes beyond data storage: keep
intellectual control of data and associated products for decades, make
research outputs reusable
● Extending FAIR to research associated products: software, workflows, services
and even infrastructures
● Acting as a template to commoditise archiving and preservation at scale in
research domains with expert European SMEs
● ARCHIVER is promoting a sustainable model with services that will exist
beyond the project lifetime in the context of the EOSC
● ARCHIVER pilot phase: validating services with end-users and early adopters
organisations to determine suitability to their needs.
24

14:30 - 14:40: Welcome from Sergey Yakubov (DESY)
14:40 - 15:00: Project overview / update - João Fernandes (CERN)
15:00 - 15:30: Expected outcomes of the Pilot Phase - Buyers Group representatives (CERN,
DESY, EMBL-EBI, PIC)
15:30 - 15:40: Interactive brainstorming
15:40 - 15:50: Break
Award ceremony
16:50 - 17:00: Closing remarks - João Fernandes (CERN)
Event Outline
25

Thank you!
info@archiver-project.eu
https://www.archiver-project.eu/
https://twitter.com/ArchiverProject
https://www.linkedin.com/company/archiver-project/
https://www.youtube.com/channel/UCCBIyLpUt-hWmQatqdIhIzw

Expected outcomes of the Pilot
Phase
Buyers Group representatives
CERN, DESY, EMBL-EBI, PIC

CERN Requirements and Expectations
Tibor Simko, Jean-Yves Le Meur, Antonio Vivace, Ignacio Peluaga
28

CERN Open Data: From data preservation to data reuse
29
https://opendata.cern.ch https://www.reana.io
CERN Open Data portal
● more than 2.5 petabytes of particle
physics data
● education use cases
● independent research use cases
REANA reproducible analysis platform
● run containerised computational
workflows on remote clouds
● CWL, Snakemake, Yadage
● HTCondor, Kubernetes, Slurm
Nature 533 (2016), 452–454
https://doi.org/10.1038/533452a

CERN Open Data: From data preservation to data reuse
CMS Higgs-to-four-lepton example analysis CMS Higgs-to-tautau example analysis
● looking at “Compute” possibilities for content preserved in “Storage”
● reproducing several open data analysis examples
● reprocessing published simulated datasets
● challenges:
○ use containerised workflows (Docker, Kubernetes)
○ serve database content (CVMFS)
rule analyze_data:
input:
config["data"],
config["code"],
"results/scramdone.txt"
output:
"results/DoubleMuParked2012C_10000_Higgs.root"
container:
"docker://cmsopendata/cmssw_5_3_32"
shell:
"source /opt/cms/cmsset_default.sh "
"&& cd CMSSW_5_3_32/src "
"&& eval `scramv1 runtime -sh` "
"&& cd HiggsExample20112012/HiggsDemoAnalyzer "
"&& cd ../Level3 "
"&& cmsRun demoanalyzer_cfg_level3data.py"
Running containerised workflows
30

CERN Digital Memory: the OAIS Archive
31
- Status:
- Tool to harvest main CERN Information Systems and
create SIP (following BagIt specs).
- Includes publications, preprints, presentations,
photos, videos, and more
- Control Interface to run ‘’on-demand’’ archiving for
people leaving CERN
- Challenges:
- Duplication
- Authorships
- Integrity
- Versioning
- Running preservation services/ file conversions to
create AIPs

CERN Digital Memory: feeding Archiver.eu
32
Archiver.eu solutions to
ingest CERN SIPs and store
artifacts (AIPs, DIPs...)

33
DESY Requirements and Expectations
Sergey Yakubov, Martin Gasthuber

Main sources of data to be archived and preserved - 2021
>30 PB annual
3-6 PB annual
● two sites
○ Hamburg
○ Zeuthen (near Berlin)
● science fields
○ particle physics (LHC, Belle 2, …)
○ photon science (EuXFEL, Petra III, FLASH)
○ accelerator research (wakefield, Petra IV, …)
○ astrophysics
● all areas “data intensive science”
34

use cases - selection and relations
● photon science focussed - pushed by 3 local independently running
accelerator facilities
○ ground zero of the data deluge
● three, partly nested, scaled use cases
○ (1) individual scientist/small working group
○ (2) large working group/experiment
○ (3) facility level (with data policy)
● existing entities/actors with data ownership
and stewardship responsibilities
35
1
2
3
referencing

More tangible / general expectations
● core functionalities completed in last phase
● focus on scaling and stability validations
● Tape integration - scaling requires this ‘low cost’ storage technology
● hybrid deployments - i.e. MD and initial data copy on-site, secondary data and MD
backup (cold & encrypted) off-site - targeting at:
○ off-site data co-located with public cloud computing for ‘open data’ and related analysis tasks
○ primary off-site data attractive to communities/groups not having an ‘IT-Home’
36

● decent volume & bandwidth requirements
○ few 10 TB, 100MB/s, 10K objects - per archive
○ ~0.2-0.5PB annual
● access mainly via a web browser (GUI)
○ pre-defined MD schema & policy and templates by admins
● scientist (in person) - owner and archivist
● challenges
○ authentication - federated AAI
○ metadata scheme - hierarchy of schemes, mandatory fields, controlled extension
○ DOI / publication
○ local & hybrid deployments
○ open-data management & access
Case 1 - individual scientist / small working group
37

Case 2 - mid size - Petra III experiment
● Case 1 plus…
● increased volume and bandwidth requirements
○ ~100 TB, 1-2GB/s, >150K objects per archive
○ ~3-6 PB annual
● mainly API access (non interactive)
○ agent - automated life cycle process integration
● challenges
○ templating - i.e. MD, lifetimes, access rights, etc.
■ compliant to data policy
○ data storage costs starts to be important
■ volume and bandwidth amplification inside your solution - rising importance
38

Case 3 - facility level - EuXFEL
● Case 2 plus...
● largest scale
○ few 100 TB up to 1-2 PB, 2-10GB/s, >30K objects - per experiment
■ 1.3PB per day raw data production seen recently - and growing
○ >30 PB annual
● non interactive - full API access
○ full templating for creation and filling process of the archive
● challenges
○ scale
○ data storage costs becomes the most important criteria
■ volume and bandwidth amplification inside your solution becomes dominating criteria
39

EMBL-EBI Requirements and Expectations
Justin Clark-Casey
40

About EMBL-EBI
● “The home for big data in biology”
● Datasets, datasets, datasets: genomics, proteomics, metagenomics,
transcriptomics, imageomics …
○ You (researchers) submit them
○ We curate them
○ We provide them
○ You download them/(cloud)-access them/analyze them
○ We archive them
● Part of the European Molecular Biology Laboratory (27 member states,
1800 ppl, 80 independent research groups)
○ An intergovernmental organization like CERN.
41

EMBL-EBI Archiver Pilot Phase Plan
● Pilot preservation of selected large datasets from across EMBL-EBI as part of an integrated
data management system
43

EMBL-EBI Archiver Pilot Phase Plan
● Deploy a pilot in a real storage environment
● Taking forward the EMBL on FIRE use case
● No new feature requests
○ But may use features developed for other buyers (e.g. flexible metadata schemas)
● Assess in pilot conditions
○ Ease of integration
○ Scalability
○ Security
○ Robustness
○ Cost
44

45
PIC Requirements and Expectations
Jordi Casals, Manuel Delfino, Jordi Delgado

Port d’Informació Científica
Use cases will be based on MAGIC Telescope data
Observatorio del Roque de los Muchachos
(La Palma, Canary Islands)
• Collecting data 365 days a year
• Except when a volcano starts activity
Hint: NOW
• 300TB per year for ranges of 5-6 years
• Random recalls during the period
ALBA Synchrotron
• More than 10 Beamlines (and growing for next years)
• Datasets ranging from 200TB up to 4PB
• Internal and external scientific users
In collaboration with
46

Pilot Requirements
● Petabyte level Storage → functional, reliable, good performance, reasonable cost
○ From 1PB in 2021 to 15PB in 2025
○ Scalability testing → upload and downloads with production environments, sizes and bandwidth
● Integrate and automate processes using services CLI or API based scripts with production systems
● Create custom metadata schemas automatically on upload from data sources
○ Run production processes based on metadata searches
● Give access to users inside the collaboration with different permissions based on roles in experiment
● Test in-archive data processing using co-located Cloud
○ Enables automatic processing of uploaded data and prevents moving data around two times
● Test future reprocessing for reusability, based on custom containers, python notebooks or any
possible system that may help the data to not expire
● Analyze production costs to evaluate if it would be a usable option
Everything will be tested by both PIC and ALBA and put in common to
better evaluate the ﬁnal results and possibilities of the resulting services
47

Go to www.menti.com and use the code
3228 5110
50

PILOT PHASE
AWARD
CEREMONY
And the winners
are….

Archiver pilot phase kick off Award Ceremony

Bringing archived data to life
SCALABLE AND SUSTAINABLE LONG TERM
DIGITAL PRESERVATION OF SCIENTIFIC DATASETS
Matthew Addis
Arkivum
ARCHIVER Pilot Kick-Oﬀ Event 29 Nov 2021

Contents
•Sustainability
• Scalability, Performance and Cost Eﬀectiveness
• Portability, Open Standards, Data Sovereignty
• FAIR Forever, Digital Preservation, Long-term Access and Reuse
• Environmental Sustainability
•Summary

Scalability, Performance and Cost
Effectiveness

ARCHIVER: requirements
https://zenodo.org/record/4574234#.YW2NyqDTWJQ

Demand side requirements

Long-term digital preservation (LTDP) factories to support sustainable FAIR data

LTDP factories that scale
• Automation
• Minimal Effort Ingest / Minimal Viable
Preservation
• Scalable and efficient use of IT resources
• Serverless computing
• Microservices
• Scale-out parallel processing
• Robust workflows
• Failures will happen
• Record and audit everything
Eugen Stoll, CC BY-NC 2.0, https://flic.kr/p/5c8cz

Microservices, serverless computing, cloud infrastructure

Ingest and archiving of a 50TB dataset
50,000 HDF5 ﬁles (50TB) ingested and stored at a rate of 150TB/day
CPU
cores
Ingest
Starts
Time
All done

Costs and Benchmarking
• Execute real world scenarios on GCP
• Record parameters
• execution time
• data volumes, number of ﬁles
• type of activity (ingest, export, preservation)
• Extract baseline costs from GCP
• Extract overhead of running scenario
• Normalise to create a ‘cost per TB’ metric
• Calculate short-term and long-term storage
• Upload/export buckets, caching, archive
buckets

Costs Projections

Portability, Open Standards, Data Sovereignty

Open Standards, Open Specifications, Open Source: Deployment

Open Standards, Open Specifications, Open Source: Interaction
Bagit & BDbags

Open Standards, Open Specifications, Open Source: Content

FAIR Forever, Digital
Preservation, Long-term Access
and Reuse

Standards, External Assessment and Certification
Self Assessment
Peer Review
Formal Certiﬁcation

Environmental Sustainability

Making LTDP more environmentally sustainable
• Keep less
• e.g. appraisal, don’t digitize everything
• Do less
• e.g. minimum eﬀort ingest, don’t normalise
• Make smarter use of storage
• e.g. deep archive, small footprint access copies
• Make more eﬃcient use of IT resources
• e.g. don’t leave idle servers running
• Use environmentally friendly infrastructures
• e.g. cloud with renewable energy
https://www.dpconline.org/blog/is-digital-preservation-bad-for-the-environment
https://dash.harvard.edu/handle/1/40741399

Cloud is greener than you might think
https://www.google.com/about/datacenters/gallery/#hamina-exterior-landscape
https://cloud.withgoogle.com/region-picker/ https://www.google.com/about/datacenters/gallery/#st-ghislain-solar-panels

Cloud providers can achieve very high energy efficiency
https://datacenters.lbl.gov/sites/default/files/Masanet_et_al_Science_2020.full_.pdf
https://www.stickermule.com/marketplace/3442-there-is-no-cloud

Strategy for energy efficiency and
low carbon footprint
• Users can choose what processing to do on their
content
• Deep archive storage for infrequently accessed
data
• Serverless computing: only consume resources
when needed
• Deployment into energy eﬃcient data centres
• Allow applications to run near to the data

Summary

Summary
• The scale of scientific datasets mean that new approaches to LTDP are necessary
• Automation, microservices, serverless computing and cloud IaaS are powerful combination
• Efficiency and costs can be measured, managed and optimized (€ and carbon)
• Reduced consumption and choice over cloud location/provider helps environmental sustainability
• Open standards, open specifications and open source are key to portability and interoperability
https://www.archiver-project.eu/

QUESTIONS?
hello@arkivum.com
www.arkivum.com
matthew.addis@arkivum.com orcid.org/0000-0002-3837-2526 www.arkivum.com

Libnova - CSIC - University of Barcelona -
Giaretta Associates - AWS - Voxility - Bidaidea

LIBNOVA USA 14 NE First Ave (2nd
floor) -Miami, Florida 33132, USA -Tel: +1 844-894-6532
LIBNOVA EU Paseo de la Castellana 153 -28046 Madrid, Spain -Tel: +34 91 449 08 94
contact@libnova.com
PILOT PHASE KICK-OFF EVENT
2021-11-29

Big thanks to the Archiver team EU Representatives and the LIBNOVA consortium team.

Market
Time to market:
Shorter development
lifecycle leading to a faster
time to market (5 to 2
years)
Market visibility:
Several EU/USA
Universities and a large
European pharmaceutical
signing contracts.
Product
Accelerated innovation:
Being able to work with
the 4 buyers maximizes
success chances in first
iteration.
Standards:
We are bringing several –
previously unknown-
standards onboard.
Team
EUnthusiasm:
LIBNOVA consortium’s
team feels enthusiastic
about being part of
something larger, relevant
to the EU. The EU is leading
the R&D in this field.
Happiness:
Everyone have been
working really hard, but
really happy about it!
How are you and the ARCHIVER project benefiting LIBNOVA?

Making the practice more efficient
- Direct: Optimized storage costs, low operational costs.
- Indirect: Less time/resources for producers/researchers to use it.
Making it available to a broader audience
- Demystifies preservation. Easy to understand and to use.
- Opens the practice to large volume datasets and to more advanced organizations.
Making it easier to apply best
practices
- Fully conformant to ISO 14721 and ISO16363 (and several other standards)
- Full support for FAIR/TRUST data models, workflows, etc.
How are we benefiting the community?

How are we benefiting the community?
Reducing the environmental impact of the community preservation activities
- Reuse what you have: on-premise deployments using existing infrastructure are possible.
- Consume only when needed: Resources are consumed when there is something to do. Scaling from
36 Kubernetes pods to ~5000 in 32 minutes. Process the workload and then back to 36 pods.
- “Environment impact” topic is included in every architecture specification/analysis. The team is
proud of many small and smart optimizations. For example, the hashing algorithm.
- Using carbon-neutral providers and data centres.
Contributing to the European digital sovereignty and digital development and
leadership
- Cloud provider independence
- Decouples content from providers
- Open to the emergence of EU-based cloud providers.

LIBNOVA USA 14 NE First Ave (2nd
floor) -Miami, Florida 33132, USA -Tel: +1 844-894-6532
LIBNOVA EU Paseo de la Castellana 153 -28046 Madrid, Spain -Tel: +34 91 449 08 94
contact@libnova.com
Thanks again!

Closing remarks
João Fernandes (CERN)

Closing Remarks
Upcoming Milestones for Buyers / Early Adopter organisations
● Today: Pilot kick-off meeting
● January 2022: Full capacity pilot platforms available
● April - June 2022: Project Review meetings with demos of the resulting services
○ Webinars for training administrators and end users in the Pilot platforms
Upcoming Public Events (February - June 2022)
● Industry related webinars about the advantages of PCP projects - February 2022
○ Procurement of research and development of new innovative solutions before they are commercially available
● Thematic webinars for potential Early Adopters of ARCHIVER - March-May 2022
Dates and Registration to be announced at https://archiver-project.eu/
92

www.archiver-project.eu
@ArchiverProject
company/archiver-project
Join our community!
Thank you!

Archiver pilot phase kick off Award Ceremony

More Related Content

What's hot (20)

Similar to Archiver pilot phase kick off Award Ceremony (20)

More from Archiver (20)

Recently uploaded (20)

Archiver pilot phase kick off Award Ceremony