The State of HDF
Download as PPTX, PDF•0 likes•89 views
This document summarizes the current status and focus of the HDF Group. It discusses that the HDF Group is located in Champaign, IL and is a non-profit organization focused on developing and maintaining HDF software and data formats. It provides an overview of recent HDF5, HDF4 and HDFView releases and notes areas of focus for software quality improvements, increased transparency, strengthening the community, and modernizing HDF products. It invites support and participation in upcoming user group meetings.
The State of HDF
- 1. Conf-DDDD-IN The State of HDF Summer ESIP 2023 This work was supported by NASA/GSFC under Raytheon Technologies contract number 80GSFC21CA001. This document does not contain technology or Technical Data controlled under either the U.S. International Traffic in Arms Regulations or the U.S. Export Administration Regulations. Dana Robinson Director of Software Engineering NASA EED-3 / The HDF Group [email protected]
- 2. Conf-DDDD-IN 2 • About us • Software status • Current focus • How we can help each other Outline
- 4. Conf-DDDD-IN 4 • Located in Champaign, IL • Spun off from NCSA in 2006 • Non-profit 501(c)(3) • ~25 employees The HDF Group
- 5. Conf-DDDD-IN 5 • Mission-driven – Sustainable development of HDF technologies – Guarantee continual accessibility of HDF data • Services – Maintain and develop HDF products – Consulting and support contracts – Training The HDF Group
- 6. Conf-DDDD-IN 6 • HDF5 1.0.0 was released in 1998 • What will the next 25 years bring? https://forum.hdfgroup.org/t/what-do-you-want-to-see-in-hdf5-2-0/10003 25 years of HDF5! 🎂 🎉
- 8. Conf-DDDD-IN 8 • HDF5 • HDF4 • HDFView Release schedules
- 9. Conf-DDDD-IN 9 • Latest: 1.14.1(May 2023) • 1.8 branch retired early this year • 1.10 and 1.12 retiring this year • Have not scheduled 2024 yet HDF5 Schedule
- 10. Conf-DDDD-IN 10 • 1.14.0 – Multi-dataset I/O – Selection I/O – Subfiling – Onion VFD • 1.14.1 – Maintenance release – bugfixes, minor features HDF5 New Features https://www.hdfgroup.org/2023/05/release-of-hdf5-1-14-1-newsletter-194/
- 11. Conf-DDDD-IN 11 • 1.14.2 – Read-only S3 VFD improvements • Better logging • Temporary security credential support – CVE-free HDF5 New Features
- 12. Conf-DDDD-IN 12 • Latest: 4.2.16-2 (June 2023) – Patch release (fixes a Java issue w/ HDFView) – Bugfixes – Build system improvements HDF4 Schedule https://www.hdfgroup.org/2023/07/release-of-hdf-4-2-16-2-a-patch-release-newsletter-195/
- 14. Conf-DDDD-IN 14 As always, we: • Strive to maintain API compatibility – HDF5's compatibility macro scheme, etc. • Are committed to file format forward and backward compatibility – Should always be able to create files in earlier formats – Should always be able to read earlier file formats HDF Product Change Policy
- 15. Conf-DDDD-IN 15 Before implementing a breaking change, we will: • Create a discussion post on GitHub • Create a post in the HDF Forum • Make an announcement in the newsletter • Attempt to contact people who we know might be affected • Allow at least 30 days for comments HDF Product Change Policy https://forum.hdfgroup.org/t/hdf4-change-procedure/11240
- 16. Conf-DDDD-IN 16 • Latest: HDFView 3.3.0 (April 2023) • Based on: – HDF5 1.14.1 – HDF4 4.2.16 (4.2.16-2 coming soon) • Releases based on older versions of HDF5 (e.g., 1.10) have been retired • Looking to modernize or replace this! HDFView https://www.hdfgroup.org/2023/04/release-of-hdfview-3-3-0-newsletter-193/
- 18. Conf-DDDD-IN 18 • Improve software quality • Increase transparency • Strengthen our community • Modernize HDF Products What are we working on?
- 21. Conf-DDDD-IN 21 • All CVE issues fixed – https://github.com/HDFGroup/cve_hdf5 – 3 currently unfixed • Testing improvements – Integration testing w/ key products (h5py, etc.) – More transparency w/ CDash • Resolve unfixed GitHub issues • Code cleanup Software quality
- 22. Conf-DDDD-IN 22 HDF5, although open source, grew up in a walled garden and this mindset persists I'm working to fix this! The goal is to do all product-specific (e.g., HDF5) work with the community Transparency
- 23. Conf-DDDD-IN 23 • HDF5 Working Group meeting – Every Thursday at 10:05 central time – Email me ([email protected]) for an invite – Covers pull requests, issues, tech discussion • Most project planning moving to GitHub • More testing via GitHub actions – Will use public CDash for non-GitHub tests Transparency
- 24. Conf-DDDD-IN 24 • Aforementioned HDF5 WG meeting • Everything in the transparency slides, really • Adding external people as code owners • Spending more time connecting with our users, both new and old Community
- 25. Conf-DDDD-IN 25 • Some of our key software has been around for a long time! – HDF(4): 35 years (1988) – HDF5: 25 years (1998) – HDFView: 16 years (2007) • Although we strive to keep our software up to date, the code is still old and could use an overhaul Modernization
- 26. Conf-DDDD-IN 26 • Better support for new compilers – Especially Intel's oneAPI • Better support for AI/ML workflows • Better cloud integration • Better support for heterogeneous computing • Windows Unicode support Modernization
- 27. Conf-DDDD-IN 27 • Better variable-length support • Multithreading • Support for sparse data • Improve performance • Internal cleanup – Easier to modify & debug – Refactor code that made sense in 1998 but not so much now Modernization
- 28. Conf-DDDD-IN 28 How can we help each other?
- 29. Conf-DDDD-IN 29 Support our non-profit mission Contact: [email protected] https://www.hdfgroup.org/donate
- 30. Conf-DDDD-IN 30 • US HUG – August 16-18, 2023 – The Ohio State University - Columbus, OH – https://www.hdfgroup.org/hug/hug23/ • European HUG (focus on compression) – September 19-21, 2023 – DESY - Hamburg, Germany – https://indico.desy.de/event/39343/ HDF User Group Meetings
- 31. Conf-DDDD-IN 31 Thanks for your time!
- 32. Conf-DDDD-IN 32 This work was supported by NASA/GSFC under Raytheon Technologies contract number 80GSFC21CA001.