Downloaded 23 times
![Write Once, Read Many
Inside Rosetta, format
identification is a ‘WORM’ process.
As a part of the ingest
routine, format identification is
automatically undertaken, written
to the file records, and the system
database, and used thereafter as
a consistent ‘label’.
We rely on the persistence of the
label to accurately plan activities and
E-272-f-001
‘measure’ the content or shape of the Abbot, John 1751-1840 :
Original drawings of insects by J Abott. [1816?]
repository. Alexander Turnbull Library
.](https://image.slidesharecdn.com/10002jgattuso-120402175609-phpapp01/85/Jay-Gattuso-Persistently-Identifying-Formats-5-320.jpg)
Uploaded byFuture Perfect 2012
Jay Gattuso Persistently Identifying Formats
DROID is a tool used by the National Digital Heritage Archive (NDHA) to automatically identify file formats during ingest into the Rosetta digital preservation system. The NDHA conducted research comparing format identification results across different versions of DROID and found that 75% of file types were identified consistently, while 26% showed multiple possible identifications. The NDHA recommends creating a test dataset spanning all file formats, more research on format identification persistence, and robust testing of new DROID signatures to limit impacts on users.
More Related Content
Similar to Jay Gattuso Persistently Identifying Formats
Jay Gattuso Persistently Identifying Formats
- 1. ‘Persistently’ Identifying Formats PRONOM, DROID and the NDHA Jay Gattuso Digital Preservation Analyst National Digital Heritage Archive National Library of New Zealand
- 2. Summary How Rosettauses DROID How DROID has changed Research NDHA completed Results Recommendations
- 3. DROID & PRONOM •PRONOM is the most widely used file format registry in the sector • DROID is a tool that ‘identifies’ file types (based on PRONOM records) • Both are from TNA (UK) • DROID Signature v59 EP/1958/2520-F – 551 signature sets Registry, Hunter Building, Victoria University of Wellington Photograph taken for the Evening Post newspaper, 31 Jul 1958 – 864 file type records Alexander Turnbull Library www.nationalarchives.gov.uk/PRONOM/Default.aspx
- 4. Rosetta – ABrief History • NLNZ Digital Preservation Repository • 4 years since inception • 18 months out of project • 8 significant upgrades/software revisions • ~6 Million digital objects to 1/1-000008-G Smiley's stables and horse repository, Whanganui date Harding, William James, 1826-1899 :Negatives of Wanganui district . Alexander Turnbull Library • Backbone of the ANZ GDAP
- 5. Write Once, ReadMany Inside Rosetta, format identification is a ‘WORM’ process. As a part of the ingest routine, format identification is automatically undertaken, written to the file records, and the system database, and used thereafter as a consistent ‘label’. We rely on the persistence of the label to accurately plan activities and E-272-f-001 ‘measure’ the content or shape of the Abbot, John 1751-1840 : Original drawings of insects by J Abott. [1816?] repository. Alexander Turnbull Library .
- 6. Behaviours and functionsbased on DROID format assertions Rosetta uses DROID to automatically establish format type.
- 7. Rosetta Overview ValidationStack Automated Format Identification via DROID
- 8. Shape Sorting... Where: • Thearea inside the box is Rosetta • Each block is a DO • Each shape is a format • The ‘Sorter’ is DROID
- 9. Shape Sorting... Process: • Arecord is kept of the ‘shape’ the DO entered the box via • The record is used by the system to trigger activities • The DO can be removed from the box using the same shaped hole it used on entry
- 10. Shape Sorting... Expectations: • The‘Sorter’ never changes • The blocks never change • A DO placed in the box yesterday will be the same shape tomorrow • A DO placed in the box yesterday will be extractable via the shape tomorrow
- 11. Shape Sorting... The realityfor NDHA: • DROID has undergone 2 major revisions • Container signatures have been included • Since Rosetta v1 release: – 406 new formats, – 600 changes to signatures – (This is generally a good thing!)
- 12. Identifying and QuantifyingChange • Rosetta has used DROID versions 3 and 5, currently testing with 6 • Rosetta has used DROID signature versions v13, v37, v45 and v49, testing with v52 • Proposal to use a new DROID method in Rosetta • How has/will this affect the way we characterise Digital Objects at EP/1958/0585-F the NDHA? Signature of Queen Elizabeth II in a visitors book Negatives of the Evening Post newspaper. Feb 1958 Alexander Turnbull Library
- 13. Identifying and QuantifyingChange • Source set: – 26,000 digital objects, – ~600 Gb of content, – spanning 61 format types – all from the live system • DROID v3, DROID v5, DROID v6 and DROID v6 ‘FAST’ tested • Signatures v13, v37, v45, v49 and v50 tested EP/1990/0432/29-F New school patrol system being tested , Wellington • All files tested with and Photograph taken by John Nicholson ca 2 Feb 1990 without file extensions Alexander Turnbull Library
- 14. Identifying and QuantifyingChange • 1 million DROID ‘assertions’ captured • Python and MySQL used to sort, clean, filter, draw graphics and otherwise interpret results • Paper competed and will be available on the OPF website www.openplanetsfoundation.org DCDL-0004533 Eric Idle. 5 December, 2007. Webb, Murray, 1947- : Digital caricatures published from 29 July 2005 onwards Alexander Turnbull Library
- 15. Summary of Results Ofthe 61 tested file types : 75% performed identically for all tested versions of DROID and signature versions fmt/49 (RTF 1.4)
- 16. Summary of Results Ofthe 61 tested file types : 40% consistently offered a single PUID across the range of DROID tests By extension: gif, avi, png, jpg, html, xml, bmp, wp, and some subsets of doc, ppt and exe fmt/12 (PNG 1.1)
- 17. Summary of Results Ofthe 61 tested file types : In 26% of the file types multiple PUIDs are equally asserted by DROID at various times. By extension: docx,xlsx,pptx, some pdf, doc, xls, ppt, txt, log, aif f, and arc fmt/7 (TIF format)
- 18. Summary of Results Ofthe 61 tested file types : In 16% of the file types DROID version 6 in ‘FAST’ mode performs differently DROID version 6 in standard mode By extension: epubs, mp4, flac, wav, zip and some subsets of pdf, xls, tif fmt/6 and exe (Waveform Audio)
- 19. Recommendation 1 There isa clear need for a community owned dataset that spans the PRONOM catalogue to support testing (This should be community created) ExL-fmt/62 - fmt/189 (MS Open Office XML 2007)
- 20. Recommendation 2 It isstrongly recommended that more research is undertaken looking at the persistence of PUID’s to give a more complete history of file type assertions by PRONOM/DROID fmt/14 (PDF 1.0)
- 21. Recommendation 3 Given thevariances observed, especially with DROID v6 ‘FAST’ mode, it is recommended that all signatures are robustly tested prior to release, and efforts are made to maintain consistency with legacy signatures, and limit x-fmt/263 (ZIP format) impact on users
- 22. Recap How Rosettauses DROID How DROID has changed Research NDHA completed Results Recommendations
- 23. Thank you jay.gattuso@dia.govt.nz Rosetta demo – Wednesday 28th March 9am to 1pm @ NLNZ - 77 Thorndon Quay Paper available through the Open Planets Website www.openplanetsfoundation.org