NOAA Big Data Project Handout
2 likes•770 views
NOAA collects over 20 terabytes of environmental data daily from various sources like satellites and weather stations, but only a small portion is accessible publicly. To address this, NOAA partnered with Amazon, Google, IBM, Microsoft and others through a Big Data Project to explore making more data available on their cloud platforms. This would allow private companies to develop new products and services using NOAA data and generate additional value. Initial datasets being released include radar and satellite imagery to demonstrate this proof of concept, with the goal of releasing more datasets over time like additional satellite data.
NOAA Big Data Project Handout
- 1. Background NOAA gathers over 20 terabytes of data every day – more than twice the data of the entire printed collection of the United States Library of Congress. This environmental intelligence comes from a wide variety of sources, including Doppler radar systems, weather satellites, buoy networks, water level stations, real-time weather stations, as well as ships and aircraft. However, right now only a small percentage of this valuable data is easily accessible to the public. The demand for this data has increased, and it is imperative to find ways to effectively and efficiently distribute this data to decision makers and industries. Accordingly, in 2014 NOAA issued a Request for Information to engage private industry to help make NOAA’s data available in a rapid, scalable manner to the public. Through this process, American companies were asked to suggest ways for NOAA to more effectively distribute its data, allowing industry to take advantage of the untapped value of NOAA’s public data resources by creating new and innovative products and services. On April 21, 2015, U.S. Secretary of Commerce Penny Pritzker announced a big data project with Amazon Web Services, Google Cloud Platform, IBM, Microsoft Corp., and the Open Cloud Consortium (OCC) to explore ways of bringing NOAA and the Department of Commerce closer to their goal of unleashing NOAA’s vast resources of environmental data and delivering on one of the Department’s key priorities – transforming Department data capabilities and supporting a data-driven economy.¹ Big Data Project NOAA’s Big Data Project provides an innovative approach to publishing NOAA’s ever growing data resources and positioning them near cost-efficient high performance computing, analytic, and storage services provided by the private sector. NOAA took this course because it believes that: ▲▲ There is additional value in NOAA data that has not yet been realized because of access and infrastructure difficulties; ▲▲ If NOAA data were accessible in the Cloud, alongside computing capability, private enterprise might generate new value-added products, services, and lines of business; and ▲▲ Private business might be willing to support the cost of transferring and storing large datasets because of these new lines of business.² U.S.D EPARTMENT OF COMM ERCE NATIONALOCEA NIC AND ATMOSPHERIC ADMINISTRATION Big Data Project Source: NOAA By 2020, NOAA will have 160 petabytes of archived data. Source: NOAA Source: NOAA
- 2. Cloud Collaborators NOAA’s cloud collaborators will provide the framework for a set of data alliances led by each of the anchor companies. Data alliances, which consist of participating organizations across the private and public sectors, will work to research and test solutions for bringing NOAA’s vast information to the cloud, where both the public and industry can easily and equally access, explore, and create new products from it, fostering new ideas and spurring economic growth. NOAA and its partners are starting small in order to minimize the collaborators’ investment costs and they will use initial datasets to establish baselines and demonstrate the proof of concept. This will be a market driven process, whereby the collaborates reach back into NOAA for the datasets being most requested by their users. Except for those restricted due to national security concerns, all NOAA datasets are open for discussion on availability. The collaborators will have equal access to NOAA’s historical archive of large datasets and each must provide users with equal access to NOAA data on equal terms. NOAA’s Role The role of NOAA in the Big Data Project is to: ▲▲ Ensure free and open access to all data, regardless of market interest; ▲▲ Provide authoratative data, metadata, information, forecasts, warnings, and analysis; ▲▲ Perform research to improve sensors, numerical models, and algorithms; ▲▲ Ensure long-term preservation of the data master copy; ▲▲ Perform scientific data stewardship as an unbiased, objective partner; and ▲▲ Provide expertise and skills to support proper use and application of data.² Current Status A real-time feed and full historical archive of original resolution (Level II) Next Generation Weather Radar (NEXRAD) data, from June 1991 to present has been released to Amazon, Microsoft, OCC, and Google. NEXRAD is a network of 160 high- resolution Doppler weather radars that detect atmospheric precipitation and winds, which allow scientists to track and anticipate weather events, such as rain, ice pellets, snow, hail, and tornadoes. NOAA is planning for the release of the next sets of data, which could include: ▲▲ Geostationary Operational Environmental Series satellite (GOES) – provides satellite information every 30 minutes using either the Visible, Infrared, Shortwave Infrared (4um), or Water Vapor images into one larger composite image using GOES East and West Imagers. ▲▲ Geostationary Operational Environmental Series satellite- R (GOES-R) – scheduled for launch in October 2016, the GOES-R will collect three times more data and provide four times better resolution and more than five times faster coverage than current satellites. This means the satellite will scan Earth’s Western Hemisphere every five minutes and as often as every 30 seconds in areas where severe weather forms. ▲▲ Multi-Radar/Multi-Sensor System (MRMS) – quickly and intelligently integrates data streams from multiple radars, surface and upper air observations, lightning detection systems, and satellite and forecast models. Numerous two-dimensional multiple-sensor products offer assistance for hail, wind, tornado, quantitative precipitation estimation forecasts, convection, icing, and turbulence diagnosis. REFERENCES: 1 commerce.gov/news/press-releases/2015/04/us-secretary- commerce-penny-pritzker-announces-new-collaboration-unleash 2 slideshare.net/AmazonWebServices/nnoaa-big-data-project- presentation-for-earth-observations-in-the-cloud-demo-day All photos and graphics courtesy of NOAA MSMR is a powerful research and development tool for the creation of new techniques to estimate and forecast precipitation amounts, locations and types. Source: NOAA. Source: NOAA NEXRAD image of Morehead City, NC. Source: NOAA Types of NOAA data. Source: NOAA This document is one in a series of fact sheets created to showcase NOAA’s products and services relative to severe convective storms, tropical cyclones, flooding, climate change and the big data project.