OpenML Tutorial: Networked Science in Machine Learning
2 likes2,399 views
The document discusses the challenges and limitations of traditional scientific publishing, such as reproducibility issues, publication bias, and lack of collaboration. It highlights the benefits of open science frameworks like OpenML and collaborative platforms that enable data sharing, real-time discussions, and joint research efforts. By promoting large-scale collaboration and visibility of results, these platforms aim to enhance scientific discovery and improve the research process.
OpenML Tutorial: Networked Science in Machine Learning
- 1. N E T W O R K E D MAC H I N E L E A R N I N G J OAQ U I N VA N S C H O R E N ( T U / E ) , 2 0 1 4 #OpenML
- 3. 1 6 1 0 G A L I L E O G A L I L E I D I S C O V E R S S A T U R N ’ S R I N G S ‘ S M A I S M R M I L M E P O E TA L E U M I B U N E N U G T TA U I R A S ’
- 4. Research different. Royal society: Take nobody’s word for it Scientific Journal: Reputation-based culture
- 5. 3 0 0 Y E A R S L AT E R J O U R N A L S S H O W L I M I T S • Complex code not included • Large data sets not included • Experiment details scant • Results hard to reproduce • Papers not updatable • Slow, incomplete tracking of paper impact • Publication bias • No online public discussion • Open access?
- 6. J O U R N A L S : L O N G - T E R M M E M O RY I N T E R N E T: S H O R T- T E R M W O R K I N G M E M O RY N E T W O R K E D S C I E N C E O N L I N E D A TA B A S E S O P E N S O U R C E C O D E W E B S E R V I C E S , A P I S C O L L A B O R A T I V E T O O L S ! O P E N , S C A L A B L E C O L L A B O R A T I O N R E A L - T I M E D I S C U S S I O N C O M B I N E , R E U S E S C I E N T I F I C R E S U LT S C I T I Z E N S C I E N C E
- 7. Research different. Polymaths: Solve math problems through massive collaboration (not competition) Broadcast question, combine many minds to solve it Solved hard problems in weeks Many (joint) publications
- 8. Research different. SDSS: Robotic telescope, data publicly online (SkyServer) +1 million distinct users vs. 10.000 astronomers Broadcast data, allow many minds to ask the right questions Thousands of papers
- 9. Research different. Galaxy Zoo: citizen scientists classify a million galaxies Offer right tools so that anybody can be a scientist Many novel discoveries by scientists and citizens
- 10. Research different. Sharing data sparks discovery Designed serendipity: - What’s hard for one scientist is easy for another - Surprising ideas, observations can spark new discoveries Share, organise data for easy, large-scale collaboration Data exploding in all sciences: collaborative data analysis needed
- 11. Building reputation Authorship: easy to contribute + contributions stored, visible online Collaboration: build trust, work with new people Citation: more people see, build upon, and cite your work. Tell people how to cite data and code. Altmetrics: track reuse/interest online (ArXiv)
- 12. N E T W O R K E D MAC H I N E L E A R N I N G
- 13. Machine learning Complex code, large-scale data, experiments (impossible to print) Experiments not shared online: impossible to build on prior work: inhibits deeper analysis (e.g. meta-learning) Low reproducibility, generalisability (studies contradict) What if we could all connect with each other, and with other scientists, to explore and apply machine learning? Few collaborative tools to speed up research
- 14. OpenML Place to share data, code, experiments in full detail All results organised, linked together for further (meta)analysis, reuse, discussion, study, education Links to (open-source) code, open data anywhere online. Anyone can post data to analyse, anyone can share code and results (models, predictions, evaluations) Integrated in ML platforms (R,Weka, Rapidminer,…) to automatically load data, upload results Scientists can work in teams, but results only publicly visible if data, code shared
- 15. OpenML: benefits for scientists More time: automates routinizable work: - find data and/or code - setup and run large-scale experiments - results compared to state-of-the-art - log experiment details for future reference More control: - state how others should cite your work - track reuse - share results more easily More knowledge: - more time for actual research - build directly on prior work - easier, large-scale collaboration + interaction
- 55. Plugins:WEKA
- 56. Plugins: MOA
- 57. Plugins: RapidMiner 1 . O P E R AT O R T O D O W N L O A D TA S K ( TA S K T Y P E S P E C I F I C ) 2 . S U B W O R K F L O W T H AT S O LV E S T H E TA S K , G E N E R AT E S R E S U LT S 3 . O P E R AT O R F O R U P L O A D I N G R E S U LT S
- 58. OpenML: under development OpenML studies - collection of datasets, flows, runs, results in a study - online counterpart of paper (with url) - construct by simply tagging resources - easily include (build on) data of others Reputation building - Profile page: statistics of activity and impact on OpenML - Collaborative leaderboards: best contributors to solving a task Teams - Add scientists in teams (circles) - Share resources, results within team only - Make public at any time (e.g. after publication) Meta-learning support - Data/Flow qualities: easy adding, better overviews - Algorithm selection techniques running on website (vs humans?)
- 59. J O I N T H E C LU B