Csise15 codehunt
Download as PPTX, PDF•1 like•664 views
This document discusses how Code Hunt crowdsources code and processes through an educational gaming platform. It began as dynamic symbolic execution tool Pex that was adapted into a game called Coding Duel. This evolved into Code Hunt, an online platform where users write code in the browser to solve puzzles. Code Hunt provides test cases to give feedback until the user's code matches the secret implementation. It has been used by hundreds of thousands of students to learn programming in a game-like environment. The document argues that Code Hunt can scale crowdsourcing of code and processes while identifying top coders through competitive coding contests hosted on the platform. Code Hunt data from these contests has been publicly released to further research.
Csise15 codehunt
- 1. Crowdsourcing Code and Process via Code Hunt Tao Xie, Judith Bishop, Nigel Horspool, Nikolai Tillmann, Jonathan de Halleux University of Illinois at Urbana-Champaign, USA; Microsoft Research, USA; University of Victoria, Canada https://www.codehunt.com
- 2. Testing Tool Educational Gaming Dynamic Symbolic Execution (Pex) Pex for Fun/Code Hunt: Interactive Gaming for Teaching and Learning Support http://research.microsoft.com/pex/ http://pex4fun.com https://www.codehunt.com
- 4. Pex is Part of Visual Studio 2015! • As new feature of “Smart Unit Tests” http://www.visualstudio.com/en-us/news/vs2015-preview-vs#Testing Nikolai Tillmann, Jonathan de Halleux, and Tao Xie. Transferring an Automated Test Generation Tool to Practice: From Pex to Fakes and Code Digger. In ASE 2014, Experience Papers. http://web.engr.illinois.edu/~taoxie/publications/ase14-pexexperiences.pdf
- 5. Coding Duels http://pex4fun.com/ Nikolai Tillmann, Jonathan De Halleux, Tao Xie, Sumit Gulwani and Judith Bishop. Teaching and Learning Programming and Software Engineering via Interactive Gaming. In ICSE 2013, SEE. http://web.engr.illinois.edu/~taoxie/publications/icse13see-pex4fun.pdf
- 6. Coding Duel Competition @ICSE 2011 http://pexforfun.com/icse2011
- 7. Code Hunt: Resigned As Game https://www.codehunt.com/
- 8. Gameplay 1. User writes code in browser 2. Cloud analyzes code – test cases show differences As long as there are differences: User must adapt code, repeat When they are no more differences: User wins level! code test cases
- 9. Behind the Scene of Code Hunt Secret Implementation class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); } } Player Implementation class Player { public static int Puzzle(int x) { return x; } } class Test { public static void Driver(int x) { if (Secret.Puzzle(x) != Player.Puzzle(x)) throw new Exception(“Mismatch”); } } behavior Secret Impl == Player Impl 9
- 10. It’s a game! • iterative gameplay • adaptive • personalized • no cheating • clear winning criterion code test cases
- 11. Towards a Course Experience
- 12. Code Hunt as Crowdsourcing Platform Secret Implementation class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); } } Player Implementation class Player { public static int Puzzle(int x) { return x; } } class Test { public static void Driver(int x) { if (Secret.Puzzle(x) != Player.Puzzle(x)) throw new Exception(“Mismatch”); } } behavior Secret Impl == Player Impl 12 Take 1: Code
- 13. Code Hunt as Crowdsourcing Platform Secret Implementation class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); } } Player Implementation class Player { public static int Puzzle(int x) { return x; } } class Test { public static void Driver(int x) { if (Secret.Puzzle(x) != Player.Puzzle(x)) throw new Exception(“Mismatch”); } } behavior Secret Impl == Player Impl 13 Take 2: Process class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); } } class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); } } class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); } } class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); } } class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); } }
- 14. Code Hunt as Crowdsourcing Platform Secret Implementation class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); } } Player Implementation class Player { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); } } class Test { public static void Driver(int x) { if (Secret.Puzzle(x) != Player.Puzzle(x)) throw new Exception(“Mismatch”); } } behavior Secret Impl == Player Impl 14 Take 3: Code (w/o secret) class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); } } Passing Player Contributed Implementations class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); } } class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); } } class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); } } class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); } } Judge Metrics
- 15. Why Code Hunt as Crowdsourcing Platform?
- 16. Why Code Hunt as Crowdsourcing Platform? Crowdsourcing Scale & Open Data
- 17. Coding Duels 1,667,521 clicked 'Ask Pex!' http://pex4fun.com/
- 18. 2014 Beauty of Programming Contest Code Hunt can identify top coders
- 19. Code Hunt - the APCS (default) Zone • Opened in March 2014 • 129 problems covering the Advanced Placement Computer Science course • By August 2014, over 45,000 users started • By now, over 725,000 users 0 10000 20000 30000 40000 50000 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 1.101.111.121.131.141.15 2.1 2.2 2.3 2.4 2.5 2.6 2.7 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 Players Sector and Level APCS Zone, First three sectors, 45K to 1K
- 20. Code Hunt Usage Code Hunt has had several hundred thousands of users since launch in March 2014 Stats from Visual Studio Analytics over the period 9/19/14 and 10/20/14 indicate 1,150 users a day Stickiness (loyalty) is very high
- 21. APCS 725,199 ICSE 627 BoPQuali 13,778 BopA 1,017 BopB 1,103 BopSemi 1,164 CSTA 61 TEALS 14 BootCamp 30 Laser Sept 12 Polska Oct 344 ImCupJan 542 ImCupDec 222 ImCupNov 379 ImCupOct 831 ImCupSept 257 Total Try Count Average Try Count Max Try Count Total Solved Users 13374 363 1306 1581 Size of Code Hunt Data (so far) For ImCupSept 257 users x 24 puzzles x approx. 10 tries = about 13,000 programs #users
- 22. Code Hunt Data Released to the Public! http://research.microsoft.com/codehuntcommunity/ https://github.com/Microsoft/Code-Hunt The currently released data set contains the programs written by students (only) worldwide during a contest over 48 hours. There are approximately 250 users, 24 puzzles and about 13,000 programs. Judith Bishop, Nigel Horspool, Tao Xie, Nikolai Tillmann, and Jonathan de Halleux. Code Hunt: Experience with Coding Contests at Scale. In ICSE 2015 JSEET. http://web.engr.illinois.edu/~taoxie/publications/icse15jseet-codehunt.pdf http://research.microsoft.com/chese2015/
- 23. Summary: Testing Tool Educational Gaming DSE/Pex Pex for Fun/Code Hunt: Interactive Gaming for Teaching and Learning Support http://pex4fun.com https://www.codehunt.com http://research.microsoft.com/codehuntcommunity/ https://github.com/Microsoft/Code-Hunt
- 24. Testing Tool Educational Gaming Support http://research.microsoft.com/pex/ http://pex4fun.com Q & A Thank you! http://research.microsoft.com/codehuntcommunity/ https://github.com/Microsoft/Code-Hunt