Extending the Virtual World Framework for Mobile Training

Editor's Notes

• #4: Discuss contractContract Period of Performance (PoP) November 2012 – March 2013Work conducted for Joint Knowledge Online (JKO) – part of the Joint Staff DDJ7 Joint & Coalition Warfighting (JCW) Support TeamCounter Trafficking in Persons course subject matterCitations:JKO logo image, Joint Knowledge Online, 2013
• #5: JKO Training PlatformFocused upon Tier IIIGetting training to learners in the fieldJKO Mobile Pilot Program collaborative partnership between JKO and ADL to advance DoD mobile training
• #6: What is the VWF?2-D and 3D based visualizationsBrowser-basedPlugin-freeUsesJavascript, WebSockets, WebGL and yamlThe key is collaborationThe VWF is a new multi-user and collaborative web-based server that utilizes evolving and emerging web standards including Javascript, WebSockets, and WebGL. The VWF employs server and next generation browser technology that enables developers to quickly create 3-D scenes and allows multiple users to collaborate within the scenes in real-time. It is designed for users to share 2-D and 3-D visualizations and interactions through web browsers.Citations:Virtual World Framework Image, Virtual World Framework Web site, 2013http://www.virtualworldframework.com
• #7: Contact requirementsThe prototype training must be presented in a 3D perspective.The prototype training must link to an optional training moduleThe prototype training must run in a browserThe prototype training must be 5-8 minutes longThe prototype designs were based upon the following contract requirements:  Design a 3D virtual environment for the student to experience a virtual vignette relating to the lesson content of the CTIP course that is interactive with the student. This optional training module should be an interactive vignette of the lesson content.  The Contractor shall develop the interactive 3d virtual training module that will last between 5-8 minutes. The course shall run inside a browser, requiring no browser plug-ins (other than Adobe Flash), and load/stream with minimal time and effort. The CTIP mobile course will link to an optional virtual training module hosted by the developer at the developer site.Citations:Clock Image, 123 Free Vectors Web site, 2013http://www.123freevectors.com/white-office-clock/Browser Icons, Tech f5 Web site, 2013http://www.techf5.com/pic/2010/08/free-internet-software.jpg
• #8: Counter Trafficking in Persons (CTIP)Course subject matterDeals with identifying and reporting suspected child soldiering and forced labor of persons, including children.2.5 million people concurrently exploited, half are children95% of victims experience physical or sexual abuse at the hands of their tormentors43% of victims are forced into prostitution32% of victims are forced into slaveryThe Department of Defense Instruction, Combating Trafficking in Persons establishes policy in regard to all DoD Component heads to provide an annual awareness training program to all component staff and to provide data to the Office of the Secretary of Defense for Personnel and Readiness (OSD P&R) for its annual report [1]. Department of Defense policy states that CTIP is designed to oppose prostitution, forced labor, and any related activities such as coercion, commercial sex act, debt bondage, involuntary servitude and sex trafficking that may contribute to the phenomenon of TIP as inherently harmful and dehumanizing. Citations:CTIP Definition, Combatting Trafficking in Persons Web site, 2013http://ctip.defense.gov/Portals/12/Documents/TIP_DODI_220001p%5B1%5D.pdfUSAID Cover Image, USAID Web site, 2013http://www.cgdev.org/userfiles/image/USAID/CounterTraffickingInPersonsPolicy.PNGHuman Trafficking: The facts, UN.GIFT Global Initiative to Fight Human Trafficking Web sitehttp://www.caritas.org/includes/pdf/coatnet/traffickingfacts.pdf‎
• #9: Two prototypes were developedUrban explorer prototypeExplore vast spaces within a cityIdentify physical and behavioral signs of CTIP in the environmentSprite avatar prototypeExplore a single scene or “shoebox”Identify physical and behavioral signs of CTIP using avatarsThe primary prototype was designed to enable learners to explore a virtual urban environment for environmental, physical and behavioral signs of CTIP. This system is referred to as the “Urban Explorer Prototype” and was developed to satisfy all requirements. The supplemental prototype was designed to enable learners to monitor a sequence of interactions between avatars for signs physical and behavioral signs of CTIP. This system is referred to as the “Sprite Avatar Prototype” Citations:Urban Explorer Prototype, Engineering and Computing Simulations, 2013Sprite Avatar Prototype, Concurrent Technologies Corporation, 2013
• #10: Traditional VWF DevelopmentIntegrate CTIP models into a VWF scene node, which is another way of saying the “scene graph.”Integrate the VWF character controller in order for the learner to move around in the scene.Integrate CTIP leaner mission content, such as terminal and enabling learning objectives and tasks for identifying physical and behavioral signs of CTIP.Finally, to score learner performance and to give the learner performance feedback.This is a great new, emerging technology. However, (in spite of reports to the contrary) the maturity of WebGL on mobile was just not there.
• #11: Revised development approachStitch 3D imagery into a 2D panorama to give the “illusion” of a regular 3D projected game environment.Integrate a character controller suited to move about the panorama-based environment.Integrate Machinima or video using 3D assets in order to inspect the environment for physical and behavioral signs of CTIP.Score learner performance and provide learner feedback, as originally planned.This worked and performed well on mobile devices!DEMO!Machinima were then incorporated into the immersive environment for learners to discover through exploration and identification and to evaluate as possible human trafficking situations. The panoramas and the machinima used the same 3-D models and the same rendering engine for capture.Citations:Urban Explorer Prototype, Engineering and Computing Simulations, 2013
• #12: Whattechnologies did we end up using?VWF nodes to run machinimacontent within the VWFJquery & HTML5 for popupsKrpanofor the panoramasThese “played nice” together and ran inside and alongside the VWF.Krpano supported HTML5 on iOS and AndroidWe did not require collaborative featuresWe also created and Android-only pure panorama implementation that allowed for The first demonstration utilized an HTML5 panorama viewer that was incorporated into the VWF server. This demonstration became JKO’s operational version linked to the CTIP mobile course as it worked across both Android and iOS web browsers. The second demonstration utilized a panorama viewer but one that was developed utilizing the VWF’s native WebGL implementation. While not a requirement for the training, this demonstration had the added benefit of natively implementing the VWF’s collaborative communication. This allowed for one mobile device to control what another mobile device was viewing or vice versa. Citations:JQuery Logo, JQuery page on Wikipedia Web site, 2013http://en.wikipedia.org/wiki/File:JQuery_logo.svgHTML5 logo, HTML 5 page on Wikipedia Web site, 2013http://en.wikipedia.org/wiki/File:HTML5_logo_and_wordmark.svgVirtual World Framework Image, Virtual World Framework Web site, 2013http://www.virtualworldframework.com
• #13: Sprite avatar prototype evolution consisted of 3 experimentsExperiment 1 – All 3DExperiment 2 – Reduced 3D and some spritesExperiment 3 – Not 3D and all spritesThe final sprite avatar design was not determined at the outset of the prototype design effort and was instead arrived upon three (3) evolutionary iterative design experiments. The primary driver for the iterative experimentation was to develop an acceptable balance of visual and hardware performance while meeting compatibility requirements for the mobile browsers of choice. A fourth and tertiary design was also employed to present the learner with a more interactive and a usable experience. Citations:Sprite Avatar Experiments Timeline Image, Concurrent Technologies Corporation, 2013
• #14: What are sprites?2D or 2.5D images projected onto a rectangleAlways facing the cameraUsed for many popular classic and contemporary games to represent characters, environments and data overlays.“With mobile devices and casual gaming becoming more and more popular the classic 2D games return. The modern devices lack support for hardware sprites but come with powerful 3D hardware. On these devices sprites are simulated using textures on rectangular shapes. Perspective is disabled for these games. Since the hardware often comes with constraints - e.g. that it can only use power-of-two sized textures (that is a width or height of 64, 128, 256, ...) sprite sheets are used to reduce memory consumption. This is done by packing many sprites into one texture which, as a whole, has to meet the hardware constraints. Apart from the memory usage this technique can also be used to reduce the number of draw calls to the graphics subsystem and speed up rendering.“Citations:Wikipedia, “Sprites”, 2013http://en.wikipedia.org/wiki/Sprite_(computer_graphics)
• #15: Sprite avatar prototype test harnessUsed four devicesThe MacBook Pro was used to baseline and verify correct function, before testing on mobile devicesThe Nexus 7 is a 7” tablet and the ASUS a larger 10” The Samsung Galaxy is a phone, incidentally, owned by one of our developers
• #16: Experiment 1Run natively on mobileIn pursuit of a 3D environment and avatars, as commonly implemented in desktop browser-based VWF scenesImport modelsImport VWF character controllerImplement mission content regarding the identification of the physical and behavioral signs of CTIPImplement player scoringThis did not work!Measure the performance of mobile devices to run native VWF nodes, meshes and materials in pursuit of a 3D environment and avatars, as commonly implemented in desktop browser-based VWF scenes.Results indicated that mobile devices would either not load a scene or would load only part of a scene, but not the whole. In testing against the standard demos provided on the VWF site, we found similar results. It appeared execution of 3D mesh-based content on the devices was causing scene failure. Citations:Humvee image, Virtual World Framework Web site, 2013http://www.virtualworldframework.com/humvee
• #17: Experiment 2Run native VWF nodes, materials and reduced-volume 2D plane meshes In pursuit of a 3D environment and avatarsImport modelsImport VWF character controllerImplement mission content regarding the identification of the physical and behavioral signs of CTIPImplement player scoringImplement collaborative functionality. This worked amazingly well out of the box!This did not work!Measure the performance of mobile devices to run native VWF nodes, materials and reduced-volume 2D plane meshes mapped to 2D imagery, called sprites, in pursuit of memory efficient and better-performing environments and avatars. To do so required moving under the VWF abstraction layer and working directly with underlying GLGE graphics code.Results showed that the desired sprite-to-mesh mapping was successful and we had achieved synchronization across clients for viewing sprite animations when activated by users. However, in attempting to animate the sprites for purposes of character locomotion we encountered significant flicker that diminished visual aesthetics. In addition, we continued to experience scene failure across mobile devices tested.Citations:Experimental sprite image, Concurrent Technologies Corporation, 2013
• #18: Experiment 3Run native VWF scene nodewith no supporting VWF mesh or material nodesIn pursuit of a 3D environment and avatarsOverlay HTML 5 canvasOverlay sprites onto canvasCould click sprites and through to canvas using native VWF picker code1500 lines of VWF YAML and JavaScript codeShowedthe best functional and visual-aesthetic performance THIS WORKED!Measure the performance of mobile devices to run only VWF nodes, with no meshes or materials, while using sprites in pursuit of better-performing environments and avatars. To do so required overlaying an HTML 5 canvas over the VWF canvas and the development of 1500 lines of VWF YAML and JavaScript code.Results showed the best functional and visual-aesthetic performance of avatars and environments while again achieving synchronization across clients. However, we continued to experience scene failure across all, but a single device on which we achieved a useable experience.Citations:Sprite Avatar Image, Concurrent Technologies Corporation, 2013
• #19: The sprite avatar prototype incorporated a tertiary experiment to create a more well-balanced experience. These included:Sprite engine developed to translate and animate avatars,Knowledge checks were developed to help gauge experiential learning,After action review was developed to provide cumulative performance feedback to the learner,Timeline developed to control avatars, knowledge checks and AAR on one or more timelines.2D character controller developed to manually translate avatar across screen.An exercise event-sequencing engine was developed using JavaScript and YAML (YAML Ain’t Markup Language) to enable sequences of linear or parallel-running narrative events to occur on a prescribed timeline.Because an efficient and consistently working sprite canvas could not be established using the VWF material node and texture in the second experiment, it was determined that development a sprite engine would be necessary to display sprites on an HTML 5 canvas.An avatar locomotion and animation component was developed to enable the learner to translate the antagonist character from the narrative around the VWF scene using the W, A, S and D keys. A knowledge check component was developed in order to test the learner’s knowledge against what they had observed within the executed narrative. Citations:Sprites, AAR, knowledge check and timelines images, Concurrent Technologies Corporation, 2013
• #20: WebSockets is an emerging standard that developers should continue to monitorRuns over Port 80 along with regular web traffic eliminating the need for firewall security allowances Communication is full duplexallowing for an open and connection oriented experienceIt may be implemented across any client and server applicationPossesses browser support for Chrome, Firefox, Opera, Safari and Internet Explorer and mobile browsersWebSockets is an emerging standard that developers should continue to monitor. It’s well known as a browser-based standard but can also be used by native applications. As libraries that implement WebSockets continue to improve, new possibilities in supporting communication between native and browser-based applications will emerge. Such improvements may allow native versions of the VWF client to be produced for mobile devices that are optimized to handle 3-D content.Citations:HTML5 logo, HTML 5 page on Wikipedia Web site, 2013http://en.wikipedia.org/wiki/File:HTML5_logo_and_wordmark.svgBrowser Icons, Tech f5 Web site, 2013http://www.techf5.com/pic/2010/08/free-internet-software.jpg
• #21: WebGLJavaScript based APIAllows for 2D/3D graphics without a pluginPaints onto the HTML5 canvas elementPossesses browser support for Chrome, Firefox, Opera, Safari and Internet Explorer and mobile browsersA question is if the WebGLimplementation for VWF is optimized for mobile?Currently, mobile WebGL is only supported out-of-the-box on Firefox for Android. Chrome for Android may be configured to support WebGL and mobile Safari installed on iOS devices only makes WebGL available for advertisers using iAd. Citations:WebGL Logo, WebGL page on Wikipedia Web site, 2013http://en.wikipedia.org/wiki/File:WebGL_logo.pngElectrical plugimage,Electrical Outlet Clip Art Page on clker Web site, 2013http://www.clker.comBrowser Icons, Tech f5 Web site, 2013http://www.techf5.com/pic/2010/08/free-internet-software.jpg
• #22: Plug-in free games are making great progress!However, native running mobile applications are still the method of choiceProject Anarchy from HavokUDK mobileUnity mobile Many web-based 3D engines are not optimized for mobile browsers. Native is still the method of choice when creating 3D immersive environments for mobile devices. There are several mature 3D engines and development environments that support mobile platforms such as Unity, Project Anarchy, and Unreal.Citations:Unreal Engine Logo, Unreal Engine page on Wikipedia Web site, 2013http://en.wikipedia.org/wiki/File:Unreal_Engine_3_logo_and_wordmark.pngUnity Engine Logo, Unity Engine page on Wikipedia Web site, 2013http://en.wikipedia.org/wiki/File:Unity_3D_logo.pngProject Anarchy Logo, Project Anarchy page on Project Anarchy Web site, 2013http://www.projectanarchy.com/
• #23: Integrated Development Environments (IDEs) and supporting Application Programming Interfaces (APIs) are needed to make the VWF practical for developersIt’s understood that the VWF exists primarily to provide the collaborative and base 2D/3D functionality. It does this very well!Who will develop the rest of the components that modern games and simulation require?One editor has been created by the ADL initiative to build environments and incorporate avatars. That’s a fantastic start!Tools or components are needed to streamline and make practical the following:Gamification2D/3D GUI creationArtificial intelligenceData feedsGeospatialCollaborationMissions and learning objectivesAnimationSound and musicPhyiscsFinally, creation of IDEs supporting practical VWF application development tasks are desirable to include support for 2D and 3D graphical user interfaces development, music and sound integration, animation & modeling integration, learning objectives and learner missions development, event sequencing development, geospatial-based terrain integration, data feed integration, character AI integration, physics integration, collaboration support and gamification support.To borrow from the late great Bob Ross, IDEs and APIs make this happy little universe go round.Citations:WebGL Logo, WebGL page on Wikipedia Web site, 2013http://en.wikipedia.org/wiki/File:WebGL_logo.pngHandshake image, Handshake page on Wikipeida Web site, 2013http://commons.wikimedia.org/wiki/File:Handshake2.svgGraduation hat image, Graduation Hat page on Wikipedia Web site, 2013http://en.m.wikipedia.org/wiki/File:Graduation_hat.svgApple image, 123 Free Vectors Web site, 2013http://img1.123freevectors.com/wp-content/uploads/misc_big/028_misc_red-apple-free-vector.jpgLIDAR Image, University of Washington Vegetation Monitoring and Remote Sensing Team Web site, 2013http://forsys.cfr.washington.edu/JFSP06/lidar_&_ifsar_tools.htmA* Pathfinding Image, A* Pathfinding page on Wikipeida Web site, 2013http://en.wikipedia.org/wiki/File:Pathfinding_2D_Illustration.svgGamification icons image, The Icon Deposit Web site, 2013http://www.icondeposit.com/theicondeposit:23
• #24: Investigate the development of an optimized WebGL driver for VWFIs the driver already optimized and can’t perform or,Is there an opportunity to improve it for the VWF?Augmentation of the WebGL VWF driver for optimal mobile browser operation is desierable.Citations:WebGL Logo, WebGL page on Wikipedia Web site, 2013http://en.wikipedia.org/wiki/File:WebGL_logo.png
• #25: Investigate the development of a standalone VWF applicationSupports native VWF contentSupports VWF communication protocolsInterim or parallel step until the plug-in free Web browser version catches upDevelopment of a standalone application supporting VWF content and communication protocols.Citations:Unreal Engine Logo, Unreal Engine page on Wikipedia Web site, 2013http://en.wikipedia.org/wiki/File:Unreal_Engine_3_logo_and_wordmark.pngUnity Engine Logo, Unity Engine page on Wikipedia Web site, 2013http://en.wikipedia.org/wiki/File:Unity_3D_logo.pngProject Anarchy Logo, Project Anarchy page on Project Anarchy Web site, 2013http://www.projectanarchy.com/