This is lecture 4 in the course on Rapid Prototyping for XR, taught by Mark Billinghurst on June 11th, 2025. This lecture is about High Level Prototyping.
5. From Sketch to Prototype
Sketch Wireframe Mock-up Prototype
6. Not about drawing, but design
Sketching is a tool to help you:
• express
• develop, and
• communicate design ideas
Sketching is part of a process
• Idea generation
• Design elaboration
• Design choices
• Engineering
“Sketching is about the Activity
not the Result” - Bill Buxton
Sketching
8. Sketching in VR
Using VR applications for rapid prototyping
- Intuitive sketching in immersive space
- Creating/testing at 1:1 scale
- Rapid UI design/layout
Examples
- Open Brush - https://openbrush.app/
- Quill - https://quill.art/
More tools see - https://www.roadtovr.com/vr-painting-drawing-modeling-animation-art-tools-quest-pc/
9. Storyboards
Storyboarding is a technique taken from film making
using a simple cartoon-like structure, key moments
from the interactive experience are represented.
The advantage of storyboarding is that it allows you to
get a feel for the ‘flow’ of the experience.
It is also a very economical way of representing the
design – a single page can hold 6–8 ‘scenes’.
10. Key Elements
1. Scenario: Storyboards are based on a scenario or a user story. The
persona or role that corresponds to that scenario is clearly specified
2. Visuals: Each step in the scenario is represented visually in a
sequence. The steps can be sketches, illustrations, or photos.
3. Captions: Each visual has a corresponding caption. The caption
describes the user’s actions, environment, emotional state, device, etc.
11. Wireframes
It’s about
- Functional specs
- Navigation and interaction
- Functionality and layout
- How interface elements work together
- Defining the interaction flow/experience
Leaving room for the design to be created
12. Mockup
It’s about
- Look and feel
- Building on wireframe
- High fidelity visuals
- Putting together final assets
- Getting feedback on design
Priyaa Sathiyaseelan - https://priyaasathiyaseelan.com/portfolio/cook-simple-ar-app/
13. Scene Assembly
•Assemble assets into 3D scene
•Prototype final UI concept
•Create high-fidelity view
•Collect user feedback
•Immersive Scene Assembly
•Sketchbox: https://www.sketchbox3d.com/
14. Vuforia Studio
• Author animated AR experiences
• Drag and drop content
• Add animations
• Import CAD models
• Combine with IOT sensors
• https://www.ptc.com/en/products/vuforia/vuforia-studio
15. Mock-up Guidelines
1. Generate final 2D/3D interface elements
2. Replace wireframe UI elements with high quality visuals
3. Use standard AR/VR UI elements
4. Simulate AR/VR views
5. Focus on visual/audio design
6. Collect feedback from target end-users
16. Sketch vs. Wireframe vs. Mock-up
Low Fidelity Low to Medium
Fidelity
Medium to High Fidelity
IDEATE FLOW VISUALIZE
19. Adding Transitions
“Unless you can show me
where you’ve fleshed out the
character and aspects of the
transitions at the same level
of … fidelity as you have the
states – you’re fired.”
Bill Buxton
21. Body Storming/Video Sketching
• Mockup/act out user interaction
• Capture key elements of experience
• Use paper prototypes + real people
• Demonstrate/video the transitions
22. 360 Paper Prototyping & Testing
Nebeling & Madier: 360proto: Making Interactive Virtual Reality & Augmented Reality Prototypes from Paper (CHI 2019)
23. 360 Paper Prototype
Nebeling & Madier: 360proto: Making Interactive Virtual Reality & Augmented Reality Prototypes from Paper (CHI 2019)
24. Video Sketching
• Process
• Capture elements of real world
• Use series of still photos/sketches in a movie format.
• Act out using the product
• Benefits
• Demonstrates the product experience
• Discover where concept needs fleshing out.
• Communicate experience and interface
• You can use whatever tools you want, e.g. iMovie.
27. Solar System AR/VR Prototype
• Goal: Use AR/VR to educate about Solar System
• Process
1. Design ideas and critique
2. Storyboard
3. 360 storyboard
4. 3D/VR storyboard
5. Paper prototype
6. 360 paper prototype
7. 3D models
8. Diorama
9. Immersive models
10. Immersive VR
11. Final AR/VR
48. XR Tools Landscape
Digital & Immersive Authoring
Proto.io, Figmin, Tour Creator, ...
Open Brush, Blocks, ShapesXR, …
Web-Based Development
THREE.js, Babylon.js, …
A-Frame, AR.js, …
Cross-Platform Development
Unity + SDKs
Unreal + SDKs
Native Development
Cardboard/Meta/Vive/... SDK
Vuforia/ARCore/ARKit/… SDK
49. XR Tools Landscape
Digital & Immersive Authoring
Good for storyboarding but limited
support for interactions
Web-Based Development
Good for basic XR apps but often
interactions feel unfinished
Native Development
Good for full-fledged XR apps but
limited to a particular platform
Cross-Platform Development
Good for full-fledged XR apps but
usually high learning curve
50. Digital Authoring Tools for VR
• Support visual authoring of 3D
scene graphs with VR previews
• Basic interactions can be
implemented without coding
• Advanced interactions require
JavaScript, C#, or C++
Niantic Studio
Unity Editor
51. Digital Authoring Tools for AR
Vuforia Studio
Lens Studio
• Support visual authoring of marker-
based and/or marker-less AR apps
• Provide default markers and support
for custom markers
• Typically enable AR previews
through emulator but need to deploy
to AR device for testing
52. Immersive Authoring Tools for VR
• Enable visual authoring of 3D
content in VR
• Make it possible to edit while
previewing VR experience
• Focus on 3D modeling rather than
animation & scripting
• Typically support export to common
3D model formats and asset sharing
platforms like Google Poly,
Sketchfab, or 3D Warehouse
Google Blocks
Oculus Quill
53. Immersive Authoring Tools for AR
• Enable visual authoring of 3D
content in AR
• Make it possible to edit while
previewing AR experience in the
environment
• Provide basic support for interactive
behaviors
• Sometimes support export to
WebXR
Apple Reality Composer
Adobe Aero
55. Mobile AR
• Many tools available for rapid prototyping
• Standard mobile phone tools
• Types
• Sketching
• Pop, etc..
• Web development
• Proto.io, Figma, etc..
• Interactive Apps
• Orginami, etc.
56. Interactive Sketching
•Pop App
● Pop - https://marvelapp.com/pop
● Combining sketching and interactivity on mobiles
● Take pictures of sketches, link pictures together
59. Proto.io
• Web based prototyping tool
• Visual drag and drop interface
• Rich transitions
• Scroll, swipe, buttons, etc
• Deploy on device
• mobile, PC, browser
• Ideal for mobile interfaces
• iOS, Android template
• For low and high fidelity prototypes
61. Figma - https://www.figma.com/
• Web based tool for interface design and prototyping
• Collaborative web application, visual editing
• Key Features
• vector graphics editor
• prototyping tools for mobile, tablet, desktop
• support for real-time collaboration
• advanced interactive behaviours
• mobile app for Android and iOS
65. Origami Studio - https://origami.design/
• Free design tool created by Meta (Mac only)
• Goal: prototype small, interactive, app-interactions
that will eventually be handed off to developers
• Features
• Rapidly build and share interactive interfaces.
• Drag and drop/visual programming
• Deploy on mobile phones
69. Case Study: Interactive Flight AR App
https://blog.prototypr.io/making-flights-more-interesting-with-augmented-reality-322eecde82
70. Flight Augmented Reality Application
● Hold up mobile phone to see
POI out of plane window
● AR tags in real world
● Tap tags for more information
72. Design Interface Screens
● Using Sketch - https://www.sketch.com/
73. Create AR View
● Use looped video
● Load into Origami Studio
● Add POI onto video
● Animate POI with video
● Add touch interactions to POI
https://www.youtube.com/watch?v=ejzS0atCtgQ
74. Add POI and Move with Video
● Use Origami Studio - https://origami.design/
77. Mobile AR Visual Programming
• Rapid prototype on desktop
• Deliver on mobile
• Visual programming
• Simple interactivity
• Examples
• Zapworks Studio
• https://zap.works/studio/
• Snap Lens Studio
• https://lensstudio.snapchat.com/
• TikTok Effect House
• https://effecthouse.tiktok.com/
79. SnapChat
• Snap Chat
• Most popular mobile AR platform
• 320+ million users/day
• Live camera input
• AR visual features
• Snap Spectacles
• Waveguide displays
• Gesture interaction
• Full integrated
80. Snap Lens Studio - SnapChat
Author and preview AR prototypes
● Tool behind Snapchat Lenses, but
also a powerful AR prototyping tool
● Can do face (using front camera)
and world lenses (rear camera)
● Simulated previews using webcam
Deploy and use advanced AR features
● Can deploy to phone running
Snapchat app via Snapcode
● Has advanced AR tracking and
segmentation capabilities
85. Creating on Device
• Adobe Aero
• Create AR on mobile devices
• Touch based interaction and authoring
• Only iOS support for now
• https://www.adobe.com/nz/products/aero.html
87. Apple Reality Composer
• Rapidly create 3D scenes and AR experiences
• Creation on device (iPhone, iPad)
• Drag and drop interface
• Loading 2D/3D content
• Adding simple interactivity
• Anchor content in real world (AR view)
• Planes (vertical, horizontal), faces, images
88. Apple Reality Composer Example
https://www.youtube.com/watch?v=tRcAGdM7878
92. ShapesXR - www.shapesxr.com
•VR creation and collaboration platform
• Mockup AR/VR interfaces
• Collaborative Support collaborative viewing
• Import 2D/3D assets
• Export to 3D development tools
• Available for Oculus Quest
99. Interactive 360 Prototyping for VR
• Create 360 images and add interactive elements
• Many possible tools
• InstaVR
• http://www.instavr.co/
• Free, fast panorama VR
• Drag and drop web interface
• Deploy to multi platforms (Quest, Vive, phone, etc)
• VR Direct
• https://www.vrdirect.com/
• Connect multiple 360 scenes
• Instant content update
• EasyVR
• https://www.360easyvr.com/
100. Demo - Using InstaVR
• https://www.youtube.com/watch?v=ce4Pww3Up10
104. Niantic Studio - www.8thwall.com
• Web based authoring for 3D/XR
• Key features
• WebXR deployment
• Visual editing (drag and drop)
• Live editing
• Interactive elements
• XR Features
• Face Effects, World Tracking, Image Targets, & VPS
• Cross platform development
• Apple Vision Pro, Meta Quests, Magic Leap, etc