2013 Lecture 5: AR Tools and Interaction

COSC 426: Augmented Reality
Mark Billinghurst
mark.billinghurst@hitlabnz.org
August 9th 2013
Lecture 5: AR Tools and Interaction

Data Gathering Techniques (1)
  Questionnaires
  Looking for specific information
  Qualitative and quantitative results
  Good for getting data from a large, dispersed group
  Interviews
  Good for exploring issues, using props
  Structured, unstructured or semi-structured
  But are time consuming and difficult to visit everyone

Data Gathering Techniques (2)
  Workshops or focus groups
  Group interviews/activities
  Good at gaining a consensus view and/or highlighting areas
of conflict
  Observations
  Spending time with users in day to day tasks
  Good for understanding task context
  requires time and can result in a huge amount of data
  Documentation
  Procedures and rules written down in manuals

Elaboration and Reduction
  Elaborate - generate solutions. These are the opportunities
  Reduce - decide on the ones worth pursuing
  Repeat - elaborate and reduce again on those solutions
Source: Laseau,P. (1980) Graphic Thinking for Architects & Designers. John Wiley and Sons

Tools for Effective Design
 Personas
 Scenarios
 Storyboards
 Wireframes and Mock-ups
 Prototypes

Persona Technique
•  Personas are a design tool to help visualize who you are
designing for and imagine how person will use the product
•  A persona is an archetype that represents the behavior and
goals of a group of users
•  Based on insights and observations from customer research
•  Not real people, but synthesised from real user characteristics
•  Bring them to life with a name, characteristics, goals, background
•  Develop multiple personas

Gunther the Ad Guy
Gunther is from Germany. He
Travels extensively for work and
As he is an advertising executive
he needs to present concepts to
clients quickly and easily. He is
a person very well-versed in new
technologies and wishes he had
easier portable solutions for his
presentations…..

Scenarios
Usage Scenarios are narrative descriptions of how the product
meets the needs of a persona
Short (2 pages max)
Focus on unmet needs of persona
Concrete story
Set of stories around essential tasks, problems...
Use to test ideas

Storyboarding
Sequence of sketches showing use of system in
everyday use context
Concrete example
Easier (faster) to grasp than text based stories
Means of communication with users and system
developers
Sketches, not drawings...
Use to test interaction and make sure design works

Sketching is about design
Sketching is not about drawing
It is about design.
Sketching is a tool to help you:
-  express
-  develop, and
-  communicate design ideas
Sketching is part of a process:
-  idea generation,
-  design elaboration
-  design choices,
-  engineering

Sketch vs. Prototype
Sketch
Prototype

Invite
A)end

Suggest
Describe

Explore
Reﬁne

Ques;on
Answer

Propose
Test

Provoke
Resolve

Tenta;ve,
non
commi)al
Speciﬁc
Depic;on

The primary differences are in the intent

Types of Prototypes
Low Fidelity – quick and dirty, easy access
materials like cardboard and paper.
High Fidelity – more involved electronic
versions similar in materials to final product.

RAPID Prototyping
  Fast and inexpensive
  Identifies problems before they’re coded
  Elicits more and better feedback from users
  Helps developers think creatively
  Gets users and other stakeholders involved early
  Fosters teamwork and communication
  Avoids opinion wars
  Helps decide design directions

Paper Prototyping (Low Fidelity)
Quick and simple means of sketching interfaces
Use office materials
Easier to criticize, quick to change
Creative process (develop in team)
Can also use for usability test (focus on flow of
interaction rather than visuals)
Used a lot to test out concepts before real design begins.

High-fidelity prototyping
•  Uses materials that you would expect to be in the
final product.
•  Prototype looks more like the final system than a
low-fidelity version.
•  For a high-fidelity software prototype common
environments include Macromedia Director,Visual
Basic, and Smalltalk.
•  Danger that users think they have a full
system…….see compromises

Rapid Prototyping
  Speed development time with quick hardware mockups
  handheld device connected to PC
  LCD screen, USB phone keypad, Camera
  Can use PC development tools for rapid development
  Flash, Visual Basic, etc

experiences
applications
tools
components
Sony CSL © 2004
Building Compelling AR Experiences
Tracking, Display
Authoring

AR Authoring Tools
  Low Level Software Libraries
  osgART, Studierstube, MXRToolKit
  Plug-ins to existing software
  DART (Macromedia Director), mARx, Unity,
  Stand Alone
  AMIRE, BuildAR, Metaio Creator etc
  Rapid Prototyping Tools
  Flash, OpenFrameworks, Processing, Arduino, etc
  Next Generation
  iaTAR (Tangible AR)

ARToolKit (Kato 1998)
  Open source – computer vision based AR tracking
  http://artoolkit.sourceforge.net/

ARToolKit Structure
  Three key libraries:
  AR32.lib – ARToolKit image processing functions
  ARgsub32.lib – ARToolKit graphics functions
  ARvideo.lib – DirectShow video capture class
DirectShow
ARvideo.lib

Software
  Cross platform
  Windows, Mac, Linux, IRIX, Symbian, iPhone, etc
  Additional basic libraries
  Video capture library (Video4Linux, VisionSDK)
  OpenGL, GLUT
  Requires a rendering library
  Open VRML, Open Inventor, osgART, etc

Additional Software
  ARToolKit just provides tracking
  For an AR application you’ll need more software
  High level rendering library
  Open VRML, Open Inventor, osgART, etc
  Audio Library
  Fmod, etc
  Peripheral support

What does ARToolKit Calculate?
  Position of makers in the camera coordinates
  Pose of markers in the camera coordinates
  Output format
  3x4 matrix format to represent the
transformation matrix from the marker
coordinates to the camera coordinates

From Marker To Camera
  Rotation & Translation
TCM : 4x4 transformation matrix
from marker coord. to camera coord.

An ARToolKit Application
  Initialization
  Load camera and pattern parameters
  Main Loop
  Step1. Image capture and display
  Step2. Marker detection
  Step3. Marker identification
  Step4. Getting pose information
  Step5. Object Interactions/Simulation
  Step6. Display virtual objects
  End Application
  Camera shut down

Sample1.c Main Function
main()!
{!
!init();!
!argMainLoop( mouseEvent, !
! !keyEvent, mainLoop); !
}!

Sample1.c - mainLoop Function
if( dataPtr = (ARUint8 *)
arVideoGetImage()) == NULL ) {
arUtilSleep(2);
return;
}
argDrawMode2D();
argDispImage(dataPtr, 0, 0 );
arVideoCapNext();
argSwapBuffers();

Ex. 2: Detecting a Marker
  Program : sample2.c
  Key points
  Threshold value
  Important external variables
  arDebug – keep thresholded image
  arImage – pointer for thresholded image
  arImageProcMode – use 50% image for image
processing
-  AR_IMAGE_PROC_IN_FULL
-  AR_IMAGE_PROC_IN_HALF

Sample2.c – marker detection
/* detect the markers in the video frame */
if(arDetectMarker(dataPtr, thresh,
&marker_info, &marker_num) < 0 ) {
cleanup();
exit(0);
}
for( i = 0; i < marker_num; i++ ) {
argDrawSquare(marker_info[i].vertex,0,0);
}

Making a pattern template
  Use of utility program:
mk_patt.exe
  Show the pattern
  Put the corner of red line
segments on the left-top
vertex of the marker
  Pattern stored as a
template in a file
  1:2:1 ratio determines the
pattern region used

Ex. 4 – Getting 3D information
  Key points
 Definition of a real marker
 Transformation matrix
-  Rotation component
-  Translation component

Sample4.c – Transformation matrix
double marker_center[2] = {0.0, 0.0};
double marker_width = 80.0;
double marker_trans[3][4];
arGetTransMat(&marker_info[i],
marker_center, marker_width,
marker_trans);

Finding the Camera Position
This function sets transformation matrix from marker
to camera into marker_trans[3][4]."
arGetTransMat(&marker_info[k], marker_center,
marker_width, marker_trans);
You can see the position information in the values of
marker_trans[3][4]."
" Xpos = marker_trans[0][3];
Ypos = marker_trans[1][3];
Zpos = marker_trans[2][3];

Ex. 5- Virtual Object Display
  Key points
 OpenGL parameter setting
 Setup of projection matrix
 Setup of modelview matrix

Appending your own OpenGL code
Set the camera parameters to OpenGL Projection matrix.
argDrawMode3D();
argDraw3dCamera( 0, 0 );
Set the transformation matrix from the marker to the camera to
the OpenGL ModelView matrix.
argConvGlpara(marker_trans, gl_para);
glMatrixMode(GL_MODELVIEW);
glLoadMatrixd( gl_para );
After calling these functions, your OpenGL objects are
drawn in the real marker coordinates.

ARToolKit in the World
  Hundreds of projects
  Large research community

ARToolKit Family
ARToolKit ARToolKit NFT
ARToolKit (Symbian)
NyToolKit
- Java, C#,
- Android, WM
JARToolKit (Java)
FLARToolKit (Flash)
FLARManager (Flash)

FLARToolKit
  Flash AS3 Version of the ARToolKit
(was ported from NyARToolkit the Java Version of the ARToolkit)
  enables augmented reality in the Browser
  uses Papervision3D for as 3D Engine
  available at http://saqoosha.net/
  dual license, GPL and commercial license

FLARToolkit
Papervision 3D
Adobe Flash
AR Application Components

private function mainEnter(e:Event):void {
/* Capture video frame*/
capture.draw(vid);
/* Detect marker */
if (detector.detectMarkerLite(raster, 80) && detector.getConfidence() > 0.5)
{
//Get the transfomration matrix for the current marker position
detector.getTransformMatrix(trans);
//Translates and rotates the mainContainer so it looks right
mainContainer.setTransformMatrix(trans);
//Render the papervision scene
renderer.render();
}
}

Papervision 3D
  http://www.papervision3d.org/
  Flash-based 3D-Engine
  Supports
  import of 3D Models
  texturing
  animation
  scene graph
  alternatives: Away3d, Sandy,…

Source Packages
  „Original“ FLARToolkit (Libspark, Saqoosha) (
http://www.libspark.org/svn/as3/FLARToolKit/trunk/ )
  Start-up-guides
  Saqoosha (http://saqoosha.net/en/flartoolkit/start-up-guide/ )
  Miko Haapoja (http://www.mikkoh.com/blog/?p=182 )
  „Frameworks“
  Squidder MultipleMarker – Example (
http://www.squidder.com/2009/03/06/flar-how-to-multiple-instances-of-multiple-
markers/ )
  FLARManager (http://words.transmote.com/wp/flarmanager/ )

Other Languages
  NyARToolKit
  http://nyatla.jp/nyartoolkit/wp/
  AS3, Java, C#, Processing, Unity, etc
  openFrameworks
  http://www.openframeworks.cc/
  https://sites.google.com/site/ofauckland/examples/8-artoolkit-example
  Support for other libraries
-  Kinect, Audio, Physics, etc

void testApp::update(){ //capture video and detect markers
mov.update();
if (mov.isFrameNew()) {
img.setFromPixels(mov.getPixels(), ofGetWidth(), ofGetHeight());
gray = img;
tracker.setFromPixels(gray.getPixels());
}
}
//--------------------------------------------------------------
void testApp::draw(){ //draw AR objects
ofSetColor(0xffffff); mov.draw(0, 0);
for (int i=0; i<tracker.markers.size(); i++) {
ARMarkerInfo &m = tracker.markers[i];
tracker.loadMarkerModelViewMatrix(m);
ofSetColor(255, 255, 0, 100); ofCircle(0,0,25); ofSetColor(0);
ofDrawBitmapString(ofToString(m.id),0,0);
}
}

Low Level Mobile AR Tools
  Vuforia Tracking Library (Qualcomm)
  Vuforia.com
  iOS, Android
  Computer vision based tracking
  Marker tracking, 3D objects, frame
markers
  Integration with Unity
  Interaction, model loading

OSGART Programming Library
  Integration of ARToolKit with a High-Level
Rendering Engine (OpenSceneGraph)
OSGART= OpenSceneGraph + ARToolKit
  Supporting Geometric + Photometric Registration

osgART:Features
  C++ (but also Python, Lua, etc).
  Multiple Video Input supports:
  Direct (Firewire/USB Camera), Files, Network by
ARvideo, PtGrey, CVCam, VideoWrapper, etc.
  Benefits of Open Scene Graph
  Rendering Engine, Plug-ins, etc

mARx Plug-in
  3D Studio Max Plug-in
  Can model and view AR content at the same time

BuildAR
  http://www.buildar.co.nz/
  Stand alone application
  Visual interface for AR model viewing application
  Enables non-programmers to build AR scenes

Metaio Creator
  Drag and drop AR
  http://www.metaio.com/creator/

Total Immersion D’Fusion Studio
  Complete commercial authoring platform
  http://www.t-immersion.com/
  Multi-platform
  Markerless tracking
  Scripting
  Face tracking
  Finger tracking
  Kinect support

Others
  AR-Media
  http://www.inglobetechnologies.com/
  Google sketch-up plug-in
  LinceoVR
  http://linceovr.seac02.it/
  AR/VR authoring package
  Libraries
  JARToolKit, MXRToolKit, ARLib, Goblin XNA

More Libraries
  JARToolKit
  MRToolKit, MXRToolKit, ARLib, OpenVIDIA
  DWARF, Goblin XNA
  AMIRE
  D’Fusion

Advanced Authoring: iaTAR (Lee 2004)
  Immersive AR Authoring
  Using real objects to create AR applications

osgART
Developing Augmented Reality
Applications with osgART

What is a Scene Graph?
  Tree-like structure for organising a virtual world
  e.g. VRML
  Hierarchy of nodes that define:
  Groups (and Switches, Sequences etc…)
  Transformations
  Projections
  Geometry
  …
  And states and attributes that define:
  Materials and textures
  Lighting and blending
  …

Benefits of a Scene Graph
  Performance
  Structuring data facilitates
optimization
-  Culling, state management, etc…
  Abstraction
  Underlying graphics pipeline is
hidden
  Low-level programming (“how do I
display this?”) replaced with high-
level concepts (“what do I want to
display?”)
Image: sgi

About Open Scene Graph
  http://www.openscenegraph.org/
  Open-source scene graph implementation
  Based on OpenGL
  Object-oriented C++ following design pattern principles
  Used for simulation, games, research, and industrial projects
  Active development community
  Maintained by Robert Osfield
  ~2000 mailing list subscribers
  Documentation project: www.osgbooks.com
  Uses the OSG Public License (similar to LGPL)

About Open Scene Graph (2)
Pirates of the XXI Century Flightgear
3DVRII Research Institute EOR
SCANeR
VRlab Umeå University

Open Scene Graph Features
  Plugins for loading and saving
  3D: 3D Studio (.3ds), OpenFlight (.flt), Wavefront (.obj)…
  2D: .png, .jpg, .bmp, QuickTime movies
  NodeKits to extend functionality
  e.g. osgShadow
  Cross-platform support for:
  Window management (osgViewer)
  Threading (OpenThreads)

Open Scene Graph Architecture
Plugins read and
write 2D image and
3D model files
NodeKits extend
core functionality,
exposing higher-level
node types
Scene graph and
rendering
functionality
Inter-operability with
other environments,
e.g. Python

Some Open Scene Graph Demos
  You may want to get the OSG data package:
  Via SVN: http://www.openscenegraph.org/svn/osg/OpenSceneGraph-Data/trunk
osgviewer osgmotionblur osgparticle
osgreflect osgdistortion osgfxbrowser

Learning OSG
  Check out the Quick Start Guide
  Free PDF download at http://osgbooks.com/, Physical copy $13US
  Join the mailing list: http://www.openscenegraph.org/projects/osg/wiki/
MailingLists
  Browse the website: http://www.openscenegraph.org/projects/osg
  Use the forum: http://forum.openscenegraph.org
  Study the examples
  Read the source? 

What is osgART?
  osgART adds AR to Open Scene Graph
  Further developed and enhanced by:
  Julian Looser
  Hartmut Seichter
  Raphael Grasset
  Current version 2.0, Open Source
  http://www.osgart.org

osgART Approach: Basic Scene Graph
Root
Transform
3D Object
0.988 -0.031 -0.145 0
-0.048 0.857 -0.512 0
0.141 0.513 0.846 0
10.939 29.859 -226.733 1
[ ]
  To add Video see-through AR:
  Integrate live video
  Apply correct projection matrix
  Update tracked transformations in
realtime

osgART Approach: AR Scene Graph
Root
Transform
3D Object

Video
Geode
Root
Transform
3D Object
Virtual
Camera
Video
Layer

Video
Geode
Root
Transform
3D Object
Virtual
Camera
Projection matrix from
tracker calibration
Transformation matrix
updated from marker
tracking in realtimeVideo
Layer
Full-screen quad
with live texture
updated from
Video source
Orthographic
projection

osgART Architecture
  Like any video see-through AR library, osgART requires video
input and tracking capabilities.
ARLibrary
Application
Video Source
e.g. DirectShow
Tracking Module
(libAR.lib)

osgART Architecture
  osgART uses a plugin architecture so that video sources and tracking
technologies can be plugged in as necessary
osgART
Application
VideoPluginTrackerPlugin
ARToolKit4 -
ARToolkitPlus -
MXRToolKit -
ARLib -
bazAR (work in progress) -
ARTag (work in progress) -
OpenCVVideo -
VidCapture -
CMU1394 -
PointGrey SDK -
VidereDesign -
VideoWrapper -
VideoInput -
VideoSource -
DSVL -
Intranel RTSP -

Basic osgART Tutorial
  Develop a working osgART application from scratch.
  Use ARToolKit 2.72 library for tracking and video
capture

osgART Tutorial 1: Basic OSG Viewer
  Start with the standard Open Scene Graph Viewer
  We will modify this to do AR!

osgART Tutorial 1: Basic OSG Viewer
  The basic osgViewer…
#include <osgViewer/Viewer>
#include <osgViewer/ViewerEventHandlers>
int main(int argc, char* argv[]) {
// Create a viewer
osgViewer::Viewer viewer;
// Create a root node
osg::ref_ptr<osg::Group> root = new osg::Group;
// Attach root node to the viewer
viewer.setSceneData(root.get());
// Add relevant event handlers to the viewer
viewer.addEventHandler(new osgViewer::StatsHandler);
viewer.addEventHandler(new osgViewer::WindowSizeHandler);
viewer.addEventHandler(new osgViewer::ThreadingHandler);
viewer.addEventHandler(new osgViewer::HelpHandler);
// Run the viewer and exit the program when the viewer is closed
return viewer.run();
}

osgART Tutorial 2: Adding Video
  Add a video plugin
  Load, configure, start video capture…
  Add a video background
  Create, link to video, add to scene-graph

  New code to load and configure a Video Plugin:
// Preload the video and tracker
int _video_id = osgART::PluginManager::getInstance()->load("osgart_video_artoolkit2");
// Load a video plugin.
osg::ref_ptr<osgART::Video> video =
dynamic_cast<osgART::Video*>(osgART::PluginManager::getInstance()->get(_video_id));
// Check if an instance of the video stream could be created
if (!video.valid()) {
// Without video an AR application can not work. Quit if none found.
osg::notify(osg::FATAL) << "Could not initialize video plugin!" << std::endl;
exit(-1);
}
// Open the video. This will not yet start the video stream but will
// get information about the format of the video which is essential
// for the connected tracker.
video->open();

  New code to add a live video background
osg::ref_ptr<osg::Group> videoBackground = createImageBackground(video.get());
videoBackground->getOrCreateStateSet()->setRenderBinDetails(0, "RenderBin");
root->addChild(videoBackground.get());
video->start();
osg::Group* createImageBackground(osg::Image* video) {
osgART::VideoLayer* _layer = new osgART::VideoLayer();
_layer->setSize(*video);
osgART::VideoGeode* _geode = new osgART::VideoGeode(osgART::VideoGeode::USE_TEXTURE_2D, video);
addTexturedQuad(*_geode, video->s(), video->t());
_layer->addChild(_geode);
return _layer;
}
  In the main function…

osgART Tutorial 3: Tracking
  Add a Tracker plugin
  Load, configure, link to video
  Add a Marker to track
  Load, activate
  Tracked node
  Create, link with marker via tracking callbacks
  Print out the tracking data

int _tracker_id = osgART::PluginManager::getInstance()->load("osgart_tracker_artoolkit2");
osg::ref_ptr<osgART::Tracker> tracker =
dynamic_cast<osgART::Tracker*>(osgART::PluginManager::getInstance()->get(_tracker_id));
if (!tracker.valid()) {
// Without tracker an AR application can not work. Quit if none found.
osg::notify(osg::FATAL) << "Could not initialize tracker plugin!" << std::endl;
exit(-1);
}
// get the tracker calibration object
osg::ref_ptr<osgART::Calibration> calibration = tracker->getOrCreateCalibration();
// load a calibration file
if (!calibration->load("data/camera_para.dat"))
{
// the calibration file was non-existing or couldnt be loaded
osg::notify(osg::FATAL) << "Non existing or incompatible calibration file" << std::endl;
exit(-1);
}
// set the image source for the tracker
tracker->setImage(video.get());
osgART::TrackerCallback::addOrSet(root.get(), tracker.get());
// create the virtual camera and add it to the scene
osg::ref_ptr<osg::Camera> cam = calibration->createCamera();
root->addChild(cam.get());
  Load a tracking plugin and associate it with the video plugin

osg::ref_ptr<osgART::Marker> marker = tracker->addMarker("single;data/patt.hiro;80;0;0");
if (!marker.valid())
{
// Without marker an AR application can not work. Quit if none found.
osg::notify(osg::FATAL) << "Could not add marker!" << std::endl;
exit(-1);
}
marker->setActive(true);
osg::ref_ptr<osg::MatrixTransform> arTransform = new osg::MatrixTransform();
osgART::attachDefaultEventCallbacks(arTransform.get(), marker.get());
cam->addChild(arTransform.get());
  Load a marker and activate it
  Associate it with a transformation node (via event callbacks)
  Add the transformation node to the virtual camera node
osgART::addEventCallback(arTransform.get(), new osgART::MarkerDebugCallback(marker.get()));
  Add a debug callback to print out information about the tracked marker

  Tracking information is
output to console

osgART Tutorial 4: Adding Content
  Now put the tracking data to use!
  Add content to the tracked transform
  Basic cube code
arTransform->addChild(osgART::testCube());
arTransform->getOrCreateStateSet()->setRenderBinDetails(100, "RenderBin");

osgART Tutorial 5: Adding 3D Model
  Open Scene Graph can load some 3D formats directly:
  e.g. Wavefront (.obj), OpenFlight (.flt), 3D Studio (.3ds), COLLADA
  Others need to be converted
  Support for some formats is much better than others
  e.g. OpenFlight good, 3ds hit and miss.
  Recommend native .osg and .ive formats
  .osg – ASCII representation of scene graph
  .ive – Binary OSG file. Can contain hold textures.
  osgExp : Exporter for 3DS Max is a good choice
  http://sourceforge.net/projects/osgmaxexp
  Otherwise .3ds files from TurboSquid can work

osgART Tutorial 5: Adding 3D Model
std::string filename = "media/hollow_cube.osg";
arTransform->addChild(osgDB::readNodeFile(filename));
  Replace the simple cube with a 3D model
  Models are loaded using the osgDB::readNodeFile() function
  Note: Scale is important. Units are in mm.
3D Studio Max
Export to .osg
osgART

osgART Tutorial 6: Multiple Markers
  Repeat the process so far to track more than
one marker simultaneously

osg::ref_ptr<osg::MatrixTransform> arTransformA = new osg::MatrixTransform();
osgART::attachDefaultEventCallbacks(arTransformA.get(), markerA.get());
arTransformA->addChild(osgDB::readNodeFile("media/hitl_logo.osg"));
arTransformA->getOrCreateStateSet()->setRenderBinDetails(100, "RenderBin");
cam->addChild(arTransformA.get());
osg::ref_ptr<osg::MatrixTransform> arTransformB = new osg::MatrixTransform();
osgART::attachDefaultEventCallbacks(arTransformB.get(), markerB.get());
arTransformB->addChild(osgDB::readNodeFile("media/gist_logo.osg"));
arTransformB->getOrCreateStateSet()->setRenderBinDetails(100, "RenderBin");
cam->addChild(arTransformB.get());
  Load and activate two markers
osg::ref_ptr<osgART::Marker> markerA = tracker->addMarker("single;data/patt.hiro;80;0;0");
markerA->setActive(true);
osg::ref_ptr<osgART::Marker> markerB = tracker->addMarker("single;data/patt.kanji;80;0;0");
markerB->setActive(true);
  Create two transformations, attach callbacks, and add models
  Repeat the process so far to track more than one marker

Basic osgART Tutorial: Summary
Standard OSGViewer Addition ofVideo Addition of Tracking
Addition of basic 3D
graphics
Addition of 3D Model Multiple Markers

experiences
applications
tools
components
Sony CSL © 2004
Building Compelling AR Experiences
Tracking, Display
Authoring
Interaction

AR Interaction
  Designing AR System = Interface Design
  Using different input and output technologies
  Objective is a high quality of user experience
  Ease of use and learning
  Performance and satisfaction

User Interface and Tool
  Human  User Interface/Tool  Machine/Object
  Human Machine Interface
© Andreas Dünser
Tools
User
Interface

User Interface: Characteristics
  Input: mono or multimodal
  Output: mono or multisensorial
  Technique/Metaphor/Paradigm
© Andreas Dünser
Input
Output
Sensation of
movement
Metaphor:
“Push” to accelerate
“Turn” to rotate

Human Computer Interface
  Human  User Interface Computer System
  Human Computer Interface=
Hardware +| Software
  Computer is everywhere now HCI electronic
devices, Home Automation, Transport vehicles, etc
© Andreas Dünser

More terminology
  Interaction Device= Input/Output of User
Interface
  Interaction Style= category of similar
interaction techniques
  Interaction Paradigm
  Modality (human sense)
  Usability

Back to AR
  You can see spatially registered AR..
how can you interact with it?

Interaction Tasks
  2D (from [Foley]):
  Selection, Text Entry, Quantify, Position
  3D (from [Bowman]):
  Navigation (Travel/Wayfinding)
  Selection
  Manipulation
  System Control/Data Input
  AR: 2D + 3D Tasks and.. more specific tasks?
[Foley] The Human Factors of Computer Graphics InteractionTechniques Foley, J. D.,V.Wallace & P. Chan. IEEE Computer
Graphics and Applications(Nov.): 13-48. 1984.
[Bowman]: 3D User Interfaces:Theory and Practice D. Bowman, E. Kruijff, J. Laviola, I. Poupyrev Addison Wesley 2005

AR Interfaces as Data Browsers
  2D/3D virtual objects are
registered in 3D
  “VR in Real World”
  Interaction
  2D/3D virtual viewpoint
control
  Applications
  Visualization, training

AR Information Browsers
  Information is registered to
real-world context
  Hand held AR displays
  Interaction
  Manipulation of a window
into information space
  Applications
  Context-aware information displays
Rekimoto, et al. 1997

Current AR Information Browsers
  Mobile AR
  GPS + compass
  Many Applications
  Layar
  Wikitude
  Acrossair
  PressLite
  Yelp
  AR Car Finder
  …

Junaio
  AR Browser from Metaio
  http://www.junaio.com/
  AR browsing
  GPS + compass
  2D/3D object placement
  Photos/live video
  Community viewing

2013 Lecture 5: AR Tools and Interaction

Adding Models in Web Interface

Advantages and Disadvantages
  Important class of AR interfaces
  Wearable computers
  AR simulation, training
  Limited interactivity
  Modification of virtual
content is difficult
Rekimoto, et al. 1997

3D AR Interfaces
  Virtual objects displayed in 3D
physical space and manipulated
  HMDs and 6DOF head-tracking
  6DOF hand trackers for input
  Interaction
  Viewpoint control
  Traditional 3D user interface
interaction: manipulation, selection,
etc.
Kiyokawa, et al. 2000

  Important class of AR interfaces
  Entertainment, design, training
  Advantages
  User can interact with 3D virtual
object everywhere in space
  Natural, familiar interaction
  Disadvantages
  Usually no tactile feedback
  User has to use different devices for
virtual and physical objects
Oshima, et al. 2000

Augmented Surfaces and
Tangible Interfaces
  Basic principles
  Virtual objects are
projected on a surface
  Physical objects are used
as controls for virtual
objects
  Support for collaboration

Augmented Surfaces
  Rekimoto, et al. 1998
  Front projection
  Marker-based tracking
  Multiple projection surfaces

Tangible User Interfaces (Ishii 97)
  Create digital shadows
for physical objects
  Foreground
  graspable UI
  Background
  ambient interfaces

Tangible Interfaces - Ambient
  Dangling String
  Jeremijenko 1995
  Ambient ethernet monitor
  Relies on peripheral cues
  Ambient Fixtures
  Dahley, Wisneski, Ishii 1998
  Use natural material qualities
for information display

Tangible Interface: ARgroove
  Collaborative Instrument
  Exploring Physically Based Interaction
  Map physical actions to Midi output
-  Translation, rotation
-  Tilt, shake

Visual Feedback
  Continuous Visual Feedback is Key
  Single Virtual Image Provides:
  Rotation
  Tilt
  Height

i/O Brush (Ryokai, Marti, Ishii)

Other Examples
  Triangles (Gorbert 1998)
  Triangular based story telling
  ActiveCube (Kitamura 2000-)
  Cubes with sensors

Lessons from Tangible Interfaces
  Physical objects make us smart
  Norman’s “Things that Make Us Smart”
  encode affordances, constraints
  Objects aid collaboration
  establish shared meaning
  Objects increase understanding
  serve as cognitive artifacts

TUI Limitations
  Difficult to change object properties
  can’t tell state of digital data
  Limited display capabilities
  projection screen = 2D
  dependent on physical display surface
  Separation between object and display
  ARgroove

  Advantages
  Natural - users hands are used for interacting
with both virtual and real objects.
-  No need for special purpose input devices
  Interaction is limited only to 2D surface
-  Full 3D interaction and manipulation is difficult

Orthogonal Nature of AR Interfaces

Back to the Real World
  AR overcomes limitation of TUIs
  enhance display possibilities
  merge task/display space
  provide public and private views
  TUI + AR = Tangible AR
  Apply TUI methods to AR interface design

  Space-multiplexed
  Many devices each with one function
-  Quicker to use, more intuitive, clutter
-  Real Toolbox
  Time-multiplexed
  One device with many functions
-  Space efficient
-  mouse

Tangible AR: Tiles (Space Multiplexed)
  Tiles semantics
  data tiles
  operation tiles
  Operation on tiles
  proximity
  spatial arrangements
  space-multiplexed

Space-multiplexed Interface
Data authoring in Tiles

Object Based Interaction: MagicCup
  Intuitive Virtual Object Manipulation
on a Table-Top Workspace
  Time multiplexed
  Multiple Markers
-  Robust Tracking
  Tangible User Interface
-  Intuitive Manipulation
  Stereo Display
-  Good Presence

Our system
  Main table, Menu table, Cup interface

Tangible AR: Time-multiplexed Interaction
  Use of natural physical object manipulations to
control virtual objects
  VOMAR Demo
  Catalog book:
-  Turn over the page
  Paddle operation:
-  Push, shake, incline, hit, scoop

  Advantages
  Natural interaction with virtual and physical tools
-  No need for special purpose input devices
  Spatial interaction with virtual objects
-  3D manipulation with virtual objects anywhere in physical
space
  Requires Head Mounted Display

Wrap-up
  Browsing Interfaces
  simple (conceptually!), unobtrusive
  3D AR Interfaces
  expressive, creative, require attention
  Tangible Interfaces
  Embedded into conventional environments
  Tangible AR
  Combines TUI input + AR display

AR User Interface: Categorization
  Traditional Desktop: keyboard, mouse,
joystick (with or without 2D/3D GUI)
  Specialized/VR Device: 3D VR devices,
specially design device

  Tangible Interface : using physical object Hand/
Touch Interface : using pose and gesture of hand,
fingers
  Body Interface: using movement of body

  Speech Interface: voice, speech control
  Multimodal Interface : Gesture + Speech
  Haptic Interface : haptic feedback
  Eye Tracking, Physiological, Brain Computer
Interface..

Websites
  Software Download
  http://artoolkit.sourceforge.net/
  ARToolKit Documentation
  http://www.hitl.washington.edu/artoolkit/
  ARToolKit Forum
  http://www.hitlabnz.org/wiki/Forum
  ARToolworks Inc
  http://www.artoolworks.com/

  ARToolKit Plus
  http://studierstube.icg.tu-graz.ac.at/handheld_ar/
artoolkitplus.php
  osgART
  http://www.osgart.org/
  FLARToolKit
  http://www.libspark.org/wiki/saqoosha/FLARToolKit/
  FLARManager
  http://words.transmote.com/wp/flarmanager/

Project Assignment
  Design/Related work exercise
  Individual
  Each person find 2 relevant papers/videos/websites
  Write two page literature review
  As a team - prototype design
  Sketch out the user interface of application
  Design the interaction flow/Screen mockups
  3 minute Presentation in class August 16th

2013 Lecture 5: AR Tools and Interaction

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to 2013 Lecture 5: AR Tools and Interaction (20)

More from Mark Billinghurst (20)

Recently uploaded (20)

2013 Lecture 5: AR Tools and Interaction