426 lecture6a osgART Development

COSC 426 Lecture 6a on using osgART

1. COSC 426: Augmented Reality
Mark Billinghurst
[email protected]
August 14
th
2012
Lecture 6: osgART

2. osgART
Developing Augmented Reality
Applications with osgART

3. What is a Scene Graph?
 Tree-like structure for organising a virtual world
 e.g. VRML
 Hierarchy of nodes that define:
 Groups (and Switches, Sequences etc…)
 Transformations
 Projections
 Geometry
 …
 And states and attributes that define:
 Materials and textures
 Lighting and blending
 …

4. Scene Graph Example

5. Benefits of a Scene Graph
 Performance
 Structuring data facilitates optimization
- Culling, state management, etc…
 Abstraction
 Underlying graphics pipeline is hidden
 Low-level programming (“how do I display
this?”) replaced with high-level concepts
(“what do I want to display?”)
Image: sgi

6. About Open Scene Graph
 http://www.openscenegraph.org/
 Open-source scene graph implementation
 Based on OpenGL
 Object-oriented C++ following design pattern principles
 Used for simulation, games, research, and industrial projects
 Active development community
 Maintained by Robert Osfield
 ~2000 mailing list subscribers
 Documentation project: www.osgbooks.com
 Uses the OSG Public License (similar to LGPL)

7. About Open Scene Graph (2)
Pirates of the XXI Century Flightgear
3DVRII Research Institute EOR
SCANeR
VRlab Umeå University

8. Open Scene Graph Features
 Plugins for loading and saving
 3D: 3D Studio (.3ds), OpenFlight (.flt), Wavefront (.obj)…
 2D: .png, .jpg, .bmp, QuickTime movies
 NodeKits to extend functionality
 e.g. osgShadow
 Cross-platform support for:
 Window management (osgViewer)
 Threading (OpenThreads)

9. Open Scene Graph Architecture
Plugins read and
write 2D image and
3D model files
NodeKits extend
core functionality,
exposing higher-level
node types
Scene graph and
rendering
functionality
Inter-operability with
other environments,
e.g. Python

10. Some Open Scene Graph Demos
 You may want to get the OSG data package:
 Via SVN: http://www.openscenegraph.org/svn/osg/OpenSceneGraph-Data/trunk
osgviewer osgmotionblur osgparticle
osgreflect osgdistortion osgfxbrowser

11. Learning OSG
 Check out the Quick Start Guide
 Free PDF download at http://osgbooks.com/, Physical copy $13US
 Join the mailing list: http://www.openscenegraph.org/projects/osg/wiki/
MailingLists
 Browse the website: http://www.openscenegraph.org/projects/osg
 Use the forum: http://forum.openscenegraph.org
 Study the examples
 Read the source? 

12. osgART

13. What is osgART?
 osgART adds AR to Open Scene Graph
 Further developed and enhanced by:
 Julian Looser
 Hartmut Seichter
 Raphael Grasset
 Current version 2.0, Open Source
 http://www.osgart.org

14. osgART Approach: Basic Scene Graph
Root
Transform
3D Object
0.988 -0.031 -0.145 0
-0.048 0.857 -0.512 0
0.141 0.513 0.846 0
10.939 29.859 -226.733 1 [ ]
 To add Video see-through AR:
 Integrate live video
 Apply correct projection matrix
 Update tracked transformations in
realtime

15. osgART Approach: AR Scene Graph
Root
Transform
3D Object

16. osgART Approach: AR Scene Graph
Video
Geode
Root
Transform
3D Object
Virtual
Camera
Video
Layer

17. osgART Approach: AR Scene Graph
Video
Geode
Root
Transform
3D Object
Virtual
Camera
Projection matrix from
tracker calibration
Transformation matrix
updated from marker
tracking in realtime
Video
Layer
Full-screen quad
with live texture
updated from
Video source
Orthographic
projection

18. osgART Approach: AR Scene Graph
Video
Geode
Root
Transform
3D Object
Virtual
Camera
Projection matrix from
tracker calibration
Transformation matrix
updated from marker
tracking in realtime
Video
Layer
Full-screen quad
with live texture
updated from
Video source
Orthographic
projection

19. osgART Architecture
 Like any video see-through AR library, osgART requires video
input and tracking capabilities.
AR Library Application
Video Source
e.g. DirectShow
Tracking Module
(libAR.lib)

20. osgART Architecture
 osgART uses a plugin architecture so that video sources and tracking
technologies can be plugged in as necessary
osgART
Application
Video Plugin
Tracker Plugin
ARToolKit4 -
ARToolkitPlus -
MXRToolKit -
ARLib -
bazAR (work in progress) -
ARTag (work in progress) -
OpenCVVideo -
VidCapture -
CMU1394 -
PointGrey SDK -
VidereDesign -
VideoWrapper -
VideoInput -
VideoSource -
DSVL -
Intranel RTSP -

21. Basic osgART Tutorial
 Develop a working osgART application from scratch.
 Use ARToolKit 2.72 library for tracking and video
capture

22. osgART Tutorial 1: Basic OSG Viewer
 Start with the standard Open Scene Graph Viewer
 We will modify this to do AR!

23. osgART Tutorial 1: Basic OSG Viewer
 The basic osgViewer…
#include <osgViewer/Viewer>
#include <osgViewer/ViewerEventHandlers>
int main(int argc, char* argv[]) {
// Create a viewer
osgViewer::Viewer viewer;
// Create a root node
osg::ref_ptr<osg::Group> root = new osg::Group;
// Attach root node to the viewer
viewer.setSceneData(root.get());
// Add relevant event handlers to the viewer
viewer.addEventHandler( new osgViewer::StatsHandler);
viewer.addEventHandler( new osgViewer::WindowSizeHandler);
viewer.addEventHandler( new osgViewer::ThreadingHandler);
viewer.addEventHandler( new osgViewer::HelpHandler);
// Run the viewer and exit the program when the viewer is closed
return viewer.run();
}

24. osgART Tutorial 2: Adding Video
 Add a video plugin
 Load, configure, start video capture…
 Add a video background
 Create, link to video, add to scene-graph

25. osgART Tutorial 2: Adding Video
 New code to load and configure a Video Plugin:
// Preload the video and tracker
int _video_id = osgART::PluginManager::getInstance()-
>load("osgart_video_artoolkit2");
// Load a video plugin.
osg::ref_ptr<osgART::Video> video =
dynamic_cast<osgART::Video*>(osgART::PluginManager::getInstance()-
>get(_video_id));
// Check if an instance of the video stream could be created
if (!video.valid()) {
// Without video an AR application can not work. Quit if none found.
osg::notify(osg::FATAL) << "Could not initialize video plugin!" <<
std::endl;
exit(-1);
}
// Open the video. This will not yet start the video stream but will
// get information about the format of the video which is essential
// for the connected tracker.
video->open();

26. osgART Tutorial 2: Adding Video
 New code to add a live video background
osg::ref_ptr<osg::Group> videoBackground = createImageBackground(video.get());
videoBackground->getOrCreateStateSet()->setRenderBinDetails(0, "RenderBin");
root->addChild(videoBackground.get());
video->start();
osg::Group* createImageBackground(osg::Image* video) {
osgART::VideoLayer* _layer = new osgART::VideoLayer();
_layer->setSize(*video);
osgART::VideoGeode* _geode = new osgART::VideoGeode(osgART::VideoGeode::USE_TEXTURE_2D, video);
addTexturedQuad(*_geode, video->s(), video->t());
_layer->addChild(_geode);
return _layer;
}
 In the main function…

27. osgART Tutorial 3: Tracking
 Add a Tracker plugin
 Load, configure, link to video
 Add a Marker to track
 Load, activate
 Tracked node
 Create, link with marker via tracking callbacks
 Print out the tracking data

28. osgART Tutorial 3: Tracking
int _tracker_id = osgART::PluginManager::getInstance()->load("osgart_tracker_artoolkit2");
osg::ref_ptr<osgART::Tracker> tracker =
dynamic_cast<osgART::Tracker*>(osgART::PluginManager::getInstance()->get(_tracker_id));
if (!tracker.valid()) {
// Without tracker an AR application can not work. Quit if none found.
osg::notify(osg::FATAL) << "Could not initialize tracker plugin!" << std::endl;
exit(-1);
}
// get the tracker calibration object
osg::ref_ptr<osgART::Calibration> calibration = tracker->getOrCreateCalibration();
// load a calibration file
if (!calibration->load("data/camera_para.dat"))
{
// the calibration file was non-existing or couldnt be loaded
osg::notify(osg::FATAL) << "Non existing or incompatible calibration file" << std::endl;
exit(-1);
}
// set the image source for the tracker
tracker->setImage(video.get());
osgART::TrackerCallback::addOrSet(root.get(), tracker.get());
// create the virtual camera and add it to the scene
osg::ref_ptr<osg::Camera> cam = calibration->createCamera();
root->addChild(cam.get());
 Load a tracking plugin and associate it with the video plugin

29. osgART Tutorial 3: Tracking
osg::ref_ptr<osgART::Marker> marker = tracker->addMarker("single;data/patt.hiro;80;0;0");
if (!marker.valid())
{
// Without marker an AR application can not work. Quit if none found.
osg::notify(osg::FATAL) << "Could not add marker!" << std::endl;
exit(-1);
}
marker->setActive(true);
osg::ref_ptr<osg::MatrixTransform> arTransform = new osg::MatrixTransform();
osgART::attachDefaultEventCallbacks(arTransform.get(), marker.get());
cam->addChild(arTransform.get());
 Load a marker and activate it
 Associate it with a transformation node (via event callbacks)
 Add the transformation node to the virtual camera node
osgART::addEventCallback(arTransform.get(), new osgART::MarkerDebugCallback(marker.get()));
 Add a debug callback to print out information about the tracked marker

30. osgART Tutorial 3: Tracking
 Tracking information is
output to console

31. osgART Tutorial 4: Adding Content
 Now put the tracking data to use!
 Add content to the tracked transform
 Basic cube code
arTransform->addChild(osgART::testCube());
arTransform->getOrCreateStateSet()->setRenderBinDetails(100, "RenderBin");

32. osgART Tutorial 5: Adding 3D Model
 Open Scene Graph can load some 3D formats directly:
 e.g. Wavefront (.obj), OpenFlight (.flt), 3D Studio (.3ds), COLLADA
 Others need to be converted
 Support for some formats is much better than others
 e.g. OpenFlight good, 3ds hit and miss.
 Recommend native .osg and .ive formats
 .osg – ASCII representation of scene graph
 .ive – Binary OSG file. Can contain hold textures.
 osgExp : Exporter for 3DS Max is a good choice
 http://sourceforge.net/projects/osgmaxexp
 Otherwise .3ds files from TurboSquid can work

33. osgART Tutorial 5: Adding 3D Model
std::string filename = "media/hollow_cube.osg";
arTransform->addChild(osgDB::readNodeFile(filename));
 Replace the simple cube with a 3D model
 Models are loaded using the osgDB::readNodeFile() function
 Note: Scale is important. Units are in mm.
3D Studio Max
Export to .osg
osgART

34. osgART Tutorial 6: Multiple Markers
 Repeat the process so far to track more than
one marker simultaneously

35. osgART Tutorial 6: Multiple Markers
osg::ref_ptr<osg::MatrixTransform> arTransformA = new osg::MatrixTransform();
osgART::attachDefaultEventCallbacks(arTransformA.get(), markerA.get());
arTransformA->addChild(osgDB::readNodeFile("media/hitl_logo.osg"));
arTransformA->getOrCreateStateSet()->setRenderBinDetails(100, "RenderBin");
cam->addChild(arTransformA.get());
osg::ref_ptr<osg::MatrixTransform> arTransformB = new osg::MatrixTransform();
osgART::attachDefaultEventCallbacks(arTransformB.get(), markerB.get());
arTransformB->addChild(osgDB::readNodeFile("media/gist_logo.osg"));
arTransformB->getOrCreateStateSet()->setRenderBinDetails(100, "RenderBin");
cam->addChild(arTransformB.get());
 Load and activate two markers
osg::ref_ptr<osgART::Marker> markerA = tracker->addMarker("single;data/patt.hiro;80;0;0");
markerA->setActive(true);
osg::ref_ptr<osgART::Marker> markerB = tracker->addMarker("single;data/patt.kanji;80;0;0");
markerB->setActive(true);
 Create two transformations, attach callbacks, and add models
 Repeat the process so far to track more than one marker

36. osgART Tutorial 6: Multiple Markers

37. Basic osgART Tutorial: Summary
Standard OSG Viewer Addition of Video Addition of Tracking
Addition of basic 3D
graphics
Addition of 3D Model Multiple Markers

38. FLARManager
http://transmote.com/flar

39. FLARManager:
 Makes building FLARToolkit apps easier
 Is open-source, with a free and commercial license
 Is designed to allow exploration of both augmented
reality and alternative controllers
 Decouples FLARToolkit from Papervision3D
 Configuration without recompilation, via xml config

40. FLARManager: features
 Gives more control over application environment
 Provides multiple input options
 Robust multiple marker management
 Supports multiple 3D frameworks
 Offers features for optimization
 Allows for customization

41. Resources

42. Websites
 Software Download
 http://artoolkit.sourceforge.net/
 ARToolKit Documentation
 http://www.hitl.washington.edu/artoolkit/
 ARToolKit Forum
 https://www.artoolworks.com/community/forum/
 ARToolworks Inc
 http://www.artoolworks.com/

43.  ARToolKit Plus
 http://studierstube.icg.tu-graz.ac.at/handheld_ar/
artoolkitplus.php
 osgART
 http://www.osgart.org/
 FLARToolKit
 http://www.libspark.org/wiki/saqoosha/FLARToolKit/
 FLARManager
 http://words.transmote.com/wp/flarmanager/

44. Books
 Interactive Environments with Open-Source
Software: 3D Walkthroughs and Augmented
Reality for Architects with Blender 2.43, DART
3.0 and ARToolKit 2.72 by Wolfgang Höhl
 A Hitchhikers Guide to Virtual Reality by Karen
McMenemy and Stuart Ferguson

45. More Information
• Mark Billinghurst!
– [email protected]!
• Websites!
– www.hitlabnz.org!