Back in November 2008, a group of Japanese coders, working largely under the radar, unveiled a project that redefined many ActionScript developers’ ideas of what the language could do. FLARToolkit, developed primarily by Tomohiko Koyama (aka Saqoosha), introduced augmented reality to the web, and to a large segment of the population as a whole.

FLARToolkit is the latest in a series of ports of ARToolkit, an augmented reality C++ library originally developed by Dr. Hirokazu Kato at the Human Interface Technology Lab at University of Washington. With the advent of ActionScript 3.0, developers like Mario Klingemann and others began experimenting with realtime image analysis techniques for Flash Player. Saqoosha picked up on this, and ported FLARToolkit from NYARToolkit, a Java/C#/Android port of ARToolkit.

FLARToolkit made its biggest initial splash at the hands of North Kingdom, the Swedish interactive agency that developed GE’s SmartGrid augmented reality campaign. Since then, a host of AR applications have made their way to the web via FLARToolkit; the majority of them are variations on the theme of 3D characters dancing on top of live video, or games. As time goes on, however, creative developers will imagine new, creative, and useful applications of the technology. With this in mind, I developed FLARManager.

FLARManager

FLARManager is a lightweight framework that makes building FLARToolkit augmented reality applications easier. When I first came across FLARToolkit, I was simultaneously excited and stymied — there seemed to be so much potential, but it was difficult to use and even harder to understand. Augmented reality is still a long way from maturity, and leveraging FLARToolkit on the ubiquity of Flash Player provides a great opportunity for further exploring its possibilities. FLARManager aims to provide quicker entry to the technology, to allow a focus on new forms of interaction via marker detection and augmented reality.

FLARManager comes bundled with a series of examples and tutorials. Documentation and walkthroughs can be found in the Inside FLARManager section of my blog. The tutorials move from a quickstart through simple 2D and 3D implementations of FLARManager. This article will provide a more detailed look into the process of marker detection and recognition in FLARToolkit, and how FLARManager can improve the accuracy and reliability of those processes via simple configuration changes.

Optimizing FLARToolkit applications with FLARManager

Computer vision in a web context has many inherent difficulties. The main problems arise from a lack of control over the end users’ environmental conditions. Poor or uneven illumination makes software analysis of an image much trickier, and these problems affect FLARToolkit substantially. The last couple of FLARManager releases have focused on functionality that gives developers more control over marker detection and tracking by attempting to mitigate the effects of suboptimal conditions.

Before wading into FLARManager’s settings, it makes sense to begin with a high-level analysis of the process FLARToolkit uses to identify and track markers. The following is my best attempt at understanding the process. Fortunately, FLARToolkit is open source; unfortunately, the minimal comments are all in Japanese (with the exception of those that I added and tagged with ‘SOC:’), and most of the errors the toolkit throws have no information encapsulated within them. If any readers have more insight into the process, please leave comments on this article.

The FLARToolkit process

[thresholding]

The first step in many computer vision applications that rely on edge detection is to threshold the source image. A binary image is made by changing pixels brighter than a threshold value to one color, and pixels darker than the threshold to another.

Thresholding separates the source image into a binary image, making analysis less computationally expensive.

[labeling]

FLARToolkit’s next step is to find contiguous areas in the thresholded image, speficially within the areas below the threshold (darker areas). Using BitmapData.getColorBoundsRect and BitmapData.floodFill, contiguous areas are ‘labeled’ with unique colors, used later to id the areas.

Each contiguous area of white (corresponding to dark areas of the source image) is 'labeled' with a different color.

[marker outline detection]

With candidates for marker locations, FLARToolkit then proceeds to search the labeled areas for shapes that could be transformed squares (i.e. marker outlines).

[pattern matching]

Once all marker outline possibilities have been established, FLARToolkit analyzes the areas of the image within the outlines and compares the contents with the list of patterns the developer has asked FLARTookit to detect. FLARToolkit assigns a ‘confidence’ value to all of the matches; matches that are at or above the confidence level specified by the developer are reported as pattern matches.

How does FLARManager fit in?

One of the central design philosophies of FLARManager is to avoid modification to the FLARToolkit source as much as possible. By remaining loosely coupled with FLARToolkit, each project can continue development separately, and FLARManager can theoretically be implemented for other Flash marker detection engines that arrive in the future.

To this end, the opportunities for FLARManager to improve on FLARToolkit’s process are limited. However, FLARToolkit has free access to both the source image that is passed into FLARToolkit, and to the results that FLARToolkit delivers after analysis. These are the areas on which FLARManager focuses.

All of the settings discussed below can be specified either as member variables of a FLARManager instance, or as attributes of the <flarManagerSettings> node in FLARManager’s external xml configuration file. For more information about the config file, please view the documentation.

[thresholding]

FLARToolkit uses static thresholding: the threshold value does not change from frame to frame. This causes problems when lighting is variable or dim; these are both often the case for web end-users. Based on some ideas described by Makc and Saqoosha, FLARManager now uses ‘adaptive’ or ‘automatic’ thresholding.

Adaptive thresholding is enabled by default in FLARManager. To toggle it, set adaptiveThresholding to either true or false. When adaptive thresholding is active, the threshold value used by FLARToolkit will be changed semi-randomly whenever no markers are detected. There are two values that can be adjusted to tweak the threshold changes.

The first is adaptiveThresholdingSpeed. This value determines the speed at which the threshold is adjusted every frame. Once markers become absent from the source image, the threshold begins to accelerate away from its current value. If adaptiveThresholdingSpeed is set to a low value (0.0 – 0.3), more gradual changes in the threshold result in an initial lag in marker detection, but a better retention of markers once detected. If adaptiveThresholdingSpeed is set to a higher value (> 0.5), an optimal threshold value may be reached more quickly, but the next threshold change may overshoot the next optimal value. This appears as quicker, but more unstable, marker detection.

The second value is adaptiveThresholdingBias. Rather than completely randomly selecting new threshold values, FLARManager applies this bias to the randomized values, to push the overall direction of change toward lower or higher threshold values. Values for adaptiveThresholdingBias below zero will cause the threshold to trend toward 0, while values above zero will push threshold toward 255. Lower thresholds favor darker environments; higher thresholds work best in brighter environments. adaptiveThresholdingBias defaults to -0.1, as darker environments tend to be more common for viewing web applications than oversaturated environments.

To see the results of your tinkering with adaptive thresholding, set FLARManager.thresholdSourceDisplay to true (note: this cannot be set via the xml config file, it must be set in ActionScript). This will overlay the thresholded image on the source image.

These images show the effects of changing thresholding. The image at far left has a threshold of 120, and the image at far right has a threshold of 0. Optimal threshold values vary with lighting conditions; adaptiveThresholding is designed to automatically seek out the optimal threshold value for every FLARManager application.

[labeling]

The labeling process can be very CPU-intensive, as it has to scour the entire thresholded image for dark areas. In fact, FLARLabeling_BitmapData.labeling is the single most CPU-intensive method in FLARToolkit, even when no markers are detected. This is particularly an issue in darker environments. One technique to mitigate this, suggested by Jeffery Bennett, is to slightly blur the source image before thresholding. This tends to increase the size and decrease the number of contiguous areas to label, resulting in higher performance.

This is where sampleBlurring comes into play. This value determines the amount of blur applied to the source image. Specifically, the BlurFilter uses 2 ^ (sampleBlurring-1) for its blurX and blurY values. (Powers of 2 perform much better than other values for BlurFilters.) Higher values result in faster performance, but because the image is more blurred, also result in worse marker detection. sampleBlurring defaults to 1. Setting to zero will turn this feature off entirely.

These images show the effects of sampleBlurring on marker detection. The image at far left has a sampleBlurring of 0; the image at far right has a sampleBlurring of 4. Notice that the total number of contiguous black areas decreases as we move to the right, but the legibility of the marker's pattern also decreases.

[changing sampleBlurring]

These images show the effects of sampleBlurring on marker detection. The image at far left has a sampleBlurring of 0; the image at far right has a sampleBlurring of 4. Notice that the total number of contiguous black areas decreases as we move to the right, but the legibility of the marker’s pattern also decreases.

[marker outline detection]

Due primarily to inaccurate camera calibration, FLARToolkit has a lot of variability in its reporting of marker outline locations. This seems to be unavoidable, given that the developer has no control over the end user’s camera model, or even settings (e.g. focus and zoom). One remaining option for correcting the resulting jitter is to smooth the results.

FLARManager applies smoothing by averaging detected markers’ positions across a number of frames. This is smoothing. Higher values average across a greater number of frames, resulting in smoother motion and less jitter, but also in less responsiveness. smoothing defaults to 3 frames, which is just enough to remove jitter, but not so much that responsiveness of moving markers is significantly diminished.

FLARManager also provides an interface that allows developers to write and plug in their own smoothing algorithms. Custom smoothing algorithms can be written up as a class that implements the IFLARMatrixSmoother interface, and applied in a given FLARManager application by setting FLARManager.smoother to an instance of that class (this cannot be done via the config xml file; it must be set in ActionScript).

[pattern matching]

As described in the [labeling] section above, sampleBlurring has an impact on pattern matching: higher values result in faster FLARToolkit performance, but also in poorer marker detection.

[marker retention]

Maintaining reliable tracking of a marker once it’s first acquired is not a part of FLARToolkit’s core process, but it’s worth mentioning here that FLARManager has a feature designed to address this. Developers can specify a delay between the time that FLARToolkit loses a marker and the time that FLARManager dispatches a MARKER_REMOVED event, by using markerRemovalDelay. If a marker with the same pattern reappears near the same location it disappeared from, within markerRemovalDelay frames, the MARKER_REMOVED event wil not be dispatched.

Unfortunately, this is an imperfect workaround for an imperfect system. Increasing markerRemovalDelay pretty quickly results in a breakdown of the illusion of augmented reality, as the objects drawn at the marker’s location get left behind when the marker moves quickly. As such, I don’t recommend setting markerRemovalDelay much higher than the default of 1 frame, but some applications may be able to make more use of this than others.

Future development

There are a number of improvements still to be done on these features, as well as a slate of additional features to come. One of the trickiest things about working with computer vision is bridging the differences between tracked objects as the computer sees them and as the developer would like to see them. This is FLARManager’s primary role: to round out results generated by FLARToolkit, and provide simple and flexible access to them.

Every new feature that makes its way into FLARManager will continue to be adjustable via the external configuration file and in ActionScript. Since end-user environments in webcam applications vary so much, a core design philosophy of FLARManager is to remain as customizable as possible.