Motion Segmentation Results
These are examples of motion segmentation using the layered dynamic texture (LDT). We also compare to segmentations using mixtures of dynamic textures (DTM) or generalized PCA (GPCA). The video is in Quicktime format (H.264).
This is a case where the LDT corrects a
noisy DTM segmentation (imprecise boundaries and spurious segments)
This is an
example where the DTM produces a poor segmentation (e.g. the border between two
textures erroneously marked as a segment), which the LDT corrects.
This is an
example where the DTM produces a poor segmentation (e.g. the border between two
textures erroneously marked as a segment), which the LDT corrects.
This is a difficult case. The initial DTM segmentation is very poor. Albeit a
substantial improvement, the LDT segmentation is still noisy.
This is a difficult case. DTM splits
the two water segments incorrectly (the two textures are very similar). The LDT substantially
improves the segmentation, but the difficulties due to the great similarity of water patterns
prove too difficult to overcome completely.
This is an example of temporal segmentation using the TS-LDT on the ocean-fire sequence.
Segmentation of Synthetic Circular Motion
These experiments were based on video containing several rings of distinct circular motion.
LDT correctly segments all the rings, favoring global homogeneity over localized grouping
of segments by texture orientation. On the other
hand, DTM tends to find incorrect segmentations based on local direction of motion.
In addition, DTM sometimes incorrectly assigns one segment to the boundaries between rings,
illustrating how the poor boundary accuracy of the patch-based segmentation framework can
create substantial problems. Finally, GPCA is able to correctly segment 2 rings,
but fails when there are more. In these cases, GPCA correctly segments one of the rings, but
randomly segments the remainder of the video.
circular pinwheel
[mov (1.6 MB)]
[mov (2.8 MB)]
[mov (2.0 MB)]
square pinwheel
[mov (2.3 MB)]
[mov (3.9 MB)]
[mov (3.0 MB)]
square tracks
[mov (2.0 MB)]
[mov (3.9 MB)]
[mov (4.2 MB)]
Segmentation of Synthetic Texture Database
The synthetic texture database contains 299 videos of synthetic textures with 2, 3, or 4 segments.
Qualitatively, LDT improves the DTM segmentation in three ways: 1) segmentation boundaries
are more precise, due to the region-level modeling (rather than patch-level); 2) segmentations
are less noisy, due to the inclusion of the MRF prior; and 3) gross errors, e.g. texture borders
marked as segments, are eliminated.
Full results are available for
synthdb2,
synthdb3, and
synthdb4.
synthdb2: texture_050
[mov (0.4 MB)]
synthdb3: texture_100
[mov (0.3 MB)]
synthdb4: texture_098
[mov (0.2 MB)]
synthdb3: texture_065
[mov (0.9 MB)]
synthdb4: texture_032
[mov (0.7 MB)]
Segmentation with TS-LDT
[mov (1.2 MB)]
These are examples of segmenting real video using the layered dynamic texture.
For K=2, both LDT and DTM segment the static background from the moving ferris wheel.
However, for K=3 regions, the plausible segmentation, by LDT, of the foreground into two
regions corresponding to the ferris wheel and a balloon moving in the wind, is not matched by
DTM. Instead, the latter segments the ferris wheel into two regions, according to the dominant
direction of its local motion (either moving up or down), ignoring the balloon motion.
For K = 3 regions, LDT segments the windmill into regions corresponding
to the moving fan blades, parts of the shaking tail piece, and the background. When segmenting
into K = 4 regions, LDT splits the fan blade segment into two regions, which correspond
to the fan blades and the internal support pieces. On the other hand, the DTM segmentations
for K = {3, 4} split the fan blades into different regions based on the orientation (vertical or
horizontal) of the optical flow.
This is an example showing the interesting property of LDT segmentation:
that it tends to produce a sequence of segmentations which captures a hierarchy of scene
dynamics. The whirlpool sequence contains different levels of moving and
turbulent water. For K = 2 layers, the LDT segments the scene into regions containing moving
water and still background (still water and grass). Adding another layer splits the moving
water segment into two regions of different water dynamics: slowly moving ripples (outside of
the whirlpool) and fast turbulent water (inside the whirlpool). Finally for K = 4 layers, LDT
splits the turbulent water region into two regions: the turbulent center of the whirlpool, and
the fast water spiraling into it.
A crowd moves in a circle around a pillar. The left side of the scene is less
congested, and the crowd moves faster than on the right side.
The crowd moves
with three levels of speed, which are stratified into horizontal layers
A crowd
gathers at the entrance of an escalator, with people moving quickly around the edges. These
segmentations show that LDT can distinguish different speeds of crowd motion, regardless of
the direction in which the crowd is traveling.
The LDT segments a highway scene
into still background, the fast moving traffic on the highway, and the slow traffic that merges into
it.
A whirlpool, where the turbulent water component is segmented
from the remaining moving water.
A windmill scene, which
the LDT segments into regions corresponding to the windmill (circular motion), the trees waving
in the wind, and the static background.
Segmentation of Real Video
ferris wheel
[mov (0.3 MB)]
mill
[mov (0.5 MB)]
whirlpool
[mov (0.8 MB)]
circular crowd (Mecca)
[mov (0.2 MB)]
horizontal crowd (Mecca)
[mov (0.2 MB)]
escalator crowd
[mov (0.3 MB)]
highway onramp
[mov (0.4 MB)]
whirlpool (close)
[mov (0.5 MB)]
windmill
[mov (0.1 MB)]
Copyright Antoni Bert Chan 2008