These are examples of semantic video annotation using DTM models and semantic multi-class labeling (SML) on the DynTex database.
The human (ground-truth) annotations alongside top-5 automated generated annotations are shown with each example.
 |
 |
 |
||
| Gtruth | plant, oscillating | flag, waving | waving | |
| DTM-HEM | oscillating, tree, plant, aquarium, dmotions | flag, waving, turbulent, laundry, river | fountain, anemone, turbulent, oscillating, dmotion | |
| GMM-HEM-DCT | oscillating, plant, tree, anemone, windmill | candle, flag, turbulent, interinsic, laundry | field, waterfall, stream, server, shower | |
| GMM-HEM-OPF | plant, toilet, field, anemone, oscillating | candle, interinsic, turbulent, fountain, windmill | toilet, plant, field, oscillating, anemone | |
 |
 |
 |
||
| Gtruth | tree, oscillating | plant, oscillating | tree, oscillating | |
| DTM-HEM | tree, oscillating, basin, dmotions, fountain | plant, oscillating, tree, waterfall, aquarium | tree, plant, oscillating, aquarium, random | |
| GMM-HEM-DCT | tree, oscillating, windmill, plant, dmotions | oscillating, plant, tree, anemone, foam | tree, oscillating, plant, windmill, waving | |
| GMM-HEM-OPF | basin, dmotion, fountain, stream, river | toilet, plant, field, sky, oscillating | net, pond, foam, source, aquarium | |
 |
 |
 |
||
| Gtruth | tree, oscillating | pond, foam, waving, dmotions | stream, dmotion, turbulent | |
| DTM-HEM | plant, oscillating, tree, aquarium, anemone | pond, foam, fountain, basin, random | stream, foam, dmotions, fountain, waterfall | |
| GMM-HEM-DCT | oscillating, plant, anemone, foam, tree | foam, waving, basin, stream, anemone | stream, fountain, basin, waterfall, dmotion | |
| GMM-HEM-OPF | field, plant, toilet, anemone, oscillating | foam, pond, random, tree, source | waterfall, fountain, stream, dmotion, turbulent | |
 |
 |
 |
||
| Gtruth | boiling, turbulent | plant, oscillating | basin, waterfall, dmotion, turbulent, dmotions | |
| DTM-HEM | boiling, turbulent, laundry, waving, flag | plant, oscillating, tree, aquarium, stream | pond, fountain, basin, waving, dmotions | |
| GMM-HEM-DCT | boiling, laundry, candle, flag, turbulent | waterfall, stream, oscillating, foam, plant | waving, foam, basin, source, random | |
| GMM-HEM-OPF | fountain, turbulent, candle, interinsic, dmotion | toilet, plant, field, oscillating, anemone | basin, random, sea, fountain, stream | |
 |
 |
 |
||
| Gtruth | waterfall, dmotion, dmotions | dmotions | turbulent | |
| DTM-HEM | dmotion, waterfall, dmotions, field, oscillating | waving, fountain, basin, dmotions, dmotion | laundry, toilet, turbulent, boiling, flag | |
| GMM-HEM-DCT | shower, plant, field, server, oscillating | road, turbulent, dmotion, fountain, dmotions | laundry, toilet, candle, source, interinsic | |
| GMM-HEM-OPF | toilet, anemone, oscillating, plant, tree | mobile, aquarium, turbulent, server, interinsic | anemone, net, tree, oscillating, aquarium | |
 |
 |
 |
||
| Gtruth | plant, oscillating | river, rain, dmotion, random | basin, fountain, waving, dmotion, turbulent | |
| DTM-HEM | plant, oscillating, tree, waterfall, basin | rain, random, river, pond, basin | fountain, basin, dmotion, turbulent, waving | |
| GMM-HEM-DCT | plant, tree, oscillating, anemone, interinsic | rain, pond, river, random, foam | turbulent, candle, laundry, sky, flag | |
| GMM-HEM-OPF | toilet, plant, field, oscillating, anemone | rain, pond, random, river, basin | net, waterfall, random, waving, river | |
 |
 |
 |
||
| Gtruth | fountain | candle, turbulent, interinsic | escalator, dmotion | |
| DTM-HEM | fountain, sea, waving, turbulent, laundry | candle, interinsic, dmotion, turbulent, waving | escalator, dmotion, waterfall, turbulent, flag | |
| GMM-HEM-DCT | fountain, anemone, waving, turbulent, dmotion | laundry, toilet, candle, interinsic, source | escalator, waterfall, dmotion, basin, plant | |
| GMM-HEM-OPF | fountain, dmotion, waterfall, laundry, shower | plant, toilet, anemone, oscillating, field | escalator, laundry, shower, waterfall, candle | |
 |
 |
 |
||
| Gtruth | boiling, turbulent | laundry, turbulent | plant, oscillating | |
| DTM-HEM | boiling, laundry, turbulent, flag, waving | laundry, flag, turbulent, waving, boiling | plant, oscillating, tree, aquarium, waterfall | |
| GMM-HEM-DCT | boiling, dmotion, server, laundry, net | laundry, windmill, candle, flag, turbulent | plant, tree, oscillating, anemone, windmill | |
| GMM-HEM-OPF | fountain, waterfall, turbulent, dmotion, interinsic | laundry, flag, waving, windmill, dmotions | plant, toilet, field, anemone, oscillating | |
 |
 |
 |
||
| Gtruth | plant, oscillating | anemone, oscillating | dmotion | |
| DTM-HEM | plant, oscillating, tree, dmotions, basin | anemone, oscillating, aquarium, tree, sky | dmotion, waving, turbulent, basin, flag | |
| GMM-HEM-DCT | tree, oscillating, plant, windmill, dmotions | source, turbulent, fountain, sky, dmotion | server, candle, dmotion, laundry, turbulent | |
| GMM-HEM-OPF | tree, anemone, oscillating, interinsic, field | toilet, plant, field, anemone, sky | anemone, tree, oscillating, plant, server | |
 |
 |
 |
||
| Gtruth | river, rain, dmotion, random | sky, dmotion | flag, waving | |
| DTM-HEM | rain, river, random, basin, fountain | sky, dmotion, fountain, toilet, turbulent | flag, waving, boiling, laundry, turbulent | |
| GMM-HEM-DCT | rain, pond, foam, random, river | sky, flag, boiling, laundry, windmill | flag, candle, laundry, windmill, turbulent | |
| GMM-HEM-OPF | rain, random, source, pond, aquarium | sky, foam, laundry, pond, fountain | flag, boiling, laundry, candle, windmill | |
 |
 |
 |
||
| Gtruth | tree, oscillating | escalator, dmotions | oscillating | |
| DTM-HEM | plant, tree, oscillating, anemone, waterfall | escalator, dmotions, waterfall, dmotion, flag | oscillating, plant, tree, anemone, field | |
| GMM-HEM-DCT | plant, tree, oscillating, anemone, basin | escalator, shower, windmill, dmotions, candle | oscillating, plant, field, sky, anemone | |
| GMM-HEM-OPF | field, plant, anemone, tree, oscillating | escalator, boiling, dmotions, fountain, waving | toilet, plant, field, sky, mobile | |
 |
 |
 |
||
| Gtruth | laundry, turbulent | flag, waving | basin, fountain, waving, dmotion, turbulent | |
| DTM-HEM | laundry, turbulent, boiling, flag, waving | flag, waving, turbulent, laundry, boiling | fountain, dmotion, turbulent, basin, waving | |
| GMM-HEM-DCT | laundry, candle, toilet, interinsic, turbulent | flag, candle, waving, road, windmill | candle, turbulent, laundry, sky, flag | |
| GMM-HEM-OPF | oscillating, interinsic, turbulent, anemone, waterfall | flag, laundry, boiling, windmill, waving | net, foam, random, aquarium, tree | |
 |
 |
 |
||
| Gtruth | stream, foam, dmotion, oscillating | anemone, oscillating | net | |
| DTM-HEM | foam, stream, source, waterfall, dmotions | anemone, boiling, laundry, flag, turbulent | net, basin, turbulent, sea, waving | |
| GMM-HEM-DCT | basin, dmotions, foam, fountain, waterfall | boiling, laundry, anemone, flag, candle | net, dmotion, fountain, turbulent, shower | |
| GMM-HEM-OPF | stream, waterfall, fountain, turbulent, dmotion | waterfall, fountain, dmotion, turbulent, dmotions | plant, toilet, anemone, oscillating, tree | |
 |
 |
 |
||
| Gtruth | fountain, waterfall, source, dmotion, dmotions | sea, dmotion, turbulent | anemone, oscillating | |
| DTM-HEM | source, fountain, dmotions, stream, waterfall | sea, turbulent, dmotion, fountain, waving | anemone, oscillating, aquarium, stream, fountain | |
| GMM-HEM-DCT | source, foam, waving, pond, river | sea, river, waving, fountain, dmotion | field, stream, anemone, server, waterfall | |
| GMM-HEM-OPF | sea, basin, river, stream, waving | rain, source, aquarium, random, field | plant, field, toilet, anemone, oscillating | |
 |
 |
 |
||
| Gtruth | fountain, waving | anemone, oscillating | tree, oscillating | |
| DTM-HEM | foam, stream, dmotions, fountain, windmill | anemone, turbulent, waving, sea, interinsic | anemone, river, oscillating, interinsic, tree | |
| GMM-HEM-DCT | aquarium, basin, dmotions, fountain, waving | anemone, stream, fountain, foam, dmotions | field, laundry, interinsic, sky, server | |
| GMM-HEM-OPF | fountain, basin, waving, dmotion, waterfall | field, plant, anemone, tree, oscillating | field, plant, toilet, anemone, oscillating | |
 |
 |
 |
||
| Gtruth | mobile, dmotion | plant, oscillating | stream, waving | |
| DTM-HEM | mobile, waving, sea, stream, fountain | plant, tree, oscillating, fountain, dmotions | stream, fountain, foam, waterfall, windmill | |
| GMM-HEM-DCT | mobile, source, turbulent, laundry, server | tree, plant, dmotions, oscillating, interinsic | stream, dmotions, fountain, anemone, basin | |
| GMM-HEM-OPF | toilet, plant, field, anemone, oscillating | random, sea, basin, river, aquarium | stream, random, fountain, waterfall, dmotion | |
 |
 |
 |
||
| Gtruth | tree, oscillating | anemone, oscillating | river | |
| DTM-HEM | plant, oscillating, tree, field, anemone | anemone, oscillating, interinsic, aquarium, fountain | river, sea, waving, dmotion, fountain | |
| GMM-HEM-DCT | plant, tree, oscillating, anemone, interinsic | source, foam, anemone, basin, turbulent | river, sea, waving, mobile, fountain | |
| GMM-HEM-OPF | toilet, plant, sky, field, oscillating | toilet, plant, field, anemone, oscillating | boiling, fountain, river, waving, waterfall | |
 |
 |
 |
||
| Gtruth | basin, waterfall, waving, dmotion, random | fountain, pond, dmotion, turbulent | shower, waving | |
| DTM-HEM | basin, random, waterfall, waving, pond | pond, random, river, foam, fountain | shower, toilet, boiling, laundry, turbulent | |
| GMM-HEM-DCT | random, road, waving, river, basin | foam, stream, source, fountain, basin | shower, net, dmotion, foam, fountain | |
| GMM-HEM-OPF | random, river, basin, sea, stream | random, aquarium, field, tree, anemone | random, shower, basin, source, sea | |
 |
 |
 |
||
| Gtruth | tree, oscillating | boiling, turbulent | fountain | |
| DTM-HEM | plant, tree, oscillating, anemone, random | boiling, laundry, turbulent, toilet, flag | fountain, turbulent, laundry, waving, dmotion | |
| GMM-HEM-DCT | interinsic, source, turbulent, sky, oscillating | boiling, laundry, candle, flag, windmill | anemone, fountain, turbulent, interinsic, basin | |
| GMM-HEM-OPF | plant, field, toilet, anemone, oscillating | boiling, fountain, candle, turbulent, flag | shower, fountain, basin, random, dmotion | |
 |
 |
 |
||
| Gtruth | fountain, dmotion, turbulent | dmotions | source, turbulent | |
| DTM-HEM | turbulent, fountain, dmotion, waving, basin | waving, fountain, basin, dmotions, dmotion | anemone, source, foam, pond, turbulent | |
| GMM-HEM-DCT | oscillating, turbulent, interinsic, basin, fountain | dmotions, dmotion, fountain, waving, basin | toilet, laundry, mobile, source, candle | |
| GMM-HEM-OPF | tree, field, plant, anemone, net | mobile, toilet, aquarium, anemone, oscillating | plant, field, toilet, anemone, oscillating | |
 |
 |
 |
||
| Gtruth | fountain, dmotion, turbulent | stream, shower, dmotion | candle, waving | |
| DTM-HEM | toilet, fountain, dmotion, candle, interinsic | shower, stream, dmotion, boiling, laundry | candle, turbulent, waving, interinsic, laundry | |
| GMM-HEM-DCT | candle, interinsic, sky, turbulent, windmill | shower, field, plant, windmill, stream | dmotion, server, oscillating, fountain, turbulent | |
| GMM-HEM-OPF | candle, interinsic, laundry, fountain, windmill | waterfall, fountain, dmotions, dmotion, turbulent | toilet, plant, anemone, oscillating, field | |
 |
 |
 |
||
| Gtruth | fountain | plant, oscillating | plant, oscillating | |
| DTM-HEM | fountain, waving, turbulent, dmotion, laundry | oscillating, plant, tree, anemone, aquarium | plant, tree, oscillating, dmotions, fountain | |
| GMM-HEM-DCT | anemone, fountain, interinsic, turbulent, basin | oscillating, stream, waterfall, foam, field | tree, basin, oscillating, dmotions, plant | |
| GMM-HEM-OPF | fountain, basin, shower, waterfall, dmotion | toilet, plant, sky, field, mobile | anemone, toilet, oscillating, waterfall, interinsic | |
 |
 |
 |
||
| Gtruth | turbulent | sea, dmotion, turbulent | field, oscillating | |
| DTM-HEM | flag, boiling, turbulent, laundry, waving | sea, dmotion, river, turbulent, waving | field, anemone, oscillating, waterfall, laundry | |
| GMM-HEM-DCT | candle, laundry, interinsic, turbulent, windmill | sea, fountain, mobile, turbulent, dmotion | field, laundry, sky, shower, server | |
| GMM-HEM-OPF | interinsic, turbulent, anemone, oscillating, server | waterfall, foam, net, random, dmotion | plant, field, toilet, anemone, oscillating | |
 |
 |
 |
||
| Gtruth | tree, oscillating | sea, waving | plant, oscillating | |
| DTM-HEM | tree, oscillating, anemone, fountain, basin | sea, river, dmotion, waving, turbulent | plant, oscillating, tree, aquarium, waterfall | |
| GMM-HEM-DCT | plant, tree, oscillating, anemone, interinsic | sea, river, waving, mobile, road | oscillating, plant, tree, anemone, foam | |
| GMM-HEM-OPF | tree, anemone, source, oscillating, field | shower, basin, dmotions, fountain, dmotion | toilet, plant, field, sky, mobile | |
 |
 |
 |
||
| Gtruth | dmotions | sea, dmotion, turbulent | stream, waving, dmotion | |
| DTM-HEM | laundry, turbulent, flag, boiling, waving | sea, dmotion, turbulent, waving, fountain | stream, waving, sea, river, basin | |
| GMM-HEM-DCT | candle, interinsic, turbulent, laundry, source | sea, river, fountain, waving, dmotion | river, sea, road, waving, random | |
| GMM-HEM-OPF | oscillating, anemone, turbulent, stream, tree | net, foam, pond, aquarium, river | net, river, sea, stream, waving | |
 |
 |
 |
||
| Gtruth | fountain, pond, dmotion, turbulent | anemone, interinsic | stream, dmotion, turbulent | |
| DTM-HEM | pond, waterfall, basin, random, foam | anemone, oscillating, tree, interinsic, plant | stream, fountain, foam, windmill, waving | |
| GMM-HEM-DCT | pond, foam, waving, river, stream | source, server, turbulent, oscillating, dmotion | stream, anemone, aquarium, dmotions, fountain | |
| GMM-HEM-OPF | sea, basin, shower, source, random | toilet, plant, field, sky, oscillating | shower, fountain, basin, waterfall, random | |
 |
 |
 |
||
| Gtruth | fountain | tree, oscillating | laundry, turbulent | |
| DTM-HEM | fountain, basin, waving, river, pond | plant, oscillating, tree, waterfall, field | laundry, flag, turbulent, waving, boiling | |
| GMM-HEM-DCT | sea, river, waving, road, random | plant, tree, oscillating, windmill, anemone | windmill, turbulent, aquarium, server, laundry | |
| GMM-HEM-OPF | waving, boiling, flag, river, fountain | toilet, plant, field, sky, mobile | laundry, turbulent, candle, windmill, dmotions | |
 |
 |
 |
||
| Gtruth | basin, fountain, waving, turbulent, dmotions | tree, oscillating | fountain, dmotion | |
| DTM-HEM | basin, fountain, dmotions, random, source | plant, oscillating, tree, aquarium, waterfall | anemone, fountain, basin, oscillating, interinsic | |
| GMM-HEM-DCT | waving, foam, basin, pond, dmotions | oscillating, tree, plant, windmill, waving | field, server, oscillating, stream, shower | |
| GMM-HEM-OPF | aquarium, random, source, stream, tree | toilet, plant, field, sky, oscillating | toilet, plant, field, anemone, oscillating | |
 |
 |
 |
||
| Gtruth | waterfall, dmotion, turbulent | river, waving | fountain, dmotion, dmotions | |
| DTM-HEM | waterfall, dmotion, turbulent, dmotions, fountain | river, waving, flag, windmill, sea | dmotions, fountain, dmotion, waterfall, flag | |
| GMM-HEM-DCT | waterfall, dmotion, basin, oscillating, dmotions | sea, waving, river, road, flag | candle, dmotions, waterfall, aquarium, basin | |
| GMM-HEM-OPF | waterfall, fountain, turbulent, interinsic, dmotion | laundry, turbulent, candle, interinsic, waving | fountain, shower, dmotions, waterfall, dmotion | |
 |
 |
 |
||
| Gtruth | stream, dmotion, turbulent | sea, dmotion, turbulent | boiling, turbulent | |
| DTM-HEM | anemone, stream, fountain, foam, aquarium | sea, turbulent, waving, dmotion, pond | boiling, turbulent, laundry, flag, waving | |
| GMM-HEM-DCT | stream, waterfall, dmotion, oscillating, foam | sea, river, waving, fountain, dmotion | boiling, candle, interinsic, turbulent, flag | |
| GMM-HEM-OPF | toilet, plant, field, sky, oscillating | rain, river, sea, basin, random | candle, laundry, turbulent, interinsic, flag | |