The Role of Visual Rhetoric in Semantic Multimedia: Strategies for Decision Making in Times of Crisis


  • Adrian M.P. Brasoveanu MODUL University Vienna, Department of New Media Technology, Austria, 1 Am Kahlenberg, 1190 Vienna,
  • Ioan Dzitac 1. Aurel Vlaicu University of Arad, Romania, 310330 Arad, 2 Elena Dragoi, and 2. Agora University Romania, 410526 Oradea, 8 Piata Tineretului


semantic multimedia, visual rhetoric, text to visual matching, interactive documentary, crisis strategies, multimedia storytelling


As semantic multimedia is approaching mainstream, even the great improvements that can be seen in its classic schools, like the data mining inspired Information Retrieval based on metadata analysis, or Computer Vision, might not be enough. We identify a new group that gains traction in the semantic multimedia community and which uses as starting point developments from psychology and visual communication. For the purposes of this article we restrict our domain to visual rhetoric as we consider it to yield the biggest potential for future developments. Living in times when the periods between crises seem to be shorter and shorter, we look at how developments in semantic multimedia can be used for predicting and overcoming crises. We analyze at least 2 aspects related to this: using information visualization to understand the evolution of crises and creating multi-layered semantic multimedia technologies that can easily be adapted to use a variety of sources and solve problems from different domains. In both cases we show how techniques inspired by visual rhetoric (information linking, framing, composition) in conjunction with named entity recognition offer a lot of benefits. The section related to multi-layered semantic multimedia technologies also draws on the lessons learned while designing a prototype application aimed at improving tourism decision making process. The article ends with a discussion on evaluation methods for multi-layered semantic technologies applications. We look at how to evaluate them on both levels: mechanisms (information linking versus raw named entity recognition when generating visuals, for example), and decision making strategies (Do such systems actually solve real problems related to crises, create jobs or at least can they be repurposed to solve other problems than the one with which we have started?).


A. Ahmed, L.P. Coelho, A. Arnold, J. Kangas, A.B. Sheikh, E. Xing, W. Cohen, R.F. Murphy, Structured literature image finder: Parsing text and figures in biomedical literature, Journal of Web Semantics: Science, Services and Agents on the World Wide Web, 8 (2-3), 151-154, 2010

R. Arnheim, Art and Visual Perception: A Psychology of The Creative Eye, 2nd Edition, Cambridge, 2004.

R. Barthes, Image, Music, Text, Hill and Wang, 1977.

J. Bertin, Semiology of Graphics: Diagrams, Networks, Graphs, 2nd Edition, ESRI Press, 2010.

S. Bocconi, F. Nack, L. Hardman, Automatic generation of matter-of-opinion video documentaries, Journal of Web Semantics, 6 (2), 139-150, 2008.

W.S. Cleveland, The Elements of Graphing Data, 2nd Edition, Hobart Press, 1994

C. Colombo, A. Del Bimbo, P. Pala, Semantics in Visual Information Retrieval, IEEE Multimedia, 6(3): 38-53, 1999.

P. Dombrowski, Ernst Haeckel's controversial visual rhetoric, Technical Communication Quarterly 12: 303-319, 2003.

E.H.Gombric, Art and Illusion. A Study in the Psychology of Pictorial Representation, Millennium Edition, Princeton University Press, 2000.

J. Heer, M. Stone, Color Naming Models for Color Selection, Image Editing and Pallette Design, ACM CHI, May 5-10, 2012, Austin, Texas, USA.

C.A. Hill, M. Helmers, Defining Visual Rhetorics, Lawrence Erlbaum Associates, London, 2004.

A.Hubmann-Haidvogel, A.M.P. Brasoveanu, A. Scharl, M. Sabou, S. Gindl. Visualizing Contextual and Dynamic Features of Micropost Streams, In Proceedings of the WWW'12 Workshopon on 'Making Sense of Microposts'. Lyon, France, April 16, 2012, CEURWorkshop Proceedings Vol-838, 34 - 40, 2012.

J. Hullman, N. Diakopoulos, Vizualization Rhetoric: Framing Effects in Narrative Visualization, IEEE TVCG, Vol. 17, No. 12, 2011, 2231-2240.

J. Itten, The Art of Color: The Subjective Experience and Objective Rationale of Color, John Wiley, New York, 1973.

Z.-Q. Jiang, W.-X. Zhou, D. Sornette, R. Woodard, K. Bastiaensen, P. Cauwels, Bubble diagnosis and prediction of the 2005-2007 and 2008-2009 chinese stock market bubbles, Journal of Economic Behavior & Organization 74 (3), 149-162, 2010.

E. R. Kandel, The Age of Insight: The Quest to Understand the Unconscious in Art, Mind, and Brain, from Vienna 1900 to the Present, Random House, 2012

G. Kress, T. van Leeuwen, Reading Images, Routledge, London, 2006.

J. Machajdik, A. Hanburry, Affective Image Classification using Features Inspired by Psychology and Art Theory, Proceedings of the ACM Multimedia, October 25-29, 2010, Firenze, Italy, 83-92.

P. Sarlin, D. Marghescu, Visual Predictions of Current Crises: A Comparison of Self- Organizing Maps with Probit Models, TUCS Techical Report, No 978, June 2010.

E. Segel, J. Heer, Narrative Visualization: Telling Stories with Data, IEEE TVCG, Vol. 16, 2010, 2231-2240.

A.W.M. Smeulders, M. Worring, S. Santini, A. Gupta, R. Jain, Content Based Image Retrieval at the End of the Early Years, IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 22 (12), page 1349-1380, 2000.

D. Sornette, Dragon-kings, black swans and the prediction of crises, International Journal of Terraspace Science and Engineering, 2 (1), pp. 1-18, 2009.

S. Staab, A. Scherp, R. Arndt, R. Troncy, M. Gregorzek, C. Saathoff, S. Schenk, L.Hardman., Semantic multimedia, Reasoning Web, 4th International Summer School, Venice, Italy. Volume 5224 of LNCS, Springer, 125-170, 2008.

R. Troncy, B. Huet, S. Schenk, Multimedia Semantics. Metadata, Analysis and Interaction, Wiley 2011.

E. R. Tufte, The Visual Display of Quantitative Information, 2nd Edition, Graphics Press, 2001.

E. R. Tufte, Beautiful Evidence, Graphics Press, 2006.

T. van Leeuwen, Introducing Social Semiotics, Routledge, 2005.

W.-N. Wang, Y.-L. Yu, Image Emotional Semantic Query Based On Color Semantic Description, Proceedings of the Fourth International Conference on Machine Learning and Cybernetics, Guangzhou, 18-21 August 2005, p.4571-4576.

L. Wilkinson, The Grammar of Graphics, Second Edition, Springer, 2005.

Z.J. Zha, L. Yang, T. Mei, M. Wang, Z. Wang, T.S. Chua, X.S. Hua, Visual Query Suggestion: Towards Capturing User Intent in Internet Image Search, ACM TOMCCAP 6, Issue 3, 1-19, 2010.

W. Yan, R. Rebib, R. Woodard, D. Sornette, Detection of Crashes and Rebounds in Major Equity Markets,, last accessed: 8.04.2012., last accessed: 08.04.2012., last accessed: 08.04.2012., last accessed: 08.04.2012., last accessed: 08.04.2012., last accessed: 12.07.2012., last accessed: 12.07.2012., last accessed: 12.07.2012., last accessed: 12.07.2012., last accessed: 12.07.2012., last accessed: 12.07.2012., last accessed: 12.07.2012., last accessed: 12.07.2012., last accessed: 12.07.2012.



Most read articles by the same author(s)

Obs.: This plugin requires at least one statistics/report plugin to be enabled. If your statistics plugins provide more than one metric then please also select a main metric on the admin's site settings page and/or on the journal manager's settings pages.