The Role of Visual Rhetoric in Semantic Multimedia: Strategies for Decision Making in Times of Crisis

Adrian M.P. Brasoveanu, Ioan Dzitac

Abstract


As semantic multimedia is approaching mainstream, even the great improvements that can be seen in its classic schools, like the data mining inspired Information Retrieval based on metadata analysis, or Computer Vision, might not be enough. We identify a new group that gains traction in the semantic multimedia community and which uses as starting point developments from psychology and visual communication. For the purposes of this article we restrict our domain to visual rhetoric as we consider it to yield the biggest potential for future developments. Living in times when the periods between crises seem to be shorter and shorter, we look at how developments in semantic multimedia can be used for predicting and overcoming crises. We analyze at least 2 aspects related to this: using information visualization to understand the evolution of crises and creating multi-layered semantic multimedia technologies that can easily be adapted to use a variety of sources and solve problems from different domains. In both cases we show how techniques inspired by visual rhetoric (information linking, framing, composition) in conjunction with named entity recognition offer a lot of benefits. The section related to multi-layered semantic multimedia technologies also draws on the lessons learned while designing a prototype application aimed at improving tourism decision making process. The article ends with a discussion on evaluation methods for multi-layered semantic technologies applications. We look at how to evaluate them on both levels: mechanisms (information linking versus raw named entity recognition when generating visuals, for example), and decision making strategies (Do such systems actually solve real problems related to crises, create jobs or at least can they be repurposed to solve other problems than the one with which we have started?).


Keywords


semantic multimedia, visual rhetoric, text to visual matching, interactive documentary, crisis strategies, multimedia storytelling

Full Text:

PDF

References


A. Ahmed, L.P. Coelho, A. Arnold, J. Kangas, A.B. Sheikh, E. Xing, W. Cohen, R.F. Murphy, Structured literature image finder: Parsing text and figures in biomedical literature, Journal of Web Semantics: Science, Services and Agents on the World Wide Web, 8 (2-3), 151-154, 2010
http://dx.doi.org/10.1016/j.websem.2010.04.002

R. Arnheim, Art and Visual Perception: A Psychology of The Creative Eye, 2nd Edition, Cambridge, 2004.

R. Barthes, Image, Music, Text, Hill and Wang, 1977.

J. Bertin, Semiology of Graphics: Diagrams, Networks, Graphs, 2nd Edition, ESRI Press, 2010.

S. Bocconi, F. Nack, L. Hardman, Automatic generation of matter-of-opinion video documentaries, Journal of Web Semantics, 6 (2), 139-150, 2008.
http://dx.doi.org/10.1016/j.websem.2008.01.004

W.S. Cleveland, The Elements of Graphing Data, 2nd Edition, Hobart Press, 1994

C. Colombo, A. Del Bimbo, P. Pala, Semantics in Visual Information Retrieval, IEEE Multimedia, 6(3): 38-53, 1999.
http://dx.doi.org/10.1109/93.790610

P. Dombrowski, Ernst Haeckel's controversial visual rhetoric, Technical Communication Quarterly 12: 303-319, 2003.
http://dx.doi.org/10.1207/s15427625tcq1203_5

E.H.Gombric, Art and Illusion. A Study in the Psychology of Pictorial Representation, Millennium Edition, Princeton University Press, 2000.

J. Heer, M. Stone, Color Naming Models for Color Selection, Image Editing and Pallette Design, ACM CHI, May 5-10, 2012, Austin, Texas, USA.

C.A. Hill, M. Helmers, Defining Visual Rhetorics, Lawrence Erlbaum Associates, London, 2004.

A.Hubmann-Haidvogel, A.M.P. Brasoveanu, A. Scharl, M. Sabou, S. Gindl. Visualizing Contextual and Dynamic Features of Micropost Streams, In Proceedings of the WWW'12 Workshopon on 'Making Sense of Microposts'. Lyon, France, April 16, 2012, CEURWorkshop Proceedings Vol-838, 34 - 40, 2012.

J. Hullman, N. Diakopoulos, Vizualization Rhetoric: Framing Effects in Narrative Visualization, IEEE TVCG, Vol. 17, No. 12, 2011, 2231-2240.

J. Itten, The Art of Color: The Subjective Experience and Objective Rationale of Color, John Wiley, New York, 1973.

Z.-Q. Jiang, W.-X. Zhou, D. Sornette, R. Woodard, K. Bastiaensen, P. Cauwels, Bubble diagnosis and prediction of the 2005-2007 and 2008-2009 chinese stock market bubbles, Journal of Economic Behavior & Organization 74 (3), 149-162, 2010.
http://dx.doi.org/10.1016/j.jebo.2010.02.007

E. R. Kandel, The Age of Insight: The Quest to Understand the Unconscious in Art, Mind, and Brain, from Vienna 1900 to the Present, Random House, 2012

G. Kress, T. van Leeuwen, Reading Images, Routledge, London, 2006.

J. Machajdik, A. Hanburry, Affective Image Classification using Features Inspired by Psychology and Art Theory, Proceedings of the ACM Multimedia, October 25-29, 2010, Firenze, Italy, 83-92.

P. Sarlin, D. Marghescu, Visual Predictions of Current Crises: A Comparison of Self- Organizing Maps with Probit Models, TUCS Techical Report, No 978, June 2010.

E. Segel, J. Heer, Narrative Visualization: Telling Stories with Data, IEEE TVCG, Vol. 16, 2010, 2231-2240.

A.W.M. Smeulders, M. Worring, S. Santini, A. Gupta, R. Jain, Content Based Image Retrieval at the End of the Early Years, IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 22 (12), page 1349-1380, 2000.
http://dx.doi.org/10.1109/34.895972

D. Sornette, Dragon-kings, black swans and the prediction of crises, International Journal of Terraspace Science and Engineering, 2 (1), pp. 1-18, 2009.

S. Staab, A. Scherp, R. Arndt, R. Troncy, M. Gregorzek, C. Saathoff, S. Schenk, L.Hardman., Semantic multimedia, Reasoning Web, 4th International Summer School, Venice, Italy. Volume 5224 of LNCS, Springer, 125-170, 2008.

R. Troncy, B. Huet, S. Schenk, Multimedia Semantics. Metadata, Analysis and Interaction, Wiley 2011.
http://dx.doi.org/10.1002/9781119970231

E. R. Tufte, The Visual Display of Quantitative Information, 2nd Edition, Graphics Press, 2001.

E. R. Tufte, Beautiful Evidence, Graphics Press, 2006.

T. van Leeuwen, Introducing Social Semiotics, Routledge, 2005.

W.-N. Wang, Y.-L. Yu, Image Emotional Semantic Query Based On Color Semantic Description, Proceedings of the Fourth International Conference on Machine Learning and Cybernetics, Guangzhou, 18-21 August 2005, p.4571-4576.

L. Wilkinson, The Grammar of Graphics, Second Edition, Springer, 2005.

Z.J. Zha, L. Yang, T. Mei, M. Wang, Z. Wang, T.S. Chua, X.S. Hua, Visual Query Suggestion: Towards Capturing User Intent in Internet Image Search, ACM TOMCCAP 6, Issue 3, 1-19, 2010.

W. Yan, R. Rebib, R. Woodard, D. Sornette, Detection of Crashes and Rebounds in Major Equity Markets, http://adsabs.harvard.edu/abs/2011arXiv1108.0077Y, last accessed: 8.04.2012.

http://mazamascience.com/WorkingWithData/?p=870, last accessed: 08.04.2012.

http://www.informationisbeautifulawards.com/, last accessed: 08.04.2012.

http://www.economicpredictions.org/who-predicted-the-financial-crisis.htm, last accessed: 08.04.2012.

http://trecvid.nist.gov/, last accessed: 08.04.2012.

http://www.mercer.com/articles/quality-of-living-survey-report-2011, last accessed: 12.07.2012.

http://www.visualisingdata.com/, last accessed: 12.07.2012.

http://www.informationisbeautiful.net/, last accessed: 12.07.2012.

http://flowingdata.com/, last accessed: 12.07.2012.

http://www.visualcomplexity.com/vc/, last accessed: 12.07.2012.

http://visual.ly/, last accessed: 12.07.2012.

http://eagereyes.org/, last accessed: 12.07.2012.

http://blog.typekit.com/2012/06/12/designing-data/, last accessed: 12.07.2012.

http://twitter.com/nytgraphics/, last accessed: 12.07.2012.




DOI: https://doi.org/10.15837/ijccc.2012.4.1361



Copyright (c) 2017 Adrian M.P. Brasoveanu, Ioan Dzitac

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

CC-BY-NC  License for Website User

Articles published in IJCCC user license are protected by copyright.

Users can access, download, copy, translate the IJCCC articles for non-commercial purposes provided that users, but cannot redistribute, display or adapt:

  • Cite the article using an appropriate bibliographic citation: author(s), article title, journal, volume, issue, page numbers, year of publication, DOI, and the link to the definitive published version on IJCCC website;
  • Maintain the integrity of the IJCCC article;
  • Retain the copyright notices and links to these terms and conditions so it is clear to other users what can and what cannot be done with the  article;
  • Ensure that, for any content in the IJCCC article that is identified as belonging to a third party, any re-use complies with the copyright policies of that third party;
  • Any translations must prominently display the statement: "This is an unofficial translation of an article that appeared in IJCCC. Agora University  has not endorsed this translation."

This is a non commercial license where the use of published articles for commercial purposes is forbiden. 

Commercial purposes include: 

  • Copying or downloading IJCCC articles, or linking to such postings, for further redistribution, sale or licensing, for a fee;
  • Copying, downloading or posting by a site or service that incorporates advertising with such content;
  • The inclusion or incorporation of article content in other works or services (other than normal quotations with an appropriate citation) that is then available for sale or licensing, for a fee;
  • Use of IJCCC articles or article content (other than normal quotations with appropriate citation) by for-profit organizations for promotional purposes, whether for a fee or otherwise;
  • Use for the purposes of monetary reward by means of sale, resale, license, loan, transfer or other form of commercial exploitation;

    The licensor cannot revoke these freedoms as long as you follow the license terms.

[End of CC-BY-NC  License for Website User]


INTERNATIONAL JOURNAL OF COMPUTERS COMMUNICATIONS & CONTROL (IJCCC), With Emphasis on the Integration of Three Technologies (C & C & C),  ISSN 1841-9836.

IJCCC was founded in 2006,  at Agora University, by  Ioan DZITAC (Editor-in-Chief),  Florin Gheorghe FILIP (Editor-in-Chief), and  Misu-Jan MANOLESCU (Managing Editor).

Ethics: This journal is a member of, and subscribes to the principles of, the Committee on Publication Ethics (COPE).

Ioan  DZITAC (Editor-in-Chief) at COPE European Seminar, Bruxelles, 2015:

IJCCC is covered/indexed/abstracted in Science Citation Index Expanded (since vol.1(S),  2006); JCR2018: IF=1.585..

IJCCC is indexed in Scopus from 2008 (CiteScore2018 = 1.56):

Nomination by Elsevier for Journal Excellence Award Romania 2015 (SNIP2014 = 1.029): Elsevier/ Scopus

IJCCC was nominated by Elsevier for Journal Excellence Award - "Scopus Awards Romania 2015" (SNIP2014 = 1.029).

IJCCC is in Top 3 of 157 Romanian journals indexed by Scopus (in all fields) and No.1 in Computer Science field by Elsevier/ Scopus.

 

 Impact Factor in JCR2018 (Clarivate Analytics/SCI Expanded/ISI Web of Science): IF=1.585 (Q3). Scopus: CiteScore2018=1.56 (Q2);

SCImago Journal & Country Rank

Editors-in-Chief: Ioan DZITAC & Florin Gheorghe FILIP.