www.iiassvietri.it

There is a need for a holistic perspective when considering the assessment of healthy/non-healthy human interaction, in order to detect features that can be able to discriminate among these two categories, in order to develop complex autonomous systems able to detect user’s moods and feelings and take actions to provide help.

The research team on Multimodal Communication, directed by professor Anna Esposito, jointly working at the Second University of Naples, Department of Psychology, and at the International Institute for Advanced Scientific Study (IIASS), investigate on dynamics of social signal exchanges and the human’s ability to decode and encode social cues while interacting in healthy/non-healthy contexts. In order to do this,behavioral data (speaking, body movements, facial, vocal and gestural emotional expressions) are gathered on groups of healthy/differently-diagnosed subjects either adults or children.

Specific behavioral tasks are designed to detect changes in the healthy perception of social cues and behavioral changes due to physical and psychological disorders. Specific scenarios are designed for these tasks intended to assess the user’s emphatic and social competencies (impaired when cognitive disorders are present). The collected data are used to gain knowledge on how behavioral and interactional features are affected by the healthy/non-healthy individual’s status.

During the last 10 years, the team has actively carried out research and dissemination on various social and cognitive disorders producing results which have been object of International publications. In this context, social researches have been carried out, within assistance, to safeguard people’s health and socio-cultural activities have been done for information and prevention. For example, an investigation was carried out on the knowledge of the Campania Region inhabitants on palliative cures.

Palliative cures are useful to “improve the quality of life of patients, who have to face an incurable and evolving illness, and their family, by means of prevention and alleviation of suffering obtained thanks to … the identification, accurate evaluation and pain treatment and … physical, psycho-social and spiritual problems” (WHO - World Health Organization). The result was that only a small part of the Campania Region population is aware of this support (Law n. 39 of 1999; Law n.38 of 2010; Law n.147 of 2013).

Investigations were carried out of the preferences of older people with chronic illnesses and slight cognitive disorders, on the use of intelligent interfaces which assured automatic assistance and improved their quality of life by reminding them to do exercise and take their medicine. Research were carried out to help improve the conditions of dyslexic and stuttering children, and at present are in course researches to support people with depressive disorders.

The research lines and relative publications are hereafter reported:

 In addition, several international events have been organized on these topics to promote cross-cultural and knowledge exchanges:

ARTICLES PUBLISHED ON THE TOPICS

 [1] Esposito, A., Jain, L.C. (2016): Modeling Social Signals and Contexts in Robotic Socially  Believable Behaving Systems. In Esposito, A. & Jain L.C. (Eds)  Toward Robotic Socially Believable Behaving Systems Volume II  - “Modeling Social Signals” (Eds). Intelligent Systems Reference Library, ISRL 106: 5-13, Springer International  Publishing Switzerland. http://www.springer.com/fr/book/9783319310527

[2] Esposito, A., Jain, L.C. (2016): Modeling Emotions in Robotic Socially Believable Behaving Systems. In Esposito, A. & Jain L.C. Eds). Toward Robotic Socially Believable Behaving Systems  Volume I - “Modeling Emotions” Intelligent Systems Reference Library,105: 9-15, Springer International  Publishing Switzerland. http://www.springer.com/gp/book/9783319310558

 [3] Esposito, A., Esposito, A.M., Likforman-Sulem L, Maldonato NM, A Vinciarelli A (2016): On the significance of speech pauses in depressive disorders: Results on read and spontaneous narratives. In Esposito et al (EDS), Springer SIST series on Recent Advances in Nonlinear Speech Processing, 48, 73-82. DOI: 10.1007/978-3-319-28109-4_8.http://link.springer.com/chapter/10.1007%2F978-3-319-28109-4_8

 [4] Kiss CG, Tulics MG, Sztahó D, Esposito, A., Vicsi K (2016): Language Independent Detection Possibilities of Depression by Speech. In Esposito et al (EDS), Springer SIST series on Recent Advances in Nonlinear Speech Processing, 48, 103-114. DOI: 10.1007/978-3-319-28109-4_11, http://link.springer.com/chapter/10.1007%2F978-3-319-28109-4_11

 [5] Esposito, A., Palumbo D, Troncone  A  (2015):  Effects of Narrative Identities and Attachment Style on the Individual’s Ability to Categorize Emotional Voices. In Bassis S. et al (eds) Advances in Neural Networks: Computationaland Theoretical Issues. Series: Smart Innovation, Systems and Technologies, Vol. 37, pages 265-272, Springer VerlagHeidelberg,DOI: 10.1007/978-3-319-18164-6_25, http://link.springer.com/chapter/10.1007/978-3-319-18164-6_25

 [6] Esposito, A., Esposito, A.M, Vogel C (2015): Needs and challenges in human computer interaction for processing social emotional information. Pattern Recognition Letters, 66, 41-51. ISSN: 0167-8655, DOI: 10.1016/j.patrec.2015.02.013, http://dl.acm.org/citation.cfm?id=2838481

 [7]  Fortunati L, Esposito, A., Sarrica M, Ferrin G (2015):  Children's Knowledge and Imaginary About Robots. International Journal of Social Robotics, 7:685–695, DOI: 10.1007/s12369-015-0316-9; http://link.springer.com/article/10.1007/s12369-015-0316-9

 [8]  Fortunati L, Esposito, A., Lugano G (2015)  Beyond Industrial Robotics: Social Robots Entering Public and Domestic Spheres, The Information Society: An International Journal,31:3, 229-236. DOI:10.1080/01972243.2015.1020195, http://dx.doi.org/10.1080/01972243.2015.1020195

 [9] Koutsombogera M, Galanis D, Riviello MT, Tsere N,  Karabetsos S,  Esposito, A., Papageorgiou H(2015):  Conflict Cues in Call Center Interactions.In D’Errico et al. (Eds), Conflict and Multimodal Communication, vol. 26, 431-447, Springer VerlagHeidelberg, DOI; 10.1007/978-3-319-14081-0_20, http://link.springer.com/chapter/10.1007/978-3-319-14081-0_20

 [10] Likforman-Sulem L, Esposito, A., Faundez-Zanuy M, Clémençon S (2015):  Extracting Style and Emotion from Handwriting. In Bassis S. et al (eds) Advances in Neural Networks: Computationaland Theoretical Issues. Series: Smart Innovation, Systems and Technologies, Vol. 37, pages 347-355, Springer VerlagHeidelberg,DOI: 10.1007/978-3-319-18164-6_34, http://link.springer.com/chapter/10.1007/978-3-319-18164-6_34

 [11] Vinciarelli A, Esposito, A., André E, Bonin F, Chetouani M, Cohn JF, Cristan M, Fuhrmann F, Gilmartin E, Hammal Z, Heylen D, Kaiser R, Koutsombogera M, Potamianos A, Renals S, Riccardi G, Salah AA (2015): Open Challenges in Modelling, Analysis and Synthesis of Human Behaviour in Human–Human and Human–Machine Interactions. Cognitive Computation, DOI10.1007/s12559-015-9326-z, 7(4). 397-413, http://link.springer.com/article/10.1007/s12559-015-9326-z#

[12] Vinciarelli A, Chatziioannou P, Esposito, A. (2015): When the words are not everything: the use of laughter, fillers, back-channel, silence, and overlapping speech in phone calls. Frontiers in ICT, vol. 4, article 4, 1-11. DOI:doi: 10.3389/fict.2015.00004, http://journal.frontiersin.org/article/10.3389/fict.2015.00004/abstract

 [13] Cordasco G, Esposito M, Masucci F, Riviello MT, Esposito, A., Chollet G, Schlögl S, Milhorat P, Pelosi G (2014): Assessing voice user interfaces: The vAssist System Prototype. Proc.  5^th IEEE international Conference on Cognitive InfoCommunications, Vietrisul  Mare, 5-7 Nov, 91-96, ISSN=978-1-4799-7280-7/14/$31.00 ©2014 IEEE, DOI: 10.1109/CogInfoCom.2014.7020425 http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=7020425

 [14] Esposito, A., Esposito, A.M, Esposito M (2014): On Distinctive Visual and Auditory Timing Cues in Language Tasks. In Lewandowska-Tomaszczyk B, Kosecki K (Eds): Timeling-Time and Temporality in Human Experience ISBN:978-3-631-64339-6,  Peter Lang/Frankfurt am Main, 201-218.

 [15] Esposito, A., Palumbo D, Troncone A (2014): Emotional consequences of the interaction between inward/outward identities and attachment style. Proc.  5^th IEEE international Conference on Cognitive InfoCommunications, Vietrisul  Mare, 5-7 Nov, 263-266, 978-1-4799-7280-7/14/$31.00 ©2014 IEEE, DOI: 10.1109/CogInfoCom.2014.702045,  http://dx.doi.org/10.1109/coginfocom.2014.7020458 

 [16] Esposito, A., Fortunati L, Lugano G  (2014):  Modeling emotion, behaviour and context in socially believable robots and ICT interfaces. Cognitive Computation, 6( 4), 623-627, DOI 10.1007/s12559-014-9309-5, http://link.springer.com/article/10.1007/s12559-014-9309-5

 [17] Esposito, A., Palumbo D, Troncone A (2014): The influence of the attachment style on the decoding accuracy of emotional vocal expressions. Cognitive Computation, 6 (4), 699-707, DOI: 10.1007/s12559-014-9292-x. http://link.springer.com/article/10.1007%2Fs12559-014-9292-x

 [18] Faundez-Zanuy  M, Sesa-Nogueras E, Roure-Alcobé J, Mekyska J, Esposito, A.,  Lopez-De-Ipiña K (2014): A preliminary study on aging examining online handwriting.  Proc.  5^th IEEE international Conference on Cognitive InfoCommunications, Vietrisul  Mare, 5-7 Nov, 221-224, 978-1-4799-7280-7/14/$31.00 ©2014 IEEE,  DOI: 10.1109/CogInfoCom.2014.7020449, http://dx.doi.org/10.1109/coginfocom.2014.7020449 

 [19] Fortunati L, Esposito, A., Ferrin G, Viel M (2014): Approaching social robots through playfulness and doing-it-yourself: children in action. Cognitive Computation, ISSN 1866-9956, Volume 6, Number 4, 789-801, DOI:10.1007/s12559-014-9303-y. SCOPUS CODE: eid=2-s2.0-84906651431 http://link.springer.com/article/10.1007/s12559-014-9303-y

 [20] Milhorat P, Schlögl S,  Chollet G,  Boudyy J,  Esposito, A., Pelosi G (2014): Building the next generation of personal digital assistants. Proc. of 1st  IEEE International Conference on Advanced Technologies for Signal and Image Processing - ATSIP'2014 March 17-19, 2014, Sousse, Tunisia, 458-463, ISSN  978-1-4799-4888-8/14/,  DOI:10.1109/ATSIP.2014.6834655, http://dx.doi.org/10.1109/atsip.2014.6834655

 [21] Riviello MT, Capuano V, Ombrato G, Baldassarre I, Cordasco C, Esposito, A. (2014): The Influence of Positive and Negative Emotions on Physiological Responses and Memory Task Scores. In Bassis S et al. (Eds), Recent Advances of Neural Network Models and Applications, vol. 26, 315-324, Springer VerlagHeidelberg, DOI 10.1007/978-3-319-041292. http://www.springer.com/engineering/computational+intelligence+and+complexity/book/978-3-319-04128-5

 [22]Troncone A, Palumbo D, Esposito, A. (2014): Mood effects on the decoding of emotional voices. In Bassis S et al. (Eds), Recent Advances of Neural Network Models and Applications, vol. 26, 325-332, Springer VerlagHeidelberg, DOI 10.1007/978-3-319-04129-2. http://dx.doi.org/10.1007/978-3-319-04129-2_32 

 [23] Vicsi K, Esposito,  A. (2014): Communicative social signals: Computational and behavioural aspects of human-human and human-machine interaction. Intelligent Decision Technologies,  8 (2014) 253–254, ISSN print 1872-4981 ISSN online 1875-8843 DOI 10.3233/IDT-140193 IOS Press http://iospress.metapress.com/content/yu35256315737352/

 [24] Esposito, A., Riviello MT, Capuano V (2013): Discriminating Human vs. Stylized Emotional Faces: Recognition Accuracy in Young Children.  In Apolloni B et al. (Eds): Neural Nets and Surroundings, 19, 395-403. Series Smart Innovation, Systems and Technologies.  Springer VerlagHeidelberg, ISBN 978-3-642-35466-3,  http://dx.doi.org/10.1007/978-3-642-35467-0_39 

 [25] Faundez-Zanuy M., Sesa-Nogueras E, Roure-Alcobe J, Garre-Olmo J, Mekyska J, Lopez-de-Ipiña K, Esposito, A. (2013):  A Preliminary Study of Online Drawings and Dementia Diagnose. In Apolloni B et al. (Eds): SISTS Series: Neural Nets and Surroundings, 19, 367-374.  http://link.springer.com/chapter/10.1007%2F978-3-642-35467-0_36#

 [26] Esposito, A. (2012): Visual and auditory emotional information: What counts in human-human and human-machine interaction. NEUROPSYCHIATRIE DE L'ENFANCE ET DE L'ADOLESCENCE, 60(55), S41, ISSN: 0222-9617,http://www.em-consulte.com/en/article/737207

 [27] Bourbakis N, Esposito, A., Kavraki D (2011): Extracting and Associating Meta-features for Understanding People’s Emotional Behaviour: Face and Speech. Cognitive Computation Journal, 3(3),  436-448, http://dx.doi.org/10.1007/s12559-010-9072-1 

[28] Esposito, A., Riviello MT (2011): The cross-modal  and cross-cultural processing of affective information. In B. Apolloni et al (Eds), Frontiers in Artificial Intelligence and Applications 226, 301-310, IOSpress, ISBN  978-1-60750-691-1 (print), ISBN  978-1-60750-692-8 (online) 301-310 (2011).  DOI: 10.3233/978-1-60750-692-8-301, http://ebooks.iospress.nl/volumearticle/6396

 [29] Esposito, A., Troncone A (2011): Emotions and Speech Disorders: Do Developmental Stutters Recognize Emotional Vocal Expressions? In A. Esposito et al. (Eds.): Toward Autonomous, Adaptive, and Context-Aware Multimodal Interfaces: Theoretical and Practical Issues , LNCS 6456, pp. 155–164, Springer-Verlag Berlin Heidelberg, http://dx.doi.org/10.1007/978-3-642-18184-9_14 

 [30] Rossini N, Esposito, A. (2011):  Modelling Human Behaviour in Emergency: A Research Agenda for the Creation of a Rescue Robot.  In Apolloni B et al.(Eds) Frontiers in Artificial Intelligence and Applications, 234, 235-246,  ISBN 978-1-60750-971-4, IOSpress, DOI: 10.3233/978-1-60750-972-1-235, http://ebooks.iospress.nl/volumearticle/6758

  [31] Troncone A,  Esposito, A. (2011): On the recognition of emotional voices by typical and speech impaired children. In Apolloni, B., Bassis, S., Esposito, A., Morabito, C.F. (Eds): Neural Nets WIRN11, Vol. 234,  Frontiers in Artificial Intelligence and Applications, pp 228-234, ISBN 978-1-60750-971-4, IOSpress, DOI: 10.3233/978-1-60750-972-1-228, http://ebooks.iospress.nl/volumearticle/6757

 [32] Keefer R, Dakopoulos D. , Esposito,  A,  Bourbakis, N (2009):  An Interaction Based Approach to Document Segmentation for the Visually Impaired.  InJacko, Julie A. (Ed.) Universal Access in Human-Computer Interaction. Applications and Services, Lecture Notes in Computer Science, 5616, PartIII, 540-549, , Springer VerlagHeidelberg   http://dx.doi.org/10.1007/978-3-642-02713-0_57 

 [33] Bourbakis N, Keefer R, Dakopoulos D, Esposito, A. (2008): A Multimodal Interaction Scheme between a Blind User and the Tyflos Assistive Prototype. In Proceedings of IEEE Int'l Conference on, Tools with Artificial Intelligence - ICTAI 2008, vol.1, 147-152, ISBN: 978-0-7695-3440-4,, Dayton, OHIO, USA, Nov 3-5, 2008,  http://dx.doi.org/10.1109/ictai.2008.52 

 [34] Bourbakis N, Esposito, A., Kavraki D (2008): Ekfrasis: A Formal Language for Representing and Generating Sequences of Facial Patterns for Studying Emotional Behavior. In A. Esposito et al. (eds) Verbal and Nonverbal Features of Human-Human  and Human-Machine Interaction, LNCS vol. 5042, 21-31, Springer VerlagHeidelberg. http://link.springer.com/chapter/10.1007%2F978-3-540-70872-8_2

 [35] Bourbakis NG, Esposito, A.,  Kavraki D (2007): Multi-Modal Interfaces for Interaction-Communication between Hearing and Visually Impaired Individuals: Problems & Issues. IEEE International Conference on Tools with Artificial Intelligence, Patras, Greece, Oct. 29-31, vol. 2, pp. 522 - 530 , ISBN: 978-0-7695-3015-4,  http://dx.doi.org/10.1109/ictai.2007.178 

 [36] Esposito, A.,  Stejskal V, Smékal Z, Bourbakis N (2007): The Significance of Empty Speech Pauses: Cognitive and Algorithmic Issues. In F. Mele et al (eds): Brain Vision Artificial Intelligence, LCNS, 4729, 542-554, Springer-Verlag Berlin Heidelberg, ISBN 80-86269-15-9. http://dx.doi.org/10.1007/978-3-540-75555-5_52 

 [37] Bourbakis NG, Esposito, A., Kavraki D (2006):  Analysis of Invariant Meta-features for Learning and Understanding Disable People’s Emotional Behavior Related to their Health Conditions: A Case Study. In Proceedings of 6^th International IEEE Symposium BioInformatics and BioEngineering, pp. 357-369, IEEE Computer Society, http://dl.acm.org/citation.cfm?id=1169418. http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=4019681&url=http%3A%2F%2Fieeexplore.ieee.org%2Fiel5%2F4019626%2F4019627%2F04019681.pdf%3Farnumber%3D4019681

 [38] Esposito, A., Bourbakis NG (2006): The Role of Timing in Speech Perception and Speech Production Processes and its Effects on Language Impaired Individuals. In Proceedings of 6^th International IEEE Symposium BioInformatics and BioEngineering, pp. 348-356, ISBN 0-7695-2727-2, IEEE Computer Society. http://dx.doi.org/10.1109/bibe.2006.253300 

 [39] Esposito, A. (2006): Children’s Organization of Discourse Structure through Pausing Means. In M. Faundez-Zanuy  et al. (eds), Nonlinear Analyses and Algorithms for Speech Processing, Lecture Notes in Computer Science(LCNS), vol.3817, pp.108-115, ISBN: 3-540-31257-9,Springer Verlag  Heidelberg.  http://dx.doi.org/10.1007/11613107_9