Conversational Agents: Mastering, Evaluating, Optimizing (CAMEO)
PRIN: PROGETTI DI RICERCA DI RILEVANTE INTERESSE NAZIONALE 2022
About
Conversational agents have overwhelmingly gained importance in the last years, being widely used in chatbots, smartphones, and smart home devices, e.g., Google Home, Amazon Alexa, Apple Siri. However, they still lack the capability to understand and support real-time conversations between humans and digital counterparts in a natural, diverse and engaging way.
CAMEO aims at enhancing conversational agents through an innovative use of user contextual information. In CAMEO, the user context becomes the internal knowledge of the system, including a representation of relevant information collected during past interactions of the user with the system. Such contextual information, enriched with external knowledge from the application domain considered, will be exploited to extract actionable insights enabling a rich and satisfying user interaction through advanced dialogue management, query answering, personalized search and recommendation. Moreover, CAMEO will, for the first time, jointly leverage search and recommendation approaches to improve conversational agents, also developing a principled unifying theoretical framework. Finally, CAMEO will provide researchers with better evaluation methodologies and a visual analytics environment for exploration and explanation of conversational agents behaviour in order to better support them in the design, implementation, and optimization of such systems.
CAMEO will be driven and validated by a real-world use case. The Industry 4.0 use case concerns remote support to field workers in maintenance operations on industrial machines.
CAMEO targets the following objectives:
The scientific advancement of CAMEO will produce new technology in terms of the open source tools for conversational agents as well as their integration into the visual analytics environment. These open source tools will be made available in public git repositories, under Apache 2.0 license (allowing both commercial and not-commercial use), accessible also after the end of the project.
CAMEO will organize an open evaluation contest, in the context of the CLEF evaluation campaign series, to share the dataset produced by the project and compare the CAMEO approaches with those developed by competing groups in academia and industry. CAMEO will conduct deep analyses and statistical validation of the experimental results.
The evaluation contest will transfer knowledge by involving the academic and industrial communities and it will produce large-scale datasets, which will be anonymized and made available in a public git repository, under Creative Commons license (allowing both commercial and not-commercial use), accessible also after the end of the project. Moreover, CAMEO will adopt an Open Science approach and will adhere to the FAIR principles for scientific data. Finally, CAMEO will integrate its dataset into the European Open Science Cloud (EOSC).
Otero, D., Parapar, J., and Ferro, N. (2023). How Discriminative Are Your Qrels? How To Study the Statistical Significance of Document Adjudication Methods. In Frommholz, I., Hopfgartner, F., Lee, M., Oakes, M., Lalmas, M., Zhang, M., and Santos, R., editors, Proc. 32nd International Conference on Information and Knowledge Management (CIKM 2023), pages 1960-1970. ACM Press, New York, USA. https://doi.org/10.1145/3583780.3614916
Nicola Ferro,
University of Padova
Marco Ferrante,
University of Padova
Guglielmo Faggioli,
University of Padova
Raffaele Perego,
Centro Nazionale delle Ricerche
Franco Maria Nardini,
Centro Nazionale delle Ricerche
Tommaso Di Noia,
Polythecnic of Bari
