January 1, 2024
The Affinity project aims to move Haru closer to the target of natural, flexible and effective speech interaction. This project is integrating a state-of-the-art conversational listener, Chatty SDK, with the goal of keeping Haru in sync with dialog partners, judging their mood and supporting natural turn-taking without unnatural pauses and long wait-times for responses. This novel part of the Haru system is actively supported by CereProc Ltd., who offer support and additional functionality throughout the project. In this report, we discuss our progress in producing test harnesses, extending the existing Tiers of Friendship demo , evaluating current systems of rapid turn-taking in context, outline current work on incorporating backchannels into dialog, and discuss findings from our studies on human/human as well as human/robot dialog.
To summarise, the Affinity project contributed the following:
- Creation of light weight chat demo integrating CereProc Chatty SDK.
- Installed cross site to support for further research.
- Human-Human baseline data.
- Statistical DM POC.
- Example data set (16 dialogs)
- Human-Robot Study on Fast/slow Turn-taking
Future work may include:
- Reproduce demo with new DM and LLMs.
- Resilient Local word spotting/ASR to Cloud ASR.
- Voice Onset Prediction (VAP).
- Multi-modal backchanneling.