Share on facebook
Save on delicious
RSS Feed
HeadTalk
Principal Investigator - Prof. Ron Carter (ronald.carter@nottingham.ac.uk)
Co-Principal Investigators - Dr Svenja Adolphs, Dr Andy Crabtree, Dr Norbert Schmitt, Dr Tony Pridmore, Dr S Mills
Start date - October 2005 Now Completed
Duration - 12 months
Location - University of Nottingham
This is an applied project that is in the early stages of development. It explores e-social science applications of a user-friendly multi-modal, multi-media corpus tool for the analysis of key aspects of everyday human interaction. The project focuses in particular on the relationship between language and gesture. It provides a rubric for explorations of the relationships between linguistic characteristics and context of specific gestures and physically descriptive representations of those gestures extracted from video data. Attention is specifically focused upon the identification, classification and representation of gestural signals of active listenership, in particular head nods, providing an investigation of the technical and practical issues involved.
The project, (which draws on upwards 100,000 words of data) also explores the possible ways in which the tools and data-sets that are required to facilitate multi-modal corpus analysis might be extended to support different communities of practice and how they might be used to enhance descriptions of language and the applications of such descriptions. Applications will lead to a greater understanding of the characteristics of verbal and non-verbal behaviour in natural conversation, allowing us to explore in more detail the relationships between linguistic form and function in discourse. It also facilitates explorations of how different, complex facets of meaning in discourse are constructed through the interplay of text, gesture and voice. This project complements work that is being carried out by the e-social science research node (Digital Records) at the University of Nottingham.
Download
The HeadTalk demonstrator, Cvision, is an interactive program which allows users to apply the visual tracking algorithm developed within the HeadTalk project to selected targets in an input video clip. This algorithm, KAMS (Kernel Annealed Mean-Shift tracking), is a hybrid particle filter/kernel mean-shift tracker formed by combination of the Annealed Particle Filter and the Kernel Mean-Shift algorithm. Though KAMS has been shown to outperform the original particle filter of Isard and Blake, Annealed Particle Filtering, and the earlier hybrid tracker of Maggio and Cavallero, Cvision also incorporates, for comparison, each of these three algorithms.
Cvision takes as its input an avi format video file and produces a text file giving the estimated position of each target in each frame of that video. This may be imported into the current version of the DreSS 2 tool DRS. An output video may also be produced if desired, this shows the results of tracking overlaid on the input video images and is a useful debugging and interpretation tool. Cvision allows multiple targets to be tracked in parallel, producing a description of the motion of each and showing intermediate results as they are obtained. Cvision is written in C++ and provided as a Windows .exe file.
Presentations
Head-Talk: Towards a multi-modal corpus
R. Carter, D. Knight, S. Adoplhs, University of Nottingham
Joint Annual Meeting of the British Assoicaiton for Applied Linguistics and the Irish Association for Applied Linguistics Conference, Cork, Ireland, 7 - 9 September 2006
The linguistic coding of verbal and non-verbal backchannels: a preliminary approach
D. Knight, S. Adoplhs, R. Carter, University of Nottingham
Node meeting, Nottingham, July 2006
Beyond the Text: Construction and Analysis of Multi-Modal Linguistic Corpora
D. Knight, S. Bayoumi, S. Mills, A. Crabtree, S. Adolphs, T. Pridmore, R. Carter, University of Nottingham
2nd International Conference on e-Social Science, Manchester, 28 ¨C 30 June 2006
The Multi-Modal Corpus: Coding and representing data- the issues
D. Knight, R. Carter, S .Adolphs, University of Nottingham
3rd IVACS conference, Nottingham, 23 ¨C 24 June 2006
Analysing spoken corpora: Methodological issues and technical challenges
S. Adolphs, D. Knight, University of Nottingham
BAAL SIG Seminar (Special Interest Group: Corpus Linguistics), Milton Keynes, 28 April 2006
Developing a Multi-Modal Corpus: Data Coding Issue
D. Knight, University of Nottingham
IVACS Annual Research Symposium, University of Limerick, Ireland, 4 February 2006
The linguistic coding of backchannels: A proposed methodological approach
D. Knight, R. Carter, S. Adolphs, University of Nottingham
Node Meeting, Nottingham, December 2005
HeadTalk: Issues in multi-modal communication analysis
D. Knight, R. Carter, S. Adolphs, University of Nottingham
Node Meeting, Nottingham, October 2005
Publications
Beyond the Text: Construction and Analysis of Multi-Modal Linguistic Corpora
D. Knight, S. Bayoumi, S. Mills, A. Crabtree, S. Adolphs, T. Pridmore, R. Carter, University of Nottingham
Published in the Proceedings of the 2nd International Conference on e-Social Science, Manchester, 28 - 30 June 2006