Work Package 7

WP7 is in charge of validating, through testing and performance evaluation, the OCTAVE platform both in the laboratories and in real-life operational scenarios. During its first year, the WP has planned in-laboratory testing and validation of all components and modules of the TBAS system, has set up a complete test environment, including a database of speech corpora, and carried out a technological test campaign. In the second year, the WP is expected to complete in-laboratory testing, and plan and carry out all in-field activities directly involving panels of end-users. To this purpose, WP7 relies on the OCTAVE business Beneficiaries providing applications and real users. There are two applications, one dealing with local access to a restricted, outdoor physical area, and another catering for remote access to banking information and services. Key performance indicators developed in conjunction with WP8 are also measured.

OBJECTIVES:

  • to create software test plans and automated tests for the assessment of functionality and performance of SV units and supporting components provided by research and industrial partners;
  • to define and extract a database of appropriate size from existing corpora, reflecting data conditions and characteristics found in practical scenarios affected by variations in terms of spoken language and communication channel types (GSM, satellite telephony, VoIP, etc.);
  • to gather user needs, preferences and requirements in accessing the system by voice-based interaction protocols. Accessibility requirements of people with disabilities are also included. This will support the whole iteration of user-centred design process and in-house evaluation;
  • to develop methods for evaluating the robustness of hybrid speaker verification systems to spoofing attacks;
  • to execute tests on individual SV components as well as component pair-wise testing;
  • to validate the TBAS resulting from the integration activities, i.e. the pilot OCTAVE Platform produced by WP6;
  • to assess, with a high degree of confidence, the performance of SV components and of the TBAS against its predetermined quality attributes;
  • to validate the Platform in the real application environments, by means of a selected user panel. Subjects in usability test will be recruited according to a statistical model of the target users;
  • to measure end-users performance in using voice biometrics and technology acceptance in selected scenarios of use or real-life applications.

DESCRIPTION OF WORK

The work is scheduled along four tasks.

  1. Trial Design, including all the work to design (a) the in-laboratory experiments at different levels, i.e. testing of single modules and units, integration testing and overall TBAS testing, and (b) the in-field trials, on the basis of user requirements for both pre-set applications.
  2. Corpora Definition and Collection”, implementing the indications coming from Design task. Existing relevant corpora and related tools are organised in a suitable structure for use by all the project partners. New corpora are foreseen, as OCTAVE requirements are in fact highly demanding, due to the presence of various voice biometric engines, the adoption of speech enhancing technologies for environmental robustness, and the inclusion of spoofing countermeasures.
  3. Trial Execution in Laboratory, in charge of the laboratory validation. Mid-project technologies are evaluated and tested in laboratory with the aim to (a) consolidate and validate the achieved performance with reference to the subsequent on-field trials involving the end-users, and (b) to assess the system performance both as a whole and from single components views.
  4. User Validation, aiming to collect the feedback from legitimate end-users, in terms of system acceptance, under ordinary working conditions. Validation is meant to prove that OCTAVE TBAS is trustworthy in real working conditions and perceived as a secure shield against spoofing and fraudulent fake of biometric credentials. In real-life assessment scenarios, the legitimate end-users (about 50 for each trial) gain experience in trying and using the system, thus coming to useful conclusions as regards performance, usability, security perception, and acceptability of the proposed ICT innovation. After bringing this information back to the development team, the design itself can be refined and the system prototype based on vocal biometrics may gain higher accuracy.

ACCOMPLISHMENTS IN MERIT OF WP 7

Trials have been thoroughly designed

The work has defined the measurement parameters and the test plans to be executed. Test and evaluation aim both at tuning the single modules and at integrating the various modules that make up the Platform. Moreover, the evaluation strategy has specifically targeted the Platform and the services it deploys in the most possible realistic scenarios, in laboratory and in field.

in-field-scenario

A “corpus of corpora” database has been designed and implemented

Availability of spoken language dataset has proved crucial to the progress of research and technology on speech processing, ever since. Bearing this in mind, we have designed and implemented a unified database, where metadata of virtually any speech-dataset can be represented. The project has defined and normalised a general format for the metadata. Open-source tools have been used to import specific datasets into the database. Tools for the management of the data have also been also realised (e.g. automatic transcription, quality measurements, etc.). In this way, speech-signals from  a specific dataset can be easily recalled. The unified database can be seen indeed as a “corpus of corpora”.

corpus of corpora

Additional ad hoc corpora have been collected

The new corpus reflects operational conditions, like the existence of specific noises related to the outdoor application selected by the Project. It covers also the need to train, test and evaluate systems against so far underestimated risks coming from replay attack spoofing. Moreover, it allows experimenting with innovative audio acquisition devices, particularly those featured with dual-microphone technology (acoustic and throat microphone), a promising solution to definitely enhance performance against environmental noise.

noise-and-replay

DELIVERABLES RELEASED SO FAR BY WP 7

The main objective and motivation of this deliverable are to conduct a usability test of the final developed trusted biometric ...
The OCTAVE Trusted Biometric Authentication Service (TBAS) has been designed and set-up during the entire project duration, with the aim ...
How well does a voice biometric authentication service work? How easy is it to use? How reliable is it? What ...
The OCTAVE T-BAS Service has been designed and setup during the entire project duration, with the aim: to build a ...

D25 – Spoofing Corpora

This document accompanies a related collection of audio material referred to as the OCTAVE spoofing corpus. In detail, the spoofing ...

D17 – Corpora Collection

The present Deliverable addresses the objective of assembling, in the most appropriate way, the necessary datasets and their associated metadata ...
This Deliverable opens the phase of the Project dedicated to the provision of the speech and audio data required for ...

D08 – Report on Trial Design

This report provides a detailed analysis of the trial and testing scenarios of the Trusted Biometric Authentication Service (TBAS) targeted ...
Since the technological target of the OCTAVE system is to deliver effective methods for introducing robustness into speaker verification to ...

WP LEADER

Mauro Falcone

Technical Leader and WP7 Leader

Mauro Falcone was born in Rome in 1961. He received a degree, with maximum vote, in Physics from the University of Rome “La Sapienza”, discussing a thesis on automatic speech recognition. He served as project advisor for several TLC and ICT companies. In 1988 he started an external collaboration with FUB. In 1990 he joined FUB to conduct a research for a European project on speech quality assessment. Then served as task leader in many projects, under the aegis of  ESPRIT, FP6 and COST programmes. He also had the role of EC reviewer. He has been expert consultant for the Italian Judicial Authority. He has been Italian representative in ITU, with a rapporteur role, carrying out several speech and audio quality campaigns for both ITU and ETSI. His research areas concern: speech and audio system evaluation/assessment; speech and audio quality; speech technology; speaker recognition; biometrics. He is in the scientific committee of the most important conferences in the field, and in a number of review committees. He is author of about 100 publications. He is a member or associate AES, ISCA, AISV, SQEG.

Michele Cornacchia

WP7 co-leader

Michele Cornacchia, degree in Physics, had leading work experience in multivariate statistics techniques applied as well to the automatic speech recognition. Since 1986 he joined the Fondazione Ugo Bordoni concerning with voice interaction, human factors and usability. On 1992 he received his Master’s degree and Specialisation in Organisational Science and at later stage he was involved in mediated communication and telework studies. He attended a number of national and international activities, i.e. socio-organisational survey committed by the Italian Communication Ministry, COST actions and the EU funded project RACE 1065 Issue. Later he took part at “Monitoring Italian e.m. fields” (2006) committed by Economic Development Ministry and as well as he coordinated (2004-2006) the final trial of ePerSpace (EU FP6, IST-506775), dedicated to develop integrated services in the home area.
Moreover about modelling and assessing by end-user, he was (2011-2013) as well technical coordinator of the final trial in EU Pandora Project (FP7-ICT-2007-1-225387) on advanced training environment for crisis scenarios. As for acceptance models and use behavioural prediction, he contributed to a number of innovative studies, i.e. the Smart Life in ICT-Wireless area, the EasyReach project (2012-2014) to foster social interaction of less educated people (FP7, AAL-JP), and the microsimulation of T-government services to reduce the digital divide of classes of not experienced citizens.
At present he concerns with issues of user modelling, behavioural change, and user experience in different ICT application areas. He is author/contributor of many scientific papers and books.