Measuring progress in robotics: benchmarking and the ‘measure-target confusion’

Onderzoeksoutput: Hoofdstuk in Boek/Rapport/CongresprocedureHoofdstukAcademicpeer review

Uittreksel

While it is often said that in order to qualify as a true science robotics should aspire to reproducible and measurable results that allow benchmarking, I argue that a focus on benchmarking will be a hindrance for progress. Several academic disciplines that have been led into pursuing only reproducible and measurable ‘scientific’ results—robotics should be careful not to fall into that trap. Results that can be benchmarked must be specific and context-dependent, but robotics targets whole complex systems independently of a specific context—so working towards progress on the technical measure risks missing that target. It would constitute aiming for the measure rather than the target: what I call ‘measure-target confusion’. The role of benchmarking in robotics shows that the more general problem to measure progress towards more intelligent machines will not be solved by technical benchmarks; we need a balanced approach with technical benchmarks, real-life testing and qualitative judgment.

Originele taal-2Engels
TitelMetrics of Sensory Motor Coordination and Integration in Robots and Animals
SubtitelHow to Measure the Success of Bioinspired Solutions with Respect to their Natural Models, and Against More ‘Artificial’ Solutions?
RedacteurenFabio Bonsignorio, Elena Messina, Angel P. del Pobil, John Hallam
Plaats van productieCham
UitgeverijSpringer
Pagina's169-179
Aantal pagina's11
ISBN van elektronische versie978-3-030-14126-4
ISBN van geprinte versie978-3-030-14124-0
DOI's
StatusGepubliceerd - 1 jan 2020

Publicatie series

NaamCognitive Systems Monographs
Volume36
ISSN van geprinte versie1867-4925
ISSN van elektronische versie1867-4933

Vingerafdruk

Benchmarking
Robotics
Large scale systems
Testing

Citeer dit

Müller, V. C. (2020). Measuring progress in robotics: benchmarking and the ‘measure-target confusion’. In F. Bonsignorio, E. Messina, A. P. del Pobil, & J. Hallam (editors), Metrics of Sensory Motor Coordination and Integration in Robots and Animals: How to Measure the Success of Bioinspired Solutions with Respect to their Natural Models, and Against More ‘Artificial’ Solutions? (blz. 169-179). (Cognitive Systems Monographs; Vol. 36). Cham: Springer. https://doi.org/10.1007/978-3-030-14126-4_9
Müller, Vincent C. / Measuring progress in robotics : benchmarking and the ‘measure-target confusion’. Metrics of Sensory Motor Coordination and Integration in Robots and Animals: How to Measure the Success of Bioinspired Solutions with Respect to their Natural Models, and Against More ‘Artificial’ Solutions?. redacteur / Fabio Bonsignorio ; Elena Messina ; Angel P. del Pobil ; John Hallam. Cham : Springer, 2020. blz. 169-179 (Cognitive Systems Monographs).
@inbook{455802cbbbd048e68b729d20440e2d8c,
title = "Measuring progress in robotics: benchmarking and the ‘measure-target confusion’",
abstract = "While it is often said that in order to qualify as a true science robotics should aspire to reproducible and measurable results that allow benchmarking, I argue that a focus on benchmarking will be a hindrance for progress. Several academic disciplines that have been led into pursuing only reproducible and measurable ‘scientific’ results—robotics should be careful not to fall into that trap. Results that can be benchmarked must be specific and context-dependent, but robotics targets whole complex systems independently of a specific context—so working towards progress on the technical measure risks missing that target. It would constitute aiming for the measure rather than the target: what I call ‘measure-target confusion’. The role of benchmarking in robotics shows that the more general problem to measure progress towards more intelligent machines will not be solved by technical benchmarks; we need a balanced approach with technical benchmarks, real-life testing and qualitative judgment.",
author = "M{\"u}ller, {Vincent C.}",
year = "2020",
month = "1",
day = "1",
doi = "10.1007/978-3-030-14126-4_9",
language = "English",
isbn = "978-3-030-14124-0",
series = "Cognitive Systems Monographs",
publisher = "Springer",
pages = "169--179",
editor = "Fabio Bonsignorio and Elena Messina and {del Pobil}, {Angel P.} and John Hallam",
booktitle = "Metrics of Sensory Motor Coordination and Integration in Robots and Animals",
address = "Germany",

}

Müller, VC 2020, Measuring progress in robotics: benchmarking and the ‘measure-target confusion’. in F Bonsignorio, E Messina, AP del Pobil & J Hallam (redactie), Metrics of Sensory Motor Coordination and Integration in Robots and Animals: How to Measure the Success of Bioinspired Solutions with Respect to their Natural Models, and Against More ‘Artificial’ Solutions?. Cognitive Systems Monographs, vol. 36, Springer, Cham, blz. 169-179. https://doi.org/10.1007/978-3-030-14126-4_9

Measuring progress in robotics : benchmarking and the ‘measure-target confusion’. / Müller, Vincent C.

Metrics of Sensory Motor Coordination and Integration in Robots and Animals: How to Measure the Success of Bioinspired Solutions with Respect to their Natural Models, and Against More ‘Artificial’ Solutions?. redactie / Fabio Bonsignorio; Elena Messina; Angel P. del Pobil; John Hallam. Cham : Springer, 2020. blz. 169-179 (Cognitive Systems Monographs; Vol. 36).

Onderzoeksoutput: Hoofdstuk in Boek/Rapport/CongresprocedureHoofdstukAcademicpeer review

TY - CHAP

T1 - Measuring progress in robotics

T2 - benchmarking and the ‘measure-target confusion’

AU - Müller, Vincent C.

PY - 2020/1/1

Y1 - 2020/1/1

N2 - While it is often said that in order to qualify as a true science robotics should aspire to reproducible and measurable results that allow benchmarking, I argue that a focus on benchmarking will be a hindrance for progress. Several academic disciplines that have been led into pursuing only reproducible and measurable ‘scientific’ results—robotics should be careful not to fall into that trap. Results that can be benchmarked must be specific and context-dependent, but robotics targets whole complex systems independently of a specific context—so working towards progress on the technical measure risks missing that target. It would constitute aiming for the measure rather than the target: what I call ‘measure-target confusion’. The role of benchmarking in robotics shows that the more general problem to measure progress towards more intelligent machines will not be solved by technical benchmarks; we need a balanced approach with technical benchmarks, real-life testing and qualitative judgment.

AB - While it is often said that in order to qualify as a true science robotics should aspire to reproducible and measurable results that allow benchmarking, I argue that a focus on benchmarking will be a hindrance for progress. Several academic disciplines that have been led into pursuing only reproducible and measurable ‘scientific’ results—robotics should be careful not to fall into that trap. Results that can be benchmarked must be specific and context-dependent, but robotics targets whole complex systems independently of a specific context—so working towards progress on the technical measure risks missing that target. It would constitute aiming for the measure rather than the target: what I call ‘measure-target confusion’. The role of benchmarking in robotics shows that the more general problem to measure progress towards more intelligent machines will not be solved by technical benchmarks; we need a balanced approach with technical benchmarks, real-life testing and qualitative judgment.

UR - http://www.scopus.com/inward/record.url?scp=85064614002&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-14126-4_9

DO - 10.1007/978-3-030-14126-4_9

M3 - Chapter

AN - SCOPUS:85064614002

SN - 978-3-030-14124-0

T3 - Cognitive Systems Monographs

SP - 169

EP - 179

BT - Metrics of Sensory Motor Coordination and Integration in Robots and Animals

A2 - Bonsignorio, Fabio

A2 - Messina, Elena

A2 - del Pobil, Angel P.

A2 - Hallam, John

PB - Springer

CY - Cham

ER -

Müller VC. Measuring progress in robotics: benchmarking and the ‘measure-target confusion’. In Bonsignorio F, Messina E, del Pobil AP, Hallam J, redacteurs, Metrics of Sensory Motor Coordination and Integration in Robots and Animals: How to Measure the Success of Bioinspired Solutions with Respect to their Natural Models, and Against More ‘Artificial’ Solutions?. Cham: Springer. 2020. blz. 169-179. (Cognitive Systems Monographs). https://doi.org/10.1007/978-3-030-14126-4_9