Open questions in evaluation
Why good measurements are hard, and how we deal with that.
Good evaluations are one of the hardest tasks in building intelligent systems.
What we measure
We look not only at individual tasks but also at how reliably a system works across many situations.
What remains open
Many questions are still unresolved. We will share our experience as soon as it is solid.