Acquiring everyday manipulation skills through games

Autonomous robots for performing manipulation tasks such as flipping a pancake or fishing a sausage out of the cooking pot require a lot of knowledge that cannot be easily expressed and formalized in knowledge bases and knowledge representation languages. We ask video game players to perform such actions in simulation-based virtual reality environments and acquire such knowledge from the logfiles of the game episodes.The Games with a Purpose dataset contains episodes of humans demonstrating pancake-making in a virtual reality setting. This data is then used for action recognition and parameterization. Knowledge base Publications


In order for future robots to successfully accomplish more and more complex manipulation tasks, they are required to be action-aware and possess naive physics reasoning capabilities. They need to possess a model regarding how the effects of their actions depend on the way they are executed. For example, a robot making pancakes should have an understanding that pouring the pancake-mix on the oven depends on the position and the way the container is held, or that sliding a spatula under a pancake may or may not damage it depending on the angle and the dynamics of the push.


Games Setup

Setup of the virtual environment game

The setup for data collection is depicted in the figure above. The system is provided with a sensor infrastructure that allows interaction with the virtual world by tracking the players’ hand motion and mapping them onto the robotic hand in the game. We tested out two different setups for the tracking. One with the help of a magnetic sensor based controller, which returns the position and orientation of the hand, together with a dataglove for finger joints positioning. In the second case we used two 3D camera sensors mounted on a frame which yields the pose and the skeleton of the tracked hand.

Description of the data

This dataset contains the following data:

  • Semantic memory (represented using Web Ontology Language, OWL) of all performed actions, and their parametrizations and outcomes
  • Physical pose and transformation data of all the objects in the virtual world


Partly supported by the EU FP7 RoboHow project


  • RoboHow Logo