Demo deadline:

final demo: 7.3., test demo 29.2.

Demo description

We want to show how to command robot for more complex tasks (e.g., put the cube to the drawer) that are composed of some sequence of already known actions. This task might need high level reactive control (e.g., open the drawer, pick up the falling cube, etc.). Command should be possible to give by either gestures, language or their combination.

Purpose of the demo

This demo should help to make results for Petr's paper testing merging of the modalities, also for testing Gabina's HTVAE approach and ability to use MVAEs for longer sequences of actions.

Subtasks to be solved

• detailed specification of the task and the objects involved @stepakar https://gitlab.ciirc.cvut.cz/imitrob/common-imitrob-setup-hw-sw/imitrob-setup/-/issues/8
• specify how to represent actions and tasks (within templates) so these can be used also for later teaching of novel tasks, easily send around and unpacked for execution by the robot @skovirad @stepakar and others https://gitlab.ciirc.cvut.cz/imitrob/common-imitrob-setup-hw-sw/imitrob-templates/-/issues/1, https://gitlab.ciirc.cvut.cz/imitrob/mirracle/mirracle_wiki/-/issues/98
• test imitrob templates directly from modality merger input
• integrate and test NL node for specifying commands @stepakar @vancpetr https://gitlab.ciirc.cvut.cz/imitrob/common-imitrob-setup-hw-sw/imitrob-hri/-/issues/23, https://gitlab.ciirc.cvut.cz/imitrob/common-imitrob-setup-hw-sw/imitrob-hri/-/issues/24, https://gitlab.ciirc.cvut.cz/imitrob/common-imitrob-setup-hw-sw/imitrob-hri/-/issues/22, https://gitlab.ciirc.cvut.cz/imitrob/common-imitrob-setup-hw-sw/imitrob-hri/-/issues/21
• NL node - microphone input doesn't work yet
• integration of behavior trees or other reactive robot control module @vancpetr @behrejan https://gitlab.ciirc.cvut.cz/imitrob/common-imitrob-setup-hw-sw/imitrob_robot_server/-/issues/4, https://gitlab.ciirc.cvut.cz/imitrob/mirracle/mirracle_wiki/-/issues/102
• try RL module instead of MVAE module @vavremic @sejnogab https://gitlab.ciirc.cvut.cz/imitrob/common-imitrob-setup-hw-sw/imitrob-mapping/imitrob-mvae/-/issues/1
• try how to deal with more complex tasks within MVAE module (sending one-by-one actions, or their sequence in one go?) @sejnogab https://gitlab.ciirc.cvut.cz/imitrob/common-imitrob-setup-hw-sw/imitrob-mapping/imitrob-mvae/-/issues/2
• calibration of the setup (at least cameras) @behrejan @skovirad https://gitlab.ciirc.cvut.cz/imitrob/common-imitrob-setup-hw-sw/imitrob-calib/-/issues/2
• possible extensions to ontology - https://gitlab.ciirc.cvut.cz/imitrob/common-imitrob-setup-hw-sw/imitrob-hri/-/issues/19
• Rules for the robot - https://gitlab.ciirc.cvut.cz/imitrob/mirracle/mirracle_wiki/-/issues/97
• Usage of computer and responsible people for computers/robots - https://gitlab.ciirc.cvut.cz/imitrob/mirracle/mirracle_wiki/-/issues/160

Corresponding papers to be prepared:

• Paper: Sensor benchmarking @krsek https://gitlab.ciirc.cvut.cz/imitrob/mirracle/mirracle_wiki/-/issues/119
• Paper: Ontological HRI paper @skovirad @stepakar https://gitlab.ciirc.cvut.cz/imitrob/mirracle/mirracle_wiki/-/issues/110
• Paper: HTVAE to TCDS @sejnogab https://gitlab.ciirc.cvut.cz/imitrob/mirracle/mirracle_wiki/-/issues/176
• Paper: Multimodal mapping paper to RAL/IROS @vancpetr https://gitlab.ciirc.cvut.cz/imitrob/mirracle/mirracle_wiki/-/issues/177