Overview

In the scope of this project I've extended the functionalities of the pre-existing Voice Jogger, a software set used to command a Franka Emika Panda robotic arm via voice commands. The updated software contains simple low-level commands for short term commanding of the robot arm, as well as high-level commands that execute a chain of subsequent motions. The functionalities of the voice jogger have been tested in two assembly tasks, one of which also included significant collaboration from a human operator. In addition, some other experiments were used to test the capabilities of the commanding to execute repeated tasks, including planar motions.

Code structure

The original code was designed to be used by an Android application, using the microphone of the mobile phone. For the ease of testing, I have used a microphone directly connected to the computer, and all tests were done with such a setup.

The server part of the Voice Jogger consists of three important Python files: microphone_input.py, commandCreator.py and robotMover.py. microphone_input.py is responsible for handling audio input from the microphone, and publishing commands on the "/text_commands" ROS topic. The actual command messages are created by commandCreator.py, which is called by microphone_input.py. commandCreator.py uses the recognized words as input, and creates the command to be sent on the ROS topic. This file has a list of the relevant command words to be detected, and uses a simple decision-tree-like heuristic to fetch the actual command from the recognized input words.

robotMover.py is the file that deals with the actual movements of the robots. This file uses the command on the "/test_commands" topic as its input, and matches the commands to robot actions. This includes using the Python interface of MoveIt to control lower-level, cartesian movements of the robot arm, and constructing high-level tasks of the robot from the low-level components. As this file handles robot actions, all physical functionality is defined in this file.

New commands

The following list contains the commands newly added to the voice-jogger repository.

• Repetition commands:
  • REPEAT [# of times] TIMES [TASKNAME]: Plays the pre-recorded task with the name specified in [TASKNAME] # times.
  • JOG [DIRECTION] [# of times] TIMES [TASKNAME]: Plays the pre-recorded task with the name specified in [TASKNAME] # times, but with each repetition, the starting position is offset in [DIRECTION] by the pre-set step size.
• Pick-place commands:
  • PICK [position name]: Perform a "pick" task at the specified position, i.e. move the gripper above the given position, open the gripper, move down to the position, close the gripper and move back up.
  • PLACE [position name]: Perform a "place" task at the specified position, i.e. move the gripper above the given position, move down to the position, open the gripper and move back up.
  • OFFSET [position name] [direction] [distance]: Similar to PLACE, but the target position for placing is offset in the given direction by the given distance. Direction can be left/right/forward/backward.
  • STACK [position name] DISTANCE [distance]: Similar to PLACE, but the target position for placing the object is above the given position by the distance specified in [distance].
  • HOLD [position name] DISTANCE [distance]: Similar to STACK, but it does not actually place the object at the position, but holds it at a distance above instead. Basically it moves the robot to a position [distance] above [position name].

A list of commands, incuding the pre-existing ones can be found in the README.md file of the repository.

Usage

The voice jogger can be used either in simulation or on a real-world robot. To run the code, open three terminal windows, and run the following commands:

1. In the first terminal, run the following:
   • For simulation:
     roslaunch panda_moveit_config demo.launch
   • For real robot:
     roslaunch panda_moveit_config franka_control.launch robot_ip:=130.230.36.115 load_gripper:=True

     Make sure to insert the actual IP-address of the robot in the command.

2. In the second terminal:
   • For using the Android app:
     python3 main.py
   • For using the computer's microphone:
     python3 microphone_input.py
   • For using terminal input to test robotMover.py:
     python3 ros_message_transmitter_for_testing_robotMover.py

   Do note, however, that the during the experiments the voice jogger has only been tested using the computer's microphone. There has been no changes to any of these files, but stable working can not be guaranteed with the other two scripts.

3. In the third terminal:
   python3 robotMover.py

Experiments

A couple of experiments were performed to test and demonstrate the capabilities and limitations of this voice jogging approach. Video recordings of some of the experiments can be found below.

Assembly tasks

Helical Gears assembly

In this experiment, voice commanding was used to assemble a helical gearset. The parts of the helical gears were 3D-printed and their models are available here. This assembly task includes the usage of the PICK, PLACE and STACK commands.

Planetary Gearhead assembly

In this experiment, voice commanding was used to assemble a planetary gearhead, showing collaboration between the robot and the human operator. The parts of the gearhead were 3D-printed and their models are available here. This task also shows the PICK, PLACE and STACK commands, and the HOLD command as well.

Planar motion with repetition

In this experiment, a single execution of a planar task was recorded, then executed while changing its starting position with each repetition using the JOG command.

Vertical motion with repetition

In this experiment, a single execution of a vertical task was recorded, then executed while changing its starting position with each repetition using the JOG command.

Circular motion with repetition

In this experiment, a task including circular motion was recorded, then executed while changing its starting position with each repetition. This experiment shows the CIRCLE and the JOG commands, as well as the ability to change the offset distance of the starting position using the step size setting.

Acknowledgements

This website is based on the academic website of Jon Barron, the source code of which can be found here.

The original Voice Jogger repository that served as the starting point for this project was developed by Ara Jo, Mikko Kulju, Niklas Sorri and Omar Hassan, and is available on GitHub.