In my industrial robotics course this past term at WPI my team and I tried to implement an object-following industrial arm using OpenCV, ROS-I, and an ABB IRB1600 arm. The premise of the work came from our MQP, a senior design and fabrication project, where we intend to have a robot ‘observe’ invasive lionfish. To demonstrate progress on the vision software and communication protocols, we were advised to develop a testing platform for an interim robot. While the project was not entirely successful (we did not get it running on the physical arm) the simulation held up very well. A video can be seen below of object tracking in RVIZ.
The first step the team took was by creating EOAT, end of arm tooling, for the express purpose of stereo visual recognition. Figuring that a target will have an x, y, and z location relative to the EOAT, such a setup was deemed necessary. Below can be seen the prototype. Special attention can be paid to the way the mounting hardware is set into the plastic mounting plate. This was accomplished by heating up a nut with a torch and carefully setting it into a press-fit tolerance hole.
Afterwards, the team went about the ugly and long process of training a visual recognition program. We used TensorFlow for distinguishing lionfish from non-lionfish. The program took the whole term to write (due to other classes and commitments) and was unfortunately too slow to be run for any worthwhile result. And when it came time to try connecting the ROS master to the ABB arm, we encountered communication errors that not even the graduate students (also trying to use ROS) could solve. Which meant we had to settle at the moment for a simulation. Eventually I hope to start this project up again when I have more skills and time.