20 years ago, when I was a young and inexperienced engineering student (and long enough ago that I don't have any photos of this particular tale of woe), we did a group project to build a robot that would move around a small space following lines on the floor and carry drink cans around.

Two took on the mechanical design of the robot, two the electronics, and another student and I were responsible for the software. The electronics pair were given sensors that would allow the robot to detect the lines (white electrical tape against a green surface).

The robot was built, and we started testing using a strip of test track. The robot moved, the sensors did their thing, and the software detected when the robot was veering off course and corrected. Huzzah! We set about coding and testing the mechanism to lift and lower the drinks cans, and tested that too. All was well. Birds sang, lambs frolicked, students struggled to get up in the morning.

On the day when the robot had to complete the task we finally had access to the small work area (maybe 2 metres on a side) that the robot would have to navigate around. We put our robot on the board and gave it a go.

It didn't follow the lines.

We hurriedly took the robot back to our workbench and looked it over. We grabbed the test track. The robot followed the lines. Relieved, we put it back on the trial space.

It didn't follow the lines.

At this point, we were running out of time before the trials began. While the other pairs tried to find problems on their end, we software devs threw together a backup plan — we ripped out all the code to do with the sensors and went with a dead-reckoning approach. The robot would just assume it was always in the right place, travelling in straight lines and turning by the right amount, and try to complete the task with no feedback. This was doomed to failure, which we knew, but we needed to demonstrate something, and this was all we had. (Even if we'd got it perfectly lined up at the start, this would never work. The robot was powered by 2 DC motors, which would never have identical characteristics, so any attempt to make the robot move in a straight line would always result in a slight curve.)

We sheepishly brought our robot to the trial area, explained the problem, and let it do its thing. It was a disaster, and we didn't score well. It was only afterwards that we finally realized the problem. The test track was a strip of board about 8 inches wide, and a few millimetres thick. As it happened, our mechanical team had made a robot that was a bit wider than most. The wheels, therefore, were either side of the piece of test track. This meant that when we were using the test track, the lines were a few millimetres closer to the sensors than they were in the real work area. When we tried to use the robot in the "real world", the sensors were too far away to see anything.

All our testing had given us confidence that our robot would work, but the tests were worthless because we weren't testing the right thing. Never assume that your software (or robot) is fine simply because the tests pass — the tests themselves need to be evaluated.