A visual sensor which could recognize fruits was made in order to develop the sight sense of the fruit harvesting robot. This visual sensor was attached to an anthropomorphic type manipulator, and a fruit detecting experiment using a method of visual feedback was done. The results were obtained as follows ; It was observed that the hand was able to grip the orange balls which positioned at 30 cm in front of the visual sensor. It was given as defects that the error depended on the fruit diameter arose, number of the image input was big, and so on. It was considered that this method was more advantageous to telescopic type manipulator than anthropomorphic type manipulator.