Visual recognition in multi-robot systems has a peculiar problem in that an observation from a different viewpoint gives a different perspective. Also, a representation of the same target object is highly affected by such a viewpoint or environmental condition. Hence it has become a challenge to realize cognitive sharing of the object or ROI between the robots without a help of artificial markers. In this paper, we propose consensus-making algorithms to acquire the viewpoint-invariant representation of the geometrical relations. Experimental results show the ability of cognitive sharing significantly improved by the proposed method.