A multi-objective optimization model for the problem of gesture segmentation is formulated, and a method of solving the model based on a two-phase estimation of distribution algorithm is presented. When building the model, the positions of a series of pixels are taken as the decision variable, and the differences between the colors of pixels and those of a hand are taken as objective functions. A method of gesture segmentation based on a two-phase estimation of distribution algorithm is proposed according to the correlation among the positions of pixels. The method divides the solution of the problem based on evolutionary optimization into two phases, and uses different estimation of distribution algorithms in different phases. In the first phase, the probability model of candidates is formulated by a number of intervals given the fact that the positions of hand pixels distribute in several intervals. In the second phase, the probability model of candidates is built through a series of segments since the positions of hand pixels further distribute around curves. A series of pixels constituting a hand region are obtained based on sampling by the above probability models. The proposed method is applied to 2515 problems of gesture segmentation, and is compared with the existing methods. The experimental results demonstrate the effectiveness of the proposed method.