Matrix optimization has various applications in finance, statistics, and engineering, etc. In this paper, we derive the Lagrangian dual of the matrix optimization problem with sparse group lasso regularization, and develop an adaptive gradient/semismooth Newton algorithm for this dual. The algorithm adaptively switches between semismooth Newton and gradient descent iterations, relying on the decrease of the residuals or values of the dual objective function. Specifically, the algorithm starts with the gradient iteration and switches to the semismooth Newton iteration when the residual decreases to a given threshold value. If the trial step size for the semismooth Newton iteration has been shrunk several times or the residual does not decrease sufficiently, the algorithm switches back to the gradient iteration and reduces the threshold value for invoking the semismooth Newton iteration. Under some mild conditions, the global convergence of the proposed algorithm is proved. Moreover, local superlinear convergence is achieved under one of two scenarios: either when the constraint nondegeneracy condition is met, or when both the strict complementarity and the local error bound conditions are simultaneously satisfied. Some numerical results on synthetic and real data sets demonstrate the efficiency and robustness of our proposed algorithm.