Best rank-k approximations for tensors: generalizing Eckart–Young

Given a tensor f in a Euclidean tensor space, we are interested in the critical points of the distance function from f to the set of tensors of rank at most k, which we call the critical rank-at-most-k tensors for f. When f is a matrix, the critical rank-one matrices for f correspond to the singular pairs of f. The critical rank-one tensors for f lie in a linear subspace H_f, the critical space of f. Our main result is that, for any k, the critical rank-at-most-k tensors for a sufficiently general f also lie in the critical space H_f. This is the part of Eckart–Young Theorem that generalizes from matrices to tensors. Moreover, we show that when the tensor format satisfies the triangle inequalities, the critical space H_f is spanned by the complex critical rank-one tensors. Since f itself belongs to H_f, we deduce that also f itself is a linear combination of its critical rank-one tensors.