Inclusions in biological membranes may interact via deformations they induce on the shape of that very membrane. Such deformations are a purely physical effect, resulting in nonspecific forces between the inclusions. In this Letter we show that this type of interaction can organize membrane domains and hence may play an important biological role. Using a simple analytical model we predict that membrane inclusions sort according to the curvature they impose. We verify this prediction by both numerical simulations and experimental observations of membrane domains in phase separated vesicles.