We propose a simple test for the existence of a cluster of black hole remnants around Sgr A* that is based on a small sample of any type of Galactic Center objects, provided they are substantially less massive than the black holes and constitute part of an old (> 1 Gyr) population. The test relies on the fact that, under the presence of such a cluster of heavy remnants and because of energy equipartition, lower mass objects would be expelled from the central regions and settle into a distribution very different than the cusp expected to be induced by the supermassive black hole alone. We show that with a sample of just 50 objects and using only their angular positions on the sky relative to Sgr A* it is possible to clearly differentiate between a distribution consistent with the presence of the cluster of black holes and a power-law cusp distribution. We argue that millisecond pulsars might currently be the best candidate to perform this test, because of the large uncertainties involved in the age determination of less exotic objects. In addition, by measuring their first and second period derivatives, millisecond pulsars offer the rare opportunity of determining the complete phase space information of the objects. We show that this extra information improves the detection of any mass segregation by about 30%.