Motivated by recent observations that suggest a low density of old stars around the Milky Way supermassive black hole, evolutionary models for the nuclear star cluster are considered that postulate a parsec-scale core as initial conditions. Gravitational encounters cause the core to shrink; a core of initial radius 1-1.5 pc evolves to a size of 0.5 pc after 10 Gyr, roughly the size of the observed core. The absence of a Bahcall-Wolf cusp is naturally explained. In these models, the time for a 10\msun black hole to spiral in to the Galactic center from an initial distance of 5 pc can be much greater than 10 Gyr. Assuming that the stellar black holes had the same phase-space distribution initially as the stars, their density after 5-10 Gyr is predicted to rise very steeply going into the stellar core, but to remain substantially below the densities inferred from steady-state models that include a steep density cusp in the stars. The implications of these models are discussed for the rates of gravitational-wave inspiral events and of other physical processes that depend on a high density of stars or stellar mass black holes near \sgr.