The intense 0.511 MeV gamma-ray line emission from the Galactic Center observed by INTEGRAL requires a large annihilation rate of nonrelativistic positrons. If these positrons are injected at even mildly relativistic energies, higher-energy gamma rays will also be produced. We calculate the gamma-ray spectrum due to inflight annihilation and compare to the observed diffuse Galactic gamma-ray data. Even in a simplified but conservative treatment, we find that the positron injection energies must be <=sssim 3 MeV, which strongly constrains models for Galactic positron production.