In recent years several hypervelocity stars (HVSs) have been observed in the halo of our Galaxy. Such stars are thought to be ejected through dynamical interactions near the massive black hole (MBH) in the Galactic center. Three scenarios have been suggested for their ejection; binary disruption by a MBH, scattering by inspiraling IMBH and scattering by stellar BHs close to MBH. These scenarios involve different stellar populations in the Galactic center. Here we use observations of the Galactic center stellar population, dynamical and evolutionary arguments to constrain the nature and origin of HVSs. We show that the IMBH inspiral scenario requires too many ( 103) main sequence B stars to exist close to the MBH (<0.01 pc) at the time of inspiral, where current observations show ( 10) such stars. Scattering by SBHs are also not likely to be consistent with the observed population of B stars in the Galactic center, although this scenario can still be compatible with observations under extreme conditions. The binary disruption scenario is still consistent with current observations. In addition we show that due to the conditions close to the MBH most binary star systems are not expected to survive for long in this region. Consequently, unique stellar populations that require long evolution in binaries are not expected to be ejected as HVSs in the BHs scattering mechanisms (this may also be related to to the recently observed asymmetry in the velocity distribution of HVSs). Furthermore, it is shown that recently suggested signatures for HVSs origin such as hypervelocity binaries and slow rotating HVSs may be much weaker than suggested and require large statistics.