We report the first time-series of broadband infrared color measurements of Sgr A*, the variable emission source associated with the supermassive black hole at the Galactic Center. Using the laser and natural guide star adaptive optics systems on the Keck II telescope, we imaged Sgr A* in multiple near-infrared broadband filters with a typical cycle time of 3 min during 4 observing runs (2005-2006), two of which were simultaneous with Chandra X-ray measurements. In spite of the large range of dereddened flux densities for Sgr A* (2 to 30 mJy), all of our near-infrared measurements are consistent with a constant spectral index of alpha = -0.6 +/- 0.2 (F_ nu \propto nu alpha ). Furthermore, this value is consistent with the spectral indices observed at X-ray wavelengths during nearly all outbursts; which is consistent with the synchrotron self-Compton model for the production of the X-ray emission. During the coordinated observations, one infrared outburst occurs <=36 min after a possibly associated X-ray outburst, while several similar infrared outbursts show no elevated X-ray emission. A variable X-ray to IR ratio and constant infrared spectral index challenge the notion that the infrared and X-ray emission are connected to the same electrons. We, therefore, posit that the population of electrons responsible for both the IR and X-ray emission are generated by an acceleration mechanism that leaves the bulk of the electron energy distribution responsible for the infrared emission unchanged, but has a variable high-energy cutoff. Occasionally a tail of electrons > 1 GeV is generated, and it is this high-energy tail that gives rise to the X-ray outbursts. One possible explanation for this type of variation is from the turbulence induced by a magnetorotational instability, in which the outer scale length of the turbulence varies and changes the high-energy cutoff.