Continuum low-frequency radio observations of the Galactic Center reveal the presence of two prominent radio sources, Sgr A East and its surrounding Halo, containing non-thermal particle distributions with power-law indices ~ 2.5-3.3 and ~ 2.4, respectively. The central 1-2 pc region is also a source of intense (stellar) UV and (dust-reprocessed) far-IR radiation that bathes these extended synchrotron-emitting structures. A recent detection of gamma -rays (2EGJ1746-2852) from within ~ 1o of the Galactic Center by EGRET onboard the Compton GRO shows that the emission from this environment extends to very high energies. We suggest that inverse Compton scatterings between the power-law electrons inferred from the radio properties of Sgr A East and its Halo, and the UV and IR photons from the nucleus, may account for the possibly diffuse gamma -ray source as well. We show that both particle distributions may be contributing to the gamma -ray emission, though their relevant strength depends on the actual physical properties (such as the magnetic field intensity) in each source. If this picture is correct, the high-energy source at the Galactic Center is extended over several arcminutes, which can be tested with the next generation of gamma -ray and hard X-ray missions.