There are several lines of evidence that the super-massive black hole at the Galactic center had higher activities in the past than directly observed at present. Here I show that these lines of evidence can quantitatively and consistently be explained if the mean accretion rate during the past 107 yrs has been 103-4 times higher than the current rate, by the picture of radiatively inefficient accretion flow (RIAF) and associated outflow that has been successfully applied to Sgr A^*. I argue that this increased rate and its duration are theoretically reasonable in the Galactic center environment, while the accretion rate suddenly dropped about 300 years ago most likely because of the shell passage of the supernova remnant Sgr A East. Then I show that a significant amount of positrons should have been created around the event horizon during the higher activity phase, and injected into interstellar medium by the outflow. The predicted positron production rate and propagation distance are close to those required to explain the observed 511 keV annihilation line emission from the Galactic bulge, giving a natural explanation for the large bulge-to-disk ratio of the emission. The expected injection energy is MeV, which is also favorable as an explanation of the 511 keV line emission.