The IRS 13E complex is an unusual concentration of massive, early-type stars at a projected distance of 0.13 pc from the Milky Way's central supermassive black hole Sagittarius A* (Sgr A*). Because of their similar proper motion and their common nature as massive, young stars it has recently been suggested that IRS 13E may be the remnant of a massive stellar cluster containing an intermediate-mass black hole (IMBH) that binds its members gravitationally in the tidal field of Sgr A*. Here, we present an analysis of the proper motions in the IRS 13E environment that combines the currently best available data with a time line of 10 years Based on observations at the Very Large Telescope (VLT) of the European Southern Observatory (ESO) on Paranal in Chile. We find that an IMBH in IRS 13E must have a minimum mass of 104 \solm in order to bind the source complex gravitationally. This high mass limit in combination with the absence so far of compelling evidence for a non-thermal radio and X-ray source in IRS 13E make it appear unlikely that an IMBH exists in IRS 13E that is sufficiently massive to bind the system gravitationally.