We have studied the correlation between 2357 Chandra X-ray point sources in a 40 * 40 parsec field and 20,000 infrared sources we observed in the corresponding subset of our 2\arcdeg * 1.4\arcdeg Spitzer/IRAC Galactic Center Survey at 3.6-8.0 micron , using various spatial and X-ray hardness thresholds. The correlation was determined for source separations of less than 0.''5, 1'' or 2''. Only the soft X-ray sources show any correlation with infrared point sources on these scales, and that correlation is very weak. The upper limit on hard X-ray sources that have infrared counterparts is <1.7% (3 sigma ). However, because of the confusion limit of the IR catalog, we only detect IR sources with absolute magnitudes <= 1. As a result, a stronger correlation with fainter sources cannot be ruled out. Only one compact infrared source, IRS 13, coincides with any of the dozen prominent X-ray emission features in the 3 * 3 parsec region centered on Sgr A*, and the diffuse X-ray and infrared emission around Sgr A* seems to be anti-correlated on a few-arcsecond scale. We compare our results with previous identifications of near-infrared companions to Chandra X-ray sources.