Future quantum technologies will require the implementation of complex quantum communication (QC) networks. Temporal modes (TMs) provide an appealing high-dimensional encoding alphabet based on the time-frequency degree of freedom of photons, leading to important advantages for QC applications. A TM-based QC scheme requires the simultaneous detection of multiple TMs of single photons, which has not yet been achieved. In this work, we demonstrate high-dimensional single-photon TM decoding with a multi-output quantum pulse gate (mQPG). The mQPG is a device that provides simultaneous projection of multiple TMs onto all the elements of a chosen alphabet (or their superpositions) and maps each result onto a different output frequency. We demonstrate that the mQPG is compatible with single-photon-level input states from a full set of five-dimensional mutually unbiased bases, and we characterize its performance through a quantum measurement tomography. We then proceed to demonstrate a proof-of-principle decoder for high-dimensional quantum key distribution based on the mQPG.