Gundogan converted from the spot in the 33rd minute but a double salvo from Japanese substitutes Ritsu Doan and Takuma Asano turned the score around in just eight minutes.
Our No.8, wearing No.21 for Hansi Flick's side, had his opportunity when David Raum was felled by Japanese goalkeeper Shuichi Gonda just after the half hour mark.
After grabbing the ball, Gundogan stepped forward and sent his shot down the middle as Gonda dived to his left.
Japan had a goal disallowed before that strike but outside of that, Germany controlled much of the first half.
Gundogan had a chance to double his side's lead, hitting the post shortly after the break before his eventual substitution in the 67th minute.
Japan rallied as the second period wore on and eventually equalised in the 75th minute through Freiburg winger Doan, who followed up as Manuel Neuer parried to the edge of his six-yard box.
The winner came in the 83rd minute when VfL Bochum forward Asano fired a shot from a tight angle high into Neuer's net.
City's players have had a mixed group stage so far. John Stones started and Phil Foden and Jack Grealish came off the bench with the latter grabbing the final goal of the game as England beat Iran 6-2 on Monday. Nathan Ake played the full match for Netherlands as they defeated Senegal 2-0.
However, Julian Alvarez was a second half substitute as Argentina suffered a shock 2-1 defeat at the hands of Saudi Arabia.
We'll have reports of every single City player's involvement at this tournament here on mancity.com and our official app.