Aro Muric was the shoot-out hero as City gained two points in their group stage opener at the New York Stadium.
The keeper – who was outstanding throughout – twice denied Rotherham from 12 yards as Lukas Nmecha, Matt Smith, Taylor Richards and Charlie Oliver converted to give the Blues a bonus point win.
City had fought back to take the game to penalties – after 90 minutes under competition rules – as substitute Benjamin Garre equalised in the final minute of stoppage time to cancel out David Ball’s 64th minute strike.
In an evenly contested first half City went close twice through Nmecha.
The captain had the first real chance as he turned sharply and pulled the trigger in the six yard box only to see his shot blocked.
Muric was called into action soon after, denying Ball from point blank range with his legs, as Rotherham looked to make their possession count.
City were threatening on the break – with Smith and Jacob Davenport doing their best to unlock the opposition’s defence and the former did for the best chance of the half, when Nmecha raced clear but fired the resulting shot into the side netting.
After the break Paul Warne’s side came into the ascendancy, with Muric and his defence required to be at their best.
The keeper was at full stretch to deny Jonson Clarke-Harris and Jon Taylor, but ex-City youngster, Ball, found a way through.
Alex Bray won possession deep in the Blues’ half and played a precise through ball, which the forward duly gathered before calmly converting.
The Yorkshire outfit continued to press and might have extended their lead if not for a last ditch block by Smith, but City rallied late on.
Substitute Rabbi Matondo flashed a shot across goal before Garre – deep into added time – found a yard of space in the box and fired home into the bottom corner to level and send the tie to penalties.
Muric continued his fine form to make two impressive saves from Will Vaulks and Taylor and with only Davenport missing for the Blues, City gained a second point for their shoot-out win, in addition to the one awarded for the draw.
What it means
With Bradford and Chesterfield yet to play, City are top of Group F in the northern section of the competition.
A trip to north London is up next for Davies' side, who travel to Arsenal in the Premier League 2 on Monday evening, before they're back at the Etihad Stadium on Friday 25 August to face Leicester City.
The Checkatrade Trophy resumes in October at Bradford's Northern Commercial Stadium on Tuesday 24 October.
How they lined up
Simon Davies made four changes to the starting XI that kicked off the season against Chelsea.
Aro Muric came in for Daniel Grimshaw in goal, with Duhaney, Latibeaudiere, Oliver and Francis in front of him in the back four.
Davenport, Dele-Bashiru and Smith formed a midfield trio and in the three-man forward line, Dilrosun and Lorenzo Gonzalez flanked Nmecha through the middle.