The Blues had taken the lead through Sergio Aguero midway through the second period but despite hitting the woodwork twice in five minutes through Yaya Toure and Fernandinho, the Gunners levelled in normal time and then won it in extra time to end hopes of bringing silverware back to the Etihad this season.
The Blues started as they meant to go on, pinning a defensive-minded Arsenal back into their own half for long periods.
The Gunners, adopting the unfamiliar 3-5-2 formation they used in last Monday’s win over Middlesbrough, sat back and allowed City plenty of possession while looking to counter-attack if the opportunity arose.
Gabriel’s over-zealous tackle on David Silva went unpunished and the Spaniard limped off injured as a result midway through the opening period.
Indeed, there were several challenges that angered Guardiola as the Gunners continued their 'back to basics' approach.
Opportunities in front of goal were few and far between save for a Silva header that was tipped over by Petr Cech, a Raheem Sterling volley from 20 yards that flew a few feet wide and a disallowed Sterling effort shortly before the break after the assistant referee judged Leroy Sane’s cross to have curled out of play.
Arsenal were much brighter after the re-start and began to press City back. There were one or two nervy moments and last-ditch clearances as the balance of play started to shift, but as the Gunners committed more men forward, so the gaps started to appear at the back and it was from one such break that the Blues finally broke the deadlock.
Yaya Toure collected the ball just outside his own box and spotted Aguero's sprint, playing him clear of Rob Holding and through on goal - he looked to have pushed the ball slightly too far ahead but as Cech raced out the Argentine gently over the keeper to put City ahead.
It was his 30th goal of another fantastic campaign but it wasn't to be the goal that settled this tie as nine minutes later, Arsenal drew level through Nacho Monreal's close-range finish.
Then, on 79 minutes, with both teams going for broke, the ball fell to Yaya Toure on the edge of the box and his acrobatic volley was superbly touched on to the post by Cech, much to the disbelief of the Ivorian.
Three minutes later thumping header smacked the crossbar with Cech beaten as the game reached a frenetic climax.
Then it was hearts-in-the-mouth time for City fans as Danny Welbeck wriggled into the box before flashing a shot across goal and off the far post.
With tiring legs in both sides, the last few minutes could have seen either side snatch a winner, but it would be extra time that was needed to separate the sides - apt considering the official London Marathon had taken place just a few hours earlier!
The Blues fell behind with 101 minutes on the clock after Alexis Sanchez poked home from close range to end this season's FA Cup adventure, cursing the Wembley woodwork and wondering what might have been.
The decision to disallow City’s first-half ‘goal’ will increase the demand for video technology to decide borderline calls.
Did the whole ball go out of play? Probably not but even with the benefit of several TV replays, it was difficult to say either way. A genuinely game changing moment.
Pep Guardiola was impressed enough with the team that beat Southampton last weekend to keep the same starting XII. That meant Claudio Bravo retaining his place but no Gabriel Jesus or John Stones.
Skipper Vincent Kompany was starting successive games for the first time since October.
City Matchday App Star Man: Sergio Aguero
The big games keep coming with the Manchester derby next on the agenda this coming Thursday with the battle for a Champions League spot reaching boiling point.
United’s win at Burnley means a defeat could see the Blues drop to fifth – while a win would re-establish some breathing space for Guardiola’s men.
Then a trip to the Riverside next Sunday to face a Middlesbrough side who could be relegated if they are beaten.