Vrancken's team were level on aggregate after a 2-2 first-leg draw at Tynecastle but the home side looked the superior outfit from kick-off.
They routinely peppered the Hearts goal and pinned them inside their box for swathes of the first half as the visitors survived multiple scares.
A bobbling ball was eventually cleared from inside the six-yard box and Petter Nosakhare Dahl's whipped cross was not tapped over the line as the home side remained relentless.
As Hearts' heads drifted towards getting in level at the interval, Romeo Amane swiped at goal from the edge of the area and Kara was unmarked in the six-yard box to convert after a deflection diverted fortuitously into his path.
Claudio Braga stole into the area, picking up on a slack pass, but his shot was met by Rapid goalkeeper Niklas Hedl before a rasping Oisin McEntee strike was dealt with similarly.
Hearts almost made a spectacular start to the second half when Blair Spittal's sweetly struck volley fizzed narrowly wide of the post. Their tactical tweaks at the interval was allowing them more time on the ball and a greater influence on the game.
James Wilson, who scored a spectacular long-range strike in the first leg, broke in behind and saw the ball deflect just wide of the post as he tried to cut back across a defender.
Braga's poked attempt was thumped clear off the line from the resulting corner as the visitors grew even more into the contest.
But, just moments after Magnusson had an incredible opportunity to nod Hearts level again, the Icelandic midfielder was shown a second yellow card for a late challenge on Nikolaus Wurmbrand.
Their attacking was not to be diminished, though, and an instinctive, if not unorthodox, finish from Kabore sent the tie to extra time.
Hemmed in their own defensive third, Hearts broke forward through another substitute, Rogers Mato, who found Miller with a deep, looping cross from the left that the winger fired in with spectacular technique, and appearing to injure his hamstring in the process.
Hearts still had 20 minutes to see out and Cvetkovic's late finish from close range was a deflating blow after such a heroic effort to rally back.
Miller, Jordi Altena, Mato, and Sabah Kerjota all scored from the spot before Oisin McEntee missed with Hearts' first concrete chance of progression.
But Rapid's Moulaye Haidara had his shot saved from the spot before former Celtic attacker Marco Tilio's was also met by the diving Reus to secure a famous win for the team from Tynecastle.
And they will play six more matches in the Conference League this autumn, with the draw for the league phase taking place on Friday.