It was a quiet start in North London and Spurs, for all of their possession, failed to find a way through the Panathinaikos defence in the early stages.
The first real chance of the game fell to the visitors and Spanish striker Toche, who scored against Spurs in the reverse fixture, should have done better when he steered shot straight at Brad Friedel following a clever free-kick.
A win would have been enough for Panathinaikos to qualify from Group J but Spurs took the lead before the half hour mark. Emmanuel Adebayor, starting his first game since the 0-0 draw at Lazio, latched onto Clint Dempsey's through ball and coolly finished past the keeper.
Jermain Defoe almost doubled the lead on either side of half-time when he lashed a shot over from six-yards before a left-footed shot four minutes into the second-half came back off the inside of the post.
Panathinaikos rarely troubled Spurs going forward but they equalised out of nowhere after 54 minutes when Portuguese midfielder Zeca met Charalampos Mavrias' cross to send a header past Friedel.
The goal turned the game in the visitors' favour and there were nervy moments for Spurs as Toche poked a shot agonisingly wide from close range.
But Spurs restored their lead on 76 minutes when Clint Dempsey's header from a Kyle Walker free-kick free-kick hit the underside of the bar and bounced on the line before going in off the goalkeeper's back.
Defoe put the icing on the cake six minutes later with a delicate chip over Karnezis into the bottom corner, his eighth goal in nine starts for Tottenham.
Andre Villas-Boas' side finished runners-up in Group J, two points behind Lazio who ran out 4-1 winners at NK Maribor, and will now await the draw for the knockout phase on December 20.