After powering past Borussia Dortmund in the Champions League on Wednesday, Spurs looked to have turned a corner at their new home.
But another two points dropped on Saturday evening means the doubts will creep back in.
Tottenham have now failed to win any of their opening three league matches under the arch, with Burnley and now Swansea each taking a point away from the national stadium after Chelsea snatched all three.
The contrast is all the more dramatic given Spurs' relentless form at White Hart Lane last season, where they were unbeaten in the league and took 18 points out of 18 against the league's bottom six.
Pochettino, however, remains adamant that Wembley is not the problem.
"The reality is that we dropped a lot of points after three games but I am not worried," he said. "We could spend one season talking about that and maybe at the end then you can see it or everyone can say, 'Yes, Wembley was a problem'.
"It's true that last season was a fantastic season at White Hart Lane. But today we are at Wembley and White Hart Lane is in the history.
"We are building a new stadium which will be fantastic and then I believe that every single season is completely different. We cannot compare last season with this season."