The 2-1 home loss means Allardyce has taken only one point from his first three fixtures since succeeding Alan Pardew as Palace boss, leaving them just one place above the bottom three.
'Big Sam' has admitted it has been tough so far for him at Selhurst Park as he looks to turn things around.
"(This is proving) a bigger challenge, after three games, than I expected," he said.
The Eagles were again as unconvincing as in Sunday's 2-0 defeat at Arsenal but Allardyce was critical of referee Paul Tierney for not giving them a first-half penalty, despite goalkeeper Lukasz Fabianski not appearing to make contact with Christian Benteke when the striker went down, and one in the second when he felt Angel Rangel had handled the ball.
"Fatigue (is the main reason for our performance)," Allardyce added. "Not having enough recovery time compared to Swansea.
"The referee blatantly allows the goalkeeper to stay on the pitch and doesn't give us a penalty, so the referee's to blame, but we can't control the referee.
"That performance in the first half just wasn't good enough. I have to take a bit of responsibility - picking almost the same team (just two days after losing at Arsenal) was the wrong thing for me to do.
"Then we had another really poor decision on the handball (by Rangel) from the referee, in the box. It's been a very, very difficult night for us all."