The 2-1 home loss means Allardyce has taken only one point from his first three fixtures since succeeding Alan Pardew as Palace boss, leaving them just one place above the bottom three.
‘Big Sam’ has admitted it has been tough so far for him at Selhurst Park as he looks to turn things around.
“(This is proving) a bigger challenge, after three games, than I expected,” he said.
The Eagles were again as unconvincing as in Sunday’s 2-0 defeat at Arsenal but Allardyce was critical of referee Paul Tierney for not giving them a first-half penalty, despite goalkeeper Lukasz Fabianski not appearing to make contact with Christian Benteke when the striker went down, and one in the second when he felt Angel Rangel had handled the ball.
“Fatigue (is the main reason for our performance),” Allardyce added. “Not having enough recovery time compared to Swansea.
“The referee blatantly allows the goalkeeper to stay on the pitch and doesn’t give us a penalty, so the referee’s to blame, but we can’t control the referee.
“That performance in the first half just wasn’t good enough. I have to take a bit of responsibility – picking almost the same team (just two days after losing at Arsenal) was the wrong thing for me to do.
“Then we had another really poor decision on the handball (by Rangel) from the referee, in the box. It’s been a very, very difficult night for us all.”
Colchester will hand late fitness tests to Ryan Jackson and Brandon Comley prior to their clash with Yeovil.
Arsenal secured their place in the quarter-finals of the Europa League following their 3-1 defeat of AC Milan at the Emirates Stadium on Thursday.
Danny Welbeck scored twice against AC Milan to toast his England recall by sending Arsenal into the last eight of the Europa League.
Ipswich striker Joe Garner will not play again this season after it emerged he suffered a fractured skull against Sheffield United last week.