Jack Lester opened the scoring for Chesterfield and although Colchester equalised through Steven Gillespie, Josh Thompson grabbed the visitors' winner.
Ward said: "I'm very disappointed. We've missed out on an opportunity with two home matches in a row, regardless who we've played.
"We haven't really taken the two games by the scruff of the neck like I thought we would. We came back into the game and played some really good football.
"But when the second half went on and we couldn't find that opening, we looked like we lost the belief we could get it back. We huffed and puffed but we couldn't get the goal and got a bit anxious.
"We lost the game in both boxes - we gave free-kicks away, which is a strength of theirs and in their box, we never really looked like scoring from our set-plays."