Simon Church grabbed the only goal after seven minutes when he latched onto a chipped Dale Stephens pass before controlling the ball and burying it with a cool side-footed volley.
It is the first time Rovers have lost two consecutive games this season and is their second straight Ewood reverse to Charlton, but Bowyer felt his side were on the wrong end of a crucial decision.
"We've seen it (the goal) again on the video," he said. "I'm not sure which part of the arm it does hit him on but it's quite clear on the video that it's handball, but these things go against you and level themselves out throughout the year.
"The ball didn't quite bounce for us and you get done at the other end with a sucker punch which is a handball for their goal and we're disappointed it hasn't been picked up."
Despite that decision, Bowyer was disappointed that his team could not find a way of breaking down the Addicks.
He said: "Credit has to go to Charlton for the amount of bodies they did get behind the ball but it's up to us to be a little bit cuter.
"There was plenty of endeavour and workmanlike performances but we needed more care in the final third."