I can definitely see why someone wouldn't like it. And yeah SBC generally relies a lot on that.
Another reason I think people didn't like it (especially LGBT groups) was that it was trying to do two things: satirize fashion/modeling culture and also trying to make statements on homophobia. The problem with this is that, well, Brüno is an idiot. GLADD was unhappy about his poor parenting bc they believed it would send the message that gays are bad parents. And since SBC does shock value and enjoys trolling he'd have Brüno doing the stupidest shit with the baby. His bad parenting wasn't bc he was gay but bc he's stupid. But since the film was trying to make statements on being gay the lines got blurry.
Although I could be wrong.
Not to mention the hotel scene where he and his lover were caught doing shit that would disgust the gayest of the gay. It was pretty funny to me though.