It seems to me that he's saying that race is playing a role in whether people see Obama as safe or risky, which is pretty clearly true. I wish it weren't true, but that's what polling data has shown throughout the campaign.

I didn't see the complete Hardball clip, but I didn't get the impression that he meant "one of us" as his own opinion, but that he was saying that seemed to be what people worried about Obama were saying.