The radio show host said he was making "a sarcastic comment" about how police unfairly target blacks when he asked about Adam "Pacman" Jones's race on the show. He said that his comment was meant to defend and not offend Jones.
Imus claims that people are "out to get him" after the Rutgers controversy.
"No man has more discussions about race than I have," he said. "Why not report on that? I know some people want to get me, but this is ridiculous."
Imus pointed out that his show has two black co-hosts.
Jones, a Dallas Cowboy defensive back, has been arrested six times since he was drafted in 2005 and is currently suspended from the National Football League.
The controversy was sparked Monday morning while Imus was speaking with sportscaster Warner Wolf.
"He's been arrested six times since being drafted by Tennessee in 2005," Wolf said.
"What color is he?" Imus asked.
"He's African-American," Wolf responded.
"Well, there you go," Imus said. "Now we know."
During his Tuesday morning show, the radio host said Jones was arrested six times because police were "picking on him."
"He's a football player. He is a lovely kid. He was out there having fun," Imus said.
Imus said he was defending blacks with his Monday remarks.
"What people should be outraged about is that they arrest blacks for no reason, and I mean there's no reason to arrest this kid six times," he said. "They shoot blacks for no reason."Rev. Al Sharpton, who called for Imus' firing over the Rutgers comments, said that he hopes that the host's reasoning for his comments is sincere.
Sharpton said he thought Monday's remark was disturbing "given Imus' background and his track record."
"If now Imus is saying he's joined the ranks of those that are raising the question of racial disparity in the criminal justice system, then he's taking a correct position," Shapton said. "I don't have any record of him saying that in the past."