Atheists have a harder time but so do non-Christians. Visible minorities, that much more.
Religious believers tend to associate their religion closely with morality. Some go so far as to think that the two are inseparable — that without their religion, or religion generally, or at least some sort of theism, morality and moral behavior are impossible. Depending on their attitude, this can lead people to insist that unless a person is a member of their religion, or is a member of some religion, or is at least a theist, then they cannot be moral and if they are given any power then they will end up promoting immorality.
Without God, All Things are Permitted, Godless Atheists Promote a Valueless, Immoral World without Order or Structure, Austin Cline