Is it rude to ask someone to repeat themselves?

Im a little hard to hearing, and when i deal with customers at work, i sometimes do not catch what they said, i say pardon, sometimes i still dont hear a second time and say sorry can you say that again, at which point most people become quite defencive and agitated, i dont understand it, why do people get embarred to repeat themselves?