Playing short-handed, as you were, means that most of the starting hand requirements go out the window. A-anything is a strong hand this late in the tourney but you want to be the aggressor and not the caller. He was wrong to berate you but there is a tiny lesson to take away from this.....
His argument is that you would put half your stack on the line with A5...a weak hand to call with. By calling his all in, your call was most likely going to be a 3-outer. Meaning that, he is either on a pair or AX himself. If he was on AX, X > your 5, which you'd have to pair in order to beat his unpaired X -OR- you'd be up against a pair > 5 and you would have to hope you hit one of your remaining 3 aces to win the hand.
I don't know if this guy was trying to steal blinds a lot and this was your way of saying "Let's see it" or if weaker hands were taking down lots of pots, and A5 made a lot of sense to call with.
In short, I see why he questioned the call....but that doesn't give anyone the right to call you names. Don't take this as criticism, because its not. I've made calls in similar situations with less. It's simply my thoughts on went went down and why he was upset.