Of course you have to call that down. Look at the posible hands that could have you beaten.
1 PK AA would have raised preflop ( vast majority of the time)
2 PK 44 ok it was there but you cant put someone on a set
3 2 5 ok BB may havethis but wouldnt go all in with the nuts
In this situation I would have put the raiser on an A with a low kicker or 2 pairs and not wanting a call.