It's your misfortune.
3 seconds sound clip from the The Skin Game (1931) movie soundboard.
You can hear this line at 00:47:59 in the Blu-ray version of the movie.
Quote context
[...]
- No. I have a good memory for faces.
- If that's all, you needn't have told me to come.
- Oh, don't go. You are playing a game with me. Aren't you ashamed? What harm have I done you?
- What have I to do with this quarrel? I couldn't help their falling out.
- You're a rotter if you can spoil anyone's life who never did you an ounce of harm.
- Oh, so they don't know about you, eh? Well, that's all right.
- I serve my employer, but I'm flesh and blood, too. And I give as good as I get.
[...]