He's never once said that he had.
4 seconds sound clip from the Beat the Devil (1953) movie soundboard.
You can hear this line at 00:55:49 in the Blu-ray version of the movie.
Quote context
[...]
- Married to the son of a boardinghouse in Earl's Court.
- A boardinghouse. That's what Harry's parents do. They run a boardinghouse for decayed gentlefolk.
- But the way he talks, the way he acts, I thought...
- It's not his fault if people take it for granted that he has a place like that.
- He's never once said that he had.
- Country gent, son of a boardinghouse or whatever he is,
- I suppose I'd better get his box back.
- Except that Harry's gone to the captain.
[...]