It's just that he sees himself in a place in the West Country with trout streams and horses, leading the life of a country squire.
8 seconds sound clip from the Beat the Devil (1953) movie soundboard.
You can hear this line at 00:55:39 in the Blu-ray version of the movie.
Quote context
[...]
- Well, in a way, you're the one to blame. I mean, you acted so superior.
- I was falling in love with you and I... I couldn't bear for you to think I was just a nobody...
- Married to the son of a boardinghouse in Earl's Court.
- A boardinghouse. That's what Harry's parents do. They run a boardinghouse for decayed gentlefolk.
- But the way he talks, the way he acts, I thought...
- It's not his fault if people take it for granted that he has a place like that.
- He's never once said that he had.
- Country gent, son of a boardinghouse or whatever he is,
- I suppose I'd better get his box back.
[...]