Well, he did, indeed.
3 seconds sound clip from the McLintock! (1963) movie soundboard.
You can hear this line at 01:06:34 in the Blu-ray version of the movie.
Quote context
[...]
- Uh, Dev could you come and help me a minute?
- I certainly was surprised to hear you went to college.
- Why?
- Junior says Purdue's a good college for a backwater place like Indiana.
- Oh, could you do this? I can't reach it.
- Why didn't you finish college?
- My father got sick, and he had to come out west. So he took out a homestead.
- You know, your mom's sure cute. It's, uh, too bad you didn't inherit her eyes.
- Well, you'd been lucky, if you'd inherited a few things from your father.
[...]