So Buddy stayed with an older elf who had always wanted a child... but had been so committed to building toys... well, he'd forgotten to settle down.
11 seconds sound clip from the Elf (2003) movie soundboard.
You can hear this line at 00:06:18 in the Blu-ray version of the movie.
Quote context
[...]
- His name is Buddy. He must have... Snuck into your sack at the orphanage.
- So Santa had a decision to make.
- Fortunately, when it comes to babies, Santa's a pushover.
- Santa.
- Yes. Yes, I raised Buddy. I was his adopted father.
- Though Buddy grew twice as fast... he wasn't any different from the other children.
[...]