I bet he knows the twins' parents. I feel like people hire girls way more often as babysitters, so what if V's sister used to bring the twins to the Kim house while she watched them? So he'd be familiar with them but was never necessarily all that close.