I would be slightly concerned about the three-year-old. Not because of you, but because of him, and here's why: If he has custody, and he starts dating someone - anyone, regardless of age - that little girl is going to know about it. Either daddy's leaving her with a babysitter, or some new person is coming over to the house a lot. That's a bit of an upheaval right there, but it sounds like you've met and get along with her, so it's more something 'to be aware of'. The real issue comes if it doesn't work out, and suddenly this new person that the little girl has met and (apparently) likes is gone from her life - not in the same way as a divorce, either.
I would proceed with caution, especially if (as you say) he's not 'ready for marriage again'.