Did he ask "is it okay?" or did he say "we're bringing the kids."...?

If the former, I don't see a problem. Some parties are kid-friendly, others aren't. Finding out which kind this one is should be just an exchange of information. If you already told him so in the invitation, he's being obtuse, but as long as he accepts it gracefully when you reconfirm your plans, no big deal.

If he's turning it into an issue, that's not okay.

If he feels like he can't afford the babysitter, that's his issue, not yours. It's a legitimate reason to decline, but not something you need to feel guilty over.

Good luck with your party!

__________________
Haunt pics: 2005, 2006