[ Apparently talking about sex is hard for everyone except Aligula. GET WITH THE SHAMELESSNESS, KIDS....... ]

It's probably not, but it's yours. You don't ever have to tell him if you don't want to—if he asks, you can just say you and your wife had your differences and you thought it'd be best if things ended between you two. It wouldn't be a lie, after all.