"You lose your home, you're part of a club nobody wants to be a part of," the actor said of he and wife Keleigh's home ...