He leaves town almost every weekend to see his "grandpa"