Brad Pitt's hectic filming schedule has reportedly got in the way of him spending time with his youngest children, while his adult kids have made their feelings on their dad clear