Raju Kher clarified, "He could have done so, that was my thought at that time." But he later realised he had taken his brother for granted