While his departure may be upsetting to some fans, there may have been hints that his time on the show was winding down.