Predicting the severity of an illness is crucial in intensive care units (ICUs) if a patient‘s life is to be saved. However, most methods do not consider the correlations between features in the patient‘s condition over time. Moreover, the existing prediction methods fail to provide sufficient evidence for the time-critical decisions required in such dynamic and changing environments. For ICU caregivers, the facts and reasoning behind a prediction are the most important criteria when deciding what medical actions to take. However, the current methods lack organ level predictions and reliable interpreted prediction results. They focus on either the overall physiological severity without insight for medical staff or the illness severity that lacks generalizability. The existing interpretable models only provide the feature importance but the importance cannot be used as reliable guidance of treatment. In this paper, we propose an interpretable organ failure prediction method as a benchmark on the MIMIC-III dataset. We build some state-of-the-art (SOTA) models to implement the predictions of different organs that are used in the Sequential Organ Failure Assessment (SOFA)scores. Then we interpret prediction results by introducing the counterfactual explanation techniques. The experiment results show the high performances of the predictions on coagulation, liver, nervous, and respiration failures with F1 scores of more than 0.9. Furthermore, the counterfactual explanations can also capture the key features that can prevent the organ failure trend by comparing with true normal records in two case studies.