The Machine Learning (ML) community has recently started to recognise the importance of model interpretability when using ML techniques. In this work, I review the literature on Explainable Artificial Intelligence (XAI) and interpretability in ML and discuss several reasons why interpretability is critical for many ML applications. Although there is now increased interest in XAI, there are significant issues with the approaches taken in a large portion of the research in XAI. In particular, the popularity of techniques that try to explain black-box models often leads to misleading explanations that are not faithful to the model being explained. The popularity of black-box models is, in large part, due to the immense size and complexity of many datasets available today. The high dimensionality of many datasets has encouraged research in ML and particular techniques such as Artificial Neural Networks (ANNs). However, I argue in this work that the high dimensionality of a dataset should not, in itself, be a reason to settle for black-box models that humans cannot understand. Instead, I argue for the need to learn inherently interpretable models, rather than black-box models with post-hoc explanations of their results. One of the most well-known ML models for supervised learning tasks that remains interpretable to humans is the Decision Tree Classifier (DTC). The DTC's interpretability is due to its simple tree structure where a human can individually inspect the splits at each node in the tree. Although a DTC's fundamental structure is interpretable to humans, even a DTC can effective become a black-box model. This may be due to the size of a DTC being too large for a human to comprehend. Alternatively, a DTC may use uninterpretable oblique splits at each node. These oblique splits most often use a hyperplane through the entire attributes space of a dataset to construct a split which is impossible for a human to interpret past three dimensions. In this work, I propose techniques for learning and visualising DTCs and datasets to produce interpretable classifiers that do not sacrifice predictive power. Moreover, I combine such visualisation with an interactive DTC building strategy and enable productive and effective Human-In-the-Loop-Learning (HILL). Not only do classifiers learnt with human involvement have the natural requirement of being humanly interpretable, but there are also several additional advantages to be gained by involving human expertise. These advantages include the ability for a domain expert to contribute their domain knowledge to a model. We can also exploit the highly sophisticated visual pattern recognition capabilities of the human to learn models that more effectively generalise to unseen data. Despite limitations of current HILL systems, a user study conducted as part of this work provides promising results for the involving the human in the construction of DTCs. However, to effective employ this learning style, we need powerful visualisation techniques for both high dimensional datasets and DTCs. Remarkably, despite being ideally suited for high dimensional datasets, the use of Parallel Coordinates (||-coords) by the ML community is minimal. First proposed by Alfred Inselberg, ||-coords is a revolutionary visualisation technique that uses parallel axis to display a dataset of an arbitrary number of dimensions. Using ||-coords, I propose a HILL system for the construction of DTCs. This work also exploits the ||-coords visualisation system to facilitate human input to the splits of internal nodes in a DTC. In addition, I propose a new form of oblique split for DTCs that uses the properties of the ||-coords plane. Unlike other oblique rules, this oblique rule can be easily visualised using ||-coords. While there has recently been renewed interest in XAI and HILL, the research that evaluates systems that facilitate XAI and HILL is limited. I report on an online survey that gathers data from 104 participants. This survey examines participants' use of visualisation systems which I argue are ideally suited for HILL and XAI. The results support my hypothesis and the proposals for HILL. I further argue that for a HILL system to succeed, comprehensive algorithm support is critical. As such, I propose two new DTC induction algorithms. These algorithms are designed to be used in conjunction with the HILL system developed in this work to provide algorithmic assistance in the form of suggestions of splits for a DTC node. The first proposed induction algorithm uses the newly proposed form of oblique split with ||-coords to learn interpretable splits that can capture correlations between attributes. The second induction algorithm advances the nested cavities algorithm originally proposed by Inselberg for classification tasks using ||-coords. Using these induction algorithms enables learning of DTCs with oblique splits that remain interpretable to a human without sacrificing predictive performance.