In this study, the authors address the problem of combining descriptors for purposes of object categorisation and classification. The authors cast the problem in a structured learning setting by viewing the classifier bank and the codewords used in the categorisation and classification tasks as random fields. In this manner, the authors can abstract the problem into a graphical model setting, in which the fusion operation is a transformation over the field of descriptors and classifiers. Thus, the problem reduces itself to that of recovering the optimal transformation using a cost function which is convex and can be converted into either a quadratic or linear programme. This cost function is related to the target function used in discrete Markov random field approaches. The authors demonstrate the utility of our algorithm for purposes of image classification and learning class categories on two datasets.