Exclusive breastfeeding is a proven benefit for both mothers and infants and is, therefore, an important public health priority. Intimate partner violence (IPV) is regarded as one of the potential psychosocial risk factors that may negatively affect exclusive breastfeeding (EBF). This study aimed to explore the influence of psychosocial factors including IPV on EBF. Cross-sectional survey data was collected from October 2015 to January 2016 in Chandpur District of Bangladesh from 426 married women, aged 15–49 years, who had at least one child 6 months of age or younger. Multivariate logistic regression models were used in order to investigate whether women who experienced IPV after childbirth, as well as other risk factors such as postpartum depression (PPD) and childhood sexual abuse, were more likely to face difficulties with EBF compared with women who had not experienced these same risk factors. Whilst the initiation rate of breastfeeding was 99.3%, at the time of the woman’s interview, the overall EBF rate had fallen to 43.7%. Based on the adjusted model, women who experienced physical IPV (AOR 0.17, 95% CI [0.07, 0.40]) and psychological IPV (AOR 0.51, 95% CI [0.26, 1.00]) after childbirth and women who reported childhood sexual abuse (AOR 0.32, 95% CI [0.13, 0.80]) and PPD (AOR 0.20, 95% CI [0.09, 0.44]) were significantly less likely to exclusively breastfeed their infants than those who had not reported these experiences. Moreover, women with an intended pregnancy and high social support exhibited a higher likelihood of EBF. Our results suggest that preventing or reducing the occurrence of physical IPV, PPD and childhood sexual abuse may improve the EBF duration. Support from family members can assist in this process.