With the growth of large-scale multimodal data and advancements in neural networks (NNs), cross-modal retrieval (CMR) relies on outsourced computation for efficiency and scalability. Existing privacy-preserving CMR approaches under the fully outsourced setup typically employ traditional machine learning models or simple NNs due to the limitations of privacypreserving techniques. It results in significant performance gaps in efficiency and accuracy compared to plaintext CMR. In this paper, we present PP-DACMR, a privacy-preserving deep adversarial hashing approach for cross-modal retrieval. With the paradigm of secure two-party computation, PP-DACMR employs a lightweight technique, additive secret sharing (ASS), to safeguard multimodal data and NNs. A series of ASS-based protocols are designed specifically for CMR based on GAN architecture, supporting in-the-cloud training and querying. Moreover, we optimize two generic secure arithmetic protocols for truncation and matrix multiplication, which are fundamental to PP-DACMR, contributing to improved performance. We conduct experimental evaluations over real-world multimodal datasets and compare PP-DACMR to the state-of-the-art approach PPCMR. The results demonstrate PP-DACMR outperforms PPCMR, achieving an improvement on the average mAP of 11.73% and being 5.3× faster.