Molecular representation learning is essential for molecular property prediction, a key step in drug design and material discovery. While graph neural networks (GNNs) effectively capture structural information from molecular graphs, they often fail to incorporate broader contextual knowledge that goes beyond the local graph structure, such as chemical reactivity and relationships derived from domain-specific knowledge. Conversely, large language models (LLMs) excel at processing sequential molecular representations like SMILES strings. However, current methods inadequately integrate these complementary modalities, particularly for complex tasks in quantum and physical chemistry. To address this challenge, we propose Uni-MRL, a unified multi-modal framework that integrates GNN-based structural embeddings with task-specific features defined by LLMs. By leveraging the reasoning ability of LLMs to generate chemically meaningful features and combining them with graph-based representations through task-driven fusion and contrastive alignment, Uni-MRL captures both structural and chemical intricacies of molecular representations. We show that this unified framework achieves state-of-the-art performance in quantum and physical chemistry tasks. Our ablation studies further demonstrate the importance of its components, highlighting the effectiveness of multi-modal integration.