Thanks to Network Function Virtualization (NFV), Internet Service Providers (ISPs) can improve network resource utilization with significantly reduced capital and operational expenditures. To dig deeper into the potential of NFV, an important challenge is the resource allocation problem in NFV (NFV-RA), which can be divided into three stages: VNFs chain composition, VNF forwarding graph embedding, and VNFs scheduling. The key to the NFV-RA problem is to design an effective and coordinated resource allocation algorithm for the three stages. Besides, the NFV-RA problem has been proved to be NP-Hard, and thus most existing approaches focus on heuristic and meta-heuristic algorithms. In this paper, we propose an NFV online coordinated resource allocation framework (OCRA) that completes the three stages simultaneously in a coordinated manner by combining parallel Multi-Agent Deep Reinforcement Learning with novel neural networks and RL training techniques. The extensive experimental results show that compared with the state-of-the-art solutions, OCRA is highly-efficient in terms of time, with up to 50% and 10.8% improvement on resource overhead and acceptance ratio, respectively.