We propose and experimentally demonstrate an optimized setup to implement quantum controlled-NOT operation using polarization and orbital angular momentum qubits. This device is more adaptive to inputs with various polarizations, and can work both in classical and quantum single-photon regime. The logic operations performed by such a setup not only possess high stability and polarization-free character, they can also be easily extended to deal with multi-qubit input states. As an example, the experimental implementation of generalized three-qubit Toffoli gate has been presented.