The segmentation and classification of retinal arterioles and venules play an important role in the diagnosis of various eye diseases and systemic diseases. The major challenges include complicated vessel structure, inhomogeneous illumination, and large background variation across subjects. In this study, we proposed an improved fully convolutional network that simultaneously segment arterioles and venules directly from the retinal image. To simultaneously segment retinal arterioles and venules, we configured the fully convolutional network to allow true color image as input and multiple labels as output. A domain-specific loss function is designed to improve the performance. The proposed method was assessed extensively on public datasets and compared with the state-of-the-art methods in literatures. The sensitivity and specificity of overall vessel segmentation on DRIVE is 0.870 and 0.980 with a misclassification rate of 23.7% and 9.8% for arteriole and venule, respectively. The proposed method outperforms the state-of-the-art methods and avoided possible error-propagation as in the segmentation-classification strategy. The proposed method holds great potential for the diagnostics and screening of various eye diseases and systemic diseases.