With the widespread use of network encryption protocols, traditional network traffic classification technology has been challenged. The current method has the following limitations: first, the model is highly dependent on the depth feature, which requires the labeled training data set to be large enough in scale, otherwise the model will have difficulty generalizing to new data; second, the model only focuses on one modal feature of traffic, and the feature differentiation of the same mode of traffic from different categories may not be obvious. To solve these problems, a deep learning-based encryption traffic classification model called Parallel Transformer Net (PTNet) is proposed in this paper. Based on the semi-supervised idea of pre-training and fine-tuning, the model makes full use of a large amount of unlabeled traffic data on the network for pre-training, and then fine-tunes on the basis of a small amount of labeled data. Additionally, the model extracts the flow characteristics of load and packet length sequences in parallel to carry out multi-mode feature fusion. Three different traffic classification tasks and their corresponding datasets (Android, USTC-TFC, and CSTNET-TLS1.3) show good results, with classification accuracies reaching 95%, 98%, and 97%, respectively.