Deep learning for communication over dispersive nonlinear channels: performance and comparison with classical digital signal processing

In this paper, we apply deep learning for communication over dispersive channels with power detection, as encountered in low-cost optical intensity modulation/direct detection (IMDD) links. We consider an autoencoder based on the recently proposed sliding window bidirectional recurrent neural network (SBRNN) design to realize the transceiver for optical IMDD communication. We show that its performance can be improved by introducing a weighted sequence estimation scheme at the receiver. Moreover, we perform bit-to-symbol mapping optimization to reduce the bit-error rate (BER) of the system. Furthermore, we carry out a detailed comparison with classical schemes based on pulse-amplitude modulation and maximum likelihood sequence detection (MLSD). Our investigation shows that for a reference 42 Gb/s transmission, the SBRNN autoencoder achieves a BER performance comparable to MLSD, when both systems account for the same amount of memory. In contrast to MLSD, the SBRNN performance is achieved without incurring a computational complexity exponentially growing with the processed memory.