Ever increasing demands for higher data rates in mobile communication present an ultimate challenge to deliver a consistent service experience to users. Filter bank multicarrier (FBMC) systems with very low out of band radiation facilitate aggregation of non-adjacent bands and asynchronous transmissions to provide the service enhancement. The primary research objective in these systems is followed by the contributions provided in this thesis.

First of all, filter bank based multicarrier systems using fast convolution approach are investigated. We show that by applying FFT-based convolution without overlapped processing, the circular distortion can be discarded as a part of orthogonal interference term. The advantages of this property are spectral efficiency enhancement in the system and complexity reduction. The results show spectral efficiency enhancement by up to 15% compared to conventional FBMC, while the complexity of the system is roughly half of conventional FBMC.

Secondly, two channel estimation methods for MIMO-FBMC are proposed. The first one is a preamble-based approach aiming to reduce the preamble duration overhead and peak to average power ratio compared to the existing methods. The results confirm that this method outperforms the existing methods in terms of the mentioned parameters with up to 55% in the overhead reduction. The second approach is a comb-based method with avoiding the contamination of pilots with symbol interference and saving time-frequency resources compared to existing methods. The results show a comparable performance of this method with is OFDM counterpart.

Finally, we investigate the problems of transmitting complex symbols in FBMC systems. The challenge in these systems is designing appropriate filters to minimize the interference between adjacent subcarriers while maintaining the Nyquist property of the filter. We exploit fractional shift of the filtered symbols to compensate the deviation from the Nyquist property. The results show the proposed filter provides approximately 3 dB improvement of signal to interference ratio compared to the conventional filters.