Efficient video / image coding techniques are desired for the dramatically growing demand of storage and transmission of visual information. The widely used coding standards, e.g., H.264 and HEVC, are all based on a hybrid video coding framework which consists of a predicting process followed by transform coding and entropy coding. Looking into the results of encoding one image, the discontinuous regions, i.e., regions containing edges / contours, consume most of the coding bits because of the inaccurate prediction of the discontinuities. Due to the arbitrariness of the contours in the image, it is impractical to accurately predict them using a model with manageable complexity. Instead of using an integrated predicting technique to encode both the discontinuous regions and the remaining regions as in a hybrid video coding framework, an alternative way is to perform a separate process to the discontinuities in the image. Separate discontinuity handling provides more freedom to design an algorithm for efficient encoding of the discontinuous regions.

In this thesis, we focus on handling the discontinuities of the video / image in two directions in order to improve the overall coding efficiency. We first discuss a contour assisted coding framework which pre-encodes the discontinuities before feeding the input image into a conventional encoder, and propose efficient algorithms to encode the discontinuities. Then we propose a new approach, adaptive coding order, for a hybrid video coding framework to reduce the discontinuities.

For a contour assisted coding framework, the object contours / edges (discontinuities) are first extracted from the input frame as side information (SI), then the input frame is encoded by utilizing SI. One key problem is how to efficiently compress SI, i.e., the contours. For the direction of contour assisted coding, we focus on the problem of lossless and lossy compression of detected contours in the image. Specifically, to encode the symbol sequence contour using arithmetic coding, we compute an optimal variable-length context tree (VCT) via a maximum a posterior (MAP) formulation to estimate the symbols' conditional probabilities. MAP can avoid overfitting given a small training set of past symbol sequences, using a geometric prior stating that image contours are more often straight than curvy. For the lossy case, we design fast dynamic programming (DP) algorithms that optimally trade off the coding rate of an approximated contour given a VCT with two notions of distortion. The proposed algorithms are integrated in most recent contour assisted coding framework based coders and applied to three applications. Experimental results show that the proposed contour coding algorithms significantly improve the overall coding performance for different applications. To deal with the problem of compressing noised contours, we further propose a joint contour denoising / compression algorithm based on a burst error model. We prove both in theory and in experiments our proposed joint scheme outperforms the competing separate schemes which first do denoising then compression.

For the direction of adaptive coding order, the order of processing the basic coding unit--a block of pixels--in hybrid video coding is adaptively adjusted based on the edge direction rather than pre-determined in a fixed manner. Adaptively adjusting the processing order can reduce the discontinuities in the image by avoiding crossing the edges when performing prediction. Specifically, determining the block coding order is formulated as a travelling salesman problem that is solved using the DP algorithm. Experimental results show that the proposed algorithm outperforms the state-of-the-art HEVC, which is mainly due to more efficient coding of the discontinuous regions.