Recent development of novel imaging technologies has led to the acquisition of massive amounts of high-dimensional brain images at various brain scales. Yet, many image analysis tasks mainly rely on human labor to obtain reliable results, making reliable and automated image analysis methods highly demanded.

Conventional machine learning approaches to image analysis tasks involve the design of hand-crafted features, thereby requiring expertise and domain knowledge about datasets. Thus, these approaches do not scale with the increasing number of brain images today. In contrast, deep learning approaches learn directly from raw image data with hierarchical features from low to high levels, making them readily applicable to large-scale image analysis tasks.

This dissertation aims at developing deep learning methods for automated brain image analysis, and makes valuable contributions toward solving problems in two general analysis tasks---multi-instance multi-task (MIMT) classification and 3D dense prediction. The proposed methods are applied to automating gene expression detection in in-situ hybridization (ISH) images of the developing mouse brain, and neurite segmentation in 3D electron microscopy (EM) images.

The MIMT problem is approached by a novel deep neural network that explicitly includes layers for computing instance-level and bag-level representations and takes advantage of transfer learning to overcome the insufficient training sample problem. Correspondingly, based on different aspects of the problem, three deep learning models are designed to approach the task of neurite segmentation: 1. A novel convolutional neural network model built with state-of-the-art techniques to allow a large receptive field, multi-scale features, and easy gradient propagation. 2. A recurrent encoder-decoder model that considers the task as a time-varying dense prediction problem. 3. An instance-aware multi-task model that encodes highly expressive representations by enforcing the network to learn dense multiple instance-level information at output.

Overall, experimental results show that the proposed deep learning approaches are able to achieve promising performance on brain images analysis. Although experiments are conducted for evaluating two specific brain analysis tasks, these deep learning methods are applicable to a broader range of bio-medical image analysis tasks. More importantly, the design of approaches to addressing various challenges provides useful insights toward automating bio-medical images analysis.