Extensive studies have been done in topics related to structural health monitoring using image processing and machine learning techniques. Trends in recent studies show increasing interests in the use of vision-enabled Unmanned Aerial Vehicle (or UAV) systems in structural health monitoring applications. Although there have been a lot of studies focused in theoretical aspects of image processing and machine learning, not much attention is given to the practical aspects. This thesis presents algorithm-based solutions to real-time infrastructure damage detection and damage identification with the intention of use with UAV systems. In this thesis, detection and identification results are quantified for intuition. In a practical scenario, a vision-enabled UAV system might be used to acquire image data of a damaged surface of interest with the goal of damage detection. However, unwanted information might be captured in the image data during such an exercise. The algorithm presented in this thesis is intended to remove the irrelevant regions from image data, detect damage, and quantify the results. The algorithm successfully detects damage in five out of seven tested categories. Generalization to different types of material surfaces is one of the objectives behind development of the algorithm. Identification of the type of damage is also attempted in another part of the algorithm. In this case, the algorithm shows reliable results in most test datasets and indicates the most prominent type of damage present.