Waist circumference is a convenient measure of obesity, abdominal obesity and fat distribution and it is a more efficient predictor than BMI for diseases including type 2 diabetes and cardiovascular disease. In this study, we investigate the GxE interplay in influencing waist circumference among the elderly males using the data from the Health and Retirement Study (HRS) that includes 4,944 males whose waist circumferences are measured at five different time points, mostly aged 50 to 100. We generate "top-hits" polygenic scores using genetic variants associated with waist circumference from previous studies and whole-genome polygenic score using whole-genome genetic variants from GIANT consortium. By incorporating the polygenic scores, socioeconomic status (SES), and lifestyle, we found that the genetic influence on waist circumference for people who do less physical activity is higher than for those who do more physical activity.