Background Scrotal swellings have a non-specific clinical picture, so their clinical diagnosis is challenging. Scrotal grayscale and color Doppler ultrasound are non-invasive methods used in both adult and childhood groups and act as accurate screening and diagnostic modalities. Purpose To evaluate the diagnostic validity of grayscale and color Doppler ultrasound in the assessment of scrotal swelling to reach accurate diagnosis. Material and Methods A retrospective study included 181 patients (mean age = 35.5 ± 7.3, age range = 1–71 years) with scrotal swelling. Examinations were performed by an experienced radiologist using grayscale and color Doppler ultrasound. The diagnostic validity of grayscale and color Doppler ultrasound for diagnosing scrotal swelling were estimated using surgical findings, histopathological results, and imaging and clinical follow-up as reference standards. Results Overall, 202 scrotal swellings were detected. The final diagnoses were 13 (6.4%) malignant and 189 (93.6%) benign alterations. Varicocele was the most common scrotal swelling (26%), followed by hydrocele (23.8%). Matched to the reference standards, grayscale and color Doppler ultrasound represented a sensitivity of 84.6% (95% confidence interval [CI] = 54.6–98.1), a specificity of 76.2% (95% CI = 69.5–82.1), a positive predictive value of 19.6% (95% CI = 10.2–32.4), and a negative predictive value of 98.6% (95% CI = 95.1–99.8) for diagnosing scrotal tumors. Conclusion Scrotal grayscale and color Doppler ultrasound provide high diagnostic validity for assessment of scrotal swellings.