Currently there are no assessment instruments available for upper-division thermal physics, though several introductory assessments are currently available. Notably missing from these introductory assessment are items targeting statistical mechanics. This leaves a gap in the content that can be assessed by upper-division thermal physics faculty. In this paper, we present a new, upper-division thermal physics assessment that explicitly addresses both classical thermodynamics and statistical mechanics: The Upper-level Statistical and Thermodynamics Evaluation for Physics (U-STEP). We piloted the U-STEP in multiple classes across various institutions during three semesters, and collected over 600 student responses in total. Here, we present multiple measures of validity and reliability for the U-STEP. We utilized classical test theory to determine item difficulties and discriminations, and differential item functioning to identify potential bias in assessment items that can manifest as performance gaps between different genders and races. The completed U-STEP will be the first upper-division thermal physics assessment available, and one of the first standardized physics assessments to explicitly address identification of possible item bias during the development process.