We use the Survey of Thermodynamic Processes and First and Second Laws-Long, a research-based survey instrument with 78 items at the level of introductory physics, to investigate introductory and advanced students' difficulties with internal energy, work, and heat transfer. We present analysis of data from 12 different introductory and advanced physics classes at four different higher education public institutions in the U.S. in which the survey was administered in person to more than 1000 students. We find that not only introductory but also advanced physics students have many common difficulties with these introductory thermodynamic concepts after traditional lecture-based instruction in relevant concepts. We utilize a wide variety of problem types and contexts and our sample includes large numbers of introductory algebra-based, calculus-based, and advanced students. Some of our findings are consistent with prior research in this area, but others expand upon them and reveal previously unreported aspects of students' thinking. Findings related to common difficulties of students before and after traditional lecture-based instruction in college physics courses can help instructors of these courses plan instruction and curricula to improve student understanding. These findings can also be valuable for developing effective research-based curricula and pedagogies to address student difficulties and help students develop a functional understanding of these fundamental thermodynamic concepts.