At present, the traditional education quality evaluation system has some problems, such as insufficient data processing ability and poor real-time performance, which is difficult to meet the needs of teaching quality improvement in rapidly developing higher vocational colleges. In order to solve such problems, this paper designs an intelligent education quality improvement system based on Swin Transformer and simulated annealing algorithm. Through the pilot experiment of applying this system in a higher vocational college, it is found that after three months' trial, the overall teaching satisfaction of the college has increased by about 12.8%, the teachers' teaching quality score has increased by 9.4%, and the students' learning efficiency has increased by 8.7% on average. The research shows that the system can provide strong support in improving teaching quality, optimizing the allocation of teaching resources and promoting teachers' teaching improvement, and has good popularization and application value.