This study aims to investigate the effect of chatbots that work with artificial intelligence on the success of students and their opinions about chatbots in the 'Matter and the changing state of matter' unit in the 5th grade science course. In addition to text-based functions, the designed chatbot includes a video accessed on the web to support students visually and aurally. The chatbot was designed using the Dialogflow program and an instant messaging program made available to students through a group created on Telegram. The study, which used a quasi-experimental pretest-posttest design, included 41 participants (n = 20 for the experimental, n = 21 for the control group) studying in the 5th grade of a state secondary school in the 2020-2021 academic year. Results suggest that although there was no significant difference between the experimental and control groups in terms of academic achievement, it was determined that the chatbot application positively affected the online learning experience of the experimental group students. Students' opinions about the chatbot included that it was useful and fun, they would like to use it for other courses, it provided useful assistance in learning outside the classroom, and it allowed them to repeat the course again. The results showed that, especially during the Covid-19 pandemic, such applications could contribute positively to students' learning.