Anxiety or nervousness is one of the affective factors that affects anyone learning and practicing English as a foreign language, especially when speaking the language. Anxiety in English speaking is commonly related to apprehension, fear, and worry that the learners experience when using the language. Most researchers in Indonesia studied speaking anxiety in terms of psychological and linguistic factors, but not specifically to relate with variables such as gender and number of years for learning the target language. This study is in addition to examining the speaking anxiety level of Indonesian learners of English, and the dominant factor triggering the anxiety, and also to attest the difference of anxiety levels related to gender and the number of years in learning English. By using a quantitative method of case study research design, this study revealed that female learners are more anxious than males in speaking English which mean gender is affected by foreign language anxiety. Besides, the number of years in learning English also influences learners controlling their anxiety during speaking which means the longer they learn English the less anxious they speak the language. Then, test anxiety most dominantly contributes to the uneasiness factor for students in speaking English. The result of this study implies that teachers or lecturers of English must provide a space for female students to feel more secure in speaking class by avoiding negative evaluation so that they feel comfortable while learning and acquiring the language.