In fewer than two years, generative artificial intelligence (GenAI) has transformed the educational experience for both students and faculty. Writing feedback and evaluation tools like MyEssayFeedback, EssayGrader, and Markr have been released with the promise that faculty will be able to focus more on teaching than simply grading. However, the proprietary training models of these tools make it difficult to discern precisely what criteria these AI-powered tools might use to evaluate and respond to student writing. Our study implemented ChatGPT3.5 to generate feedback and evaluative scores on short assignments in a broad range of disciplines. We took a descriptive approach to better understand the features and characteristics of student writing highlighted in AI evaluation and feedback responses. Project members engaged in iterations of open and axial coding resulting in two major themes, each with a handful of specific codes. The themes were Content and Tone of AI-Generated Feedback (Criteria Invention, Summarization of Student Response, and Encouragement Hedging Criticism) and Accuracy and Logistical Issues with AI Feedback Generation (Scoring, Inaccuracy, Context Window, and Attention to Purpose of Task). Based on a perceived relationship between student response length and ChatGPT score, we also performed an ad hoc hypothesis test to determine how likely the correlation (r=0.211; weak positive) we observed would occur under different random grading conditions. Results suggested statistically significant evidence that ChatGPT used response length as one, but not only, metric for providing an evaluation score. The somewhat mysterious nature of AI-generated feedback is one of many limitations to its use; yet our findings suggest that those limitations can be characterized, documented, and potentially addressed or compensated for through instructor interventions.