Inter-rater reliability, commonly assessed by intra-class correlation coefficient ICC, is an important index for describing the extent to which there is consistency amongst two or more raters in assigned measures. In organizational research, the data structure is often hierarchical and designs deviate substantially from the ideal of a balanced (fully crossed or nested) design. Also, often it is necessary to include covariates in the model, making it impossible to use traditional correlation-based or analysis of variance (ANOVA)-based methods for estimation of inter-rater reliability. We advocate the use of hierarchical (mixed effect model)-based methods, where variance components can be estimated by restricted maximum likelihood or Bayesian approaches. In this work, we use data from teacher hiring in Spokane public schools to demonstrate the usage of hierarchical (mixed effect) models to estimate inter-rater reliability and to demonstrate how reliability can be estimated with more complex data structures. We generally find low levels of inter-rater reliability, though this overall reliability varies according to whether the measure is assessed "across' or "within" schools, or even within job openings. We also find evidence that inter-rater reliability of some subcomponents of the hiring rubric varies according to the type of position to which applicants are applying, or according to whether the applicant is internal or from outside of the district. The direct effect of reliability on predictive power of the selection instrument is demonstrated and policy implications for public school hiring are discussed. [Additional support for this study was provided by the Czech Science Foundation through grant #15-15856Y.]