A frequent criticism of exploratory factor analysis (EFA) is that it does not allow correlated residuals to be modelled, while they can be routinely specified in the confirmatory (CFA) model. In this article, we propose an EFA approach in which both the common factor solution and the residual matrix are unrestricted (i.e., the correlated residuals need not be specified a priori). The estimation procedures are two-stage and based on the unweighted least squares principle. Procedures for judging the solution appropriateness (including goodness of fit) are also proposed. The simulation studies and illustrative example suggest that the approach works quite well Although the proposal is based on existing results, most of the developments can be considered to be new contributions, and are expected to be particularly useful in the earlier stages of item calibration. The whole procedure has been implemented in both R language and a well-known non-commercial EFA program.