Natural environments provide a rich spatiotemporal context that allows for visual objects to be differentiated based on different types of information: their absolute or relative spatial or temporal coordinates, or their ordinal positions in a spatial or temporal sequence. Here, we investigated which spatial and temporal properties are incidentally encoded along with to-be-remembered features to provide reference frames in visual working memory (VWM). We tested the different possibilities in a spatiotemporal color change-detection task by transforming spatial and/or temporal structures of item presentation at retrieval relative to encoding. More precisely, spatial and/or temporal coordinates were (a) switched, changing the order of items in a spatial or temporal sequence (ordinal transformation); (b) multiplied by different factors, changing interitem distances (relational transformation); or (c) multiplied by a constant factor, expanding or shrinking the entire configuration (global transformation). Such transformations of the external reference frame at retrieval should only interfere with VWM if the internal reference frame relies on the spatial or temporal properties affected by the respective transformation. We found that ordinal and relational transformations of either the spatial or temporal structure impaired performance, whereas global transformations did not. Thus, reference frames appear to be primarily defined by interitem relations--including relative distances between items as well as their order--rather than absolute positions in space or time. These results corroborate and extend previous findings for the spatial domain, and highlight functional similarities of the spatial and temporal dimensions in VWM by revealing the same metrical properties for temporal reference frames.