This paper is focused on a central aspect in the design of a planned digital library for human movement, i.e. on the aspect of representation and recognition of human activity from video data. The method of representation is important since it has a major impact on the design of all the other building blocks of the system such as the user interface/query block or the activity recognition/storage block. This paper evaluates a representation method for human movement that is based on sequences of angular poses and angular velocities of the human skeletal joints, for storage and retrieval of human actions in video databases. The choice of a representation method plays an important role in the database structure, search methods, storage efficiency, et cetera. For this representation, the study develops a novel approach for complex human activity recognition by employing multi-dimensional indexing combined with temporal or sequential correlation. This scheme is then evaluated with respect to its efficiency in storage and retrieval. For the indexing, the study uses postures of humans in videos that are decomposed into a set of multidimensional tuples which are present the poses/velocities of human body parts, such as arms, legs and torso. Three novel methods for human activity recognition are theoretically and experimentally compared. The methods require only a few sparsely sampled human postures. Speed invariant recognition of activities is also achieved by eliminating the time factor and replacing it with sequence information. The indexing approach also provides robust recognition and an efficient storage/retrieval of all the activities in a small set of hash tables. (Contains 24 references.) (Author/AEF)