In this paper we integrate spatial and temporal information, which are extracted separately from a video sequence, for indexing and retrieval purposes. We focus on two filter families that are suitable models of the human visual system for spatial and temporal information encoding. They are special cases of polynomial transforms that perform local decompositions of a signal. Spatial primitives are extracted using Hermite filters, which agree with the Gaussian derivative model of receptive field profiles. Temporal events are characterized by Laguerre filters, which preserve the causality constraint in the temporal domain. Integration of both models gives a spatio-temporal feature extractor based on early vision.. Results encourage our model for video indexing and retrieval.