This paper addresses the anomaly detection problem for videosurveillance. Due to the inherent rarity and heterogeneity of abnormal events, this problem is tackled from a normality modeling perspective, where our model learns object-centric normal patterns without seeing anomalous samples during training. Our main contributions consist in coupling objectlevel action features with a cosine distance-based anomaly estimation function. We therefore extend previous methods by introducing explicit geometric constraints to the mainstream reconstruction-based strategy. Our framework leverages both appearance and motion information to learn object-level behavior and captures prototypical patterns within a memory module. Experiments on several well-known datasets demonstrate the effectiveness of our method as it outperforms current state-of-the-art on most relevant spatio-temporal evaluation metrics.