Most object detection frameworks rely on rectangular bounding boxes and recognizing object instances individually. However, the bounding box provides only a coarse localization of objects and the context information between objects is not fully utilized, which result in a degradation of classification performance. In this paper, combining a lightweight contextual attention module with the representation of pure regression points, we present a novel context-based pure regression object detector. Moreover, a threshold filter mask module is designed to speed up the detector by removing a few insignificant points and keeping meaningful positions. Nonetheless, both of them do not require handcrafted clustering or post-processing steps and are easy to embed in networks. The proposed contextual attention module and threshold filter mask not only improve detection performance, but also promote training speed. We show through experiments that the proposed context-based pure regression detector can improve the representation of the regression points method about 1.5–1.8 AP on the COCO test-dev detection benchmark.