Skip to content

Fetching a document takes very long when it has a large map with a large index #351

@Cvratingen

Description

@Cvratingen

We have a database where we store information for some IOT devices.
Now our devices routinely send out a value every [5 or 60 or 300] seconds. I'm using a firestore (native mode) database to save these values.

I've got a program which aggregates the messages and updates all the documents every ~10 minutes with all messages received.
The structure of the database is as following:

- sensors (collection)
    - sensor1 (document)
        -  sensor_readings (subcollection)
            - readings_10-05-2021 (document)
            - readings_11-05-2021 (document)
            - readings_12-05-2021 (document)
    - sensor2 
    - etc. 

The document "readings_10-05-2021" will have a map with the values

- readings_10-05-2021
      - values (map)
             - "00:00:01" : 120
             - "00:01:01" : 121
             - "00:02:01" : 122
             - etc.

I'm trying to retreive this document with:

db.collection("sensors").document("sensor1").collection("sensor_readings").document("readings_10-05-2021").get()

When the map has only 1 value it takes about ~.19 seconds
When the map has 1440 values it takes about ~1.5 seconds
When the map has 19200 values it takes about ~400 seconds

I've tested this with documents of the roughly same size in bytes to account for download speeds.

note: fetching this document with the large map on the google cloud terminal takes ~1 second.

As advised by @samtstern posting this here.

Environment details

Python 3.9.2
pip 21.1.1
google-cloud-firestore 2.1.1

Metadata

Metadata

Assignees

Labels

api: firestoreIssues related to the googleapis/python-firestore API.priority: p2Moderately-important priority. Fix may not be included in next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions