Firestore error when paginating with document snapshot with 'null' field

Environment details

OS type and version: Windows 10
Python version: 3.6.8
google-cloud-firestore version: 2.1.3

Steps to reproduce

Try to paginate a query with a document snapshot and where clause.

Code example

def getPayments ():
    col = firestoreClient.collection('payments')
    query = col.where('paymentId', '==', None).limit(10).order_by('__name__')

    lastDoc = None
    for doc in query.stream():
        lastDoc = doc

    query = col.where('paymentId', '==', None).limit(10).order_by('__name__').start_after(lastDoc)

    lastDoc = None
    for doc in query.stream(): # Errors here
        lastDoc = doc

Stack trace

Traceback (most recent call last):
  File "\project-et4z2F\lib\site-packages\google\api_core\grpc_helpers.py", line 160, in error_remapped_callable
    return _StreamingResponseIterator(result, prefetch_first_result=prefetch_first)
  File "\project-et4z2F\lib\site-packages\google\api_core\grpc_helpers.py", line 83, in __init__      
    self._stored_first_result = six.next(self._wrapped)
  File "\project-et4z2F\lib\site-packages\grpc\_channel.py", line 426, in __next__
    return self._next()
  File "\project-et4z2F\lib\site-packages\grpc\_channel.py", line 826, in _next
    raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
        status = StatusCode.INVALID_ARGUMENT
        details = "order by clause cannot contain more fields after the key"
        debug_error_string = "{"created":"@1629750504.795000000","description":"Error received from peer ipv6:[2607:f8b0:4005:805::200a]:443","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"order by clause cannot contain more fields after the 
key","grpc_status":3}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./transactions/pay_publishers.py", line 359, in <module>
    main()
  File "./transactions/pay_publishers.py", line 353, in main
    payoutMultiplier=payoutMultiplier
  File "./transactions/pay_publishers.py", line 115, in payPublishers
    pendingPayouts = firestore_db.getPayments()
  File "C:\Users\Eric\workspace\arc-core\common\db\firestore_db.py", line 211, in getPendingPayouts
    for doc in query.stream():
  File "\project-et4z2F\lib\site-packages\google\cloud\firestore_v1\query.py", line 209, in stream    
    request=request, metadata=self._client._rpc_metadata, **kwargs,
  File "\project-et4z2F\lib\site-packages\google\cloud\firestore_v1\services\firestore\client.py", line 945, in run_query
    response = rpc(request, retry=retry, timeout=timeout, metadata=metadata,)
  File "\project-et4z2F\lib\site-packages\google\api_core\gapic_v1\method.py", line 145, in __call__  
    return wrapped_func(*args, **kwargs)
  File "\project-et4z2F\lib\site-packages\google\api_core\retry.py", line 290, in retry_wrapped_func  
    on_error=on_error,
  File "\project-et4z2F\lib\site-packages\google\api_core\retry.py", line 188, in retry_target        
    return target()
  File "\project-et4z2F\lib\site-packages\google\api_core\grpc_helpers.py", line 162, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.InvalidArgument: 400 order by clause cannot contain more fields after the key

On the 2nd query.stream(), i get this error: google.api_core.exceptions.InvalidArgument: 400 order by clause cannot contain more fields after the key.

I followed the pagination documentation, but added a where clause which broke it. If I remove the where clauses it works fine.

..After much trial and error, I got it working by changing .start_after(lastDoc) to start_after({ '__name__': lastDoc.id })

Not sure why that works but it does, the error went away.

I did some investigation and it seems particular to using the null type field.

Using empty strings and not null value works, so there may be something particular about ingesting when a field is a null type.

Also, stumbled on an interesting corner case where lastDoc was null that we can fix. Will open a separate bug for that. (#439)

Here is a standalone Repro though for this bug that should reproduce:

from google.cloud import firestore
import datetime

client = firestore.Client()
col = client.collection('0-test-collection-0')
col.add({"paymentId": None})

query = col.where('paymentId', '==', None).limit(10).order_by('__name__')

lastDoc = None
for doc in query.stream():
    print(doc)
    lastDoc = doc

query = col.where('paymentId', '==', None).limit(10).order_by('__name__').start_after(lastDoc)

lastDoc = None
for doc in query.stream(): # Errors here
    lastDoc = doc

@crwilcox

@crwilcox Any relationship here to your fix in PR #440?

#440 fixed #439 which was opened by me when I saw that issue related to this one. That part is resolved. The issue with null fields isn't though.

fix: unbreak query orders w/ non-orderable operators

fix: unbreak query orders w/ non-orderable operators (#453)

Navigation Menu

Repository navigation

Firestore error when paginating with document snapshot with 'null' field #429

Environment details

Steps to reproduce

Code example

Stack trace

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Firestore error when paginating with document snapshot with 'null' field #429

Description

Environment details

Steps to reproduce

Code example

Stack trace

Activity

guanzo commented on Sep 1, 2021

crwilcox commented on Sep 1, 2021

tseaver commented on Sep 21, 2021

crwilcox commented on Sep 22, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions