Idempotent methods: hide exceptions on retries if there is evidence the original operation succeeded

[mderka]

This issue follows up on #883. Some idempotent methods can report an exception when retries. This was observed for example within #808 and #816. The intuition is that there are cases when the socket connection reports a timeout while waiting for a server response. A subsequent retry then receives an error because the operation finished in the meantime.

We introduced a new attribute rejected in BaseServiceException which indicates if an error has been certainly refused by the server (for example when quota has been exceeded). We need to adjust error handling in all the modules to properly work with idempotent and rejected attributes. The discussion regarding create and delete operations in #883 should be kept in mind.

We established that 500-type errors should not be treated as rejected because we cannot guarantee that the server did not process anything. We should assume that we can get a 500 while the request has been processed and committed on the service side.

[aozarov]

Consider this scenario as an example of the problem:
If Dns create zone operation fails there are cases that we can't tell for sure if the service actually applied the request or not. In that cases a retry may trigger an "already-exists" exception.

Though in this case an exception will be thrown with or without a retry, a retry mask the original failure
and misreport as "already-exists".

In this particular case if we want to fix this issue we need to capture the "already-exists" error and do an extra RPC call to get the zone information. If the zone information matches the request we should return without an error otherwise we should propagate the "already-exists".

[garrettjonesgoogle]

I'm going to reclassify this as an enhancement. The error returned from a retry isn't technically "wrong", it's just maybe not really what the user would want.

[garrettjonesgoogle]

This has been added to our feature backlog: https://github.com/GoogleCloudPlatform/google-cloud-java/wiki/Feature-backlog . This issue will be closed but is linked in the backlog and can continue to be used for comment and discussion.