feat: add Lineage metrics for CloudBigtableIO

[ad548]

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

• Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
• Ensure the tests and linter pass
• Code coverage does not decrease (if any source code was changed)
• Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> ☕️

If you write sample code, please follow the samples format.

[conventional-commit-lint-gcf]

🤖 I detect that the PR title and the commit message differ and there's only one commit. To use the PR title for the commit history, you can use Github's automerge feature with squashing, or use automerge label. Good luck human!

-- conventional-commit-lint bot
https://conventionalcommits.org/

[andre-sampaio]

Can you also add some brief documentation to https://github.com/googleapis/java-bigtable-hbase/blob/main/bigtable-dataflow-parent/bigtable-beam-import/README.md?

[andre-sampaio]

IIUC this would break existing workloads for projects without the lineage API enabled, is this correct?

If so we need to provide this as an option (probably defaulted to false). It might still be a good idea to have this behind a flag anyway

[ad548]

Can you also add some brief documentation to https://github.com/googleapis/java-bigtable-hbase/blob/main/bigtable-dataflow-parent/bigtable-beam-import/README.md?

That's a good idea! I added some documentation.

[ad548]

IIUC this would break existing workloads for projects without the lineage API enabled, is this correct?

If so we need to provide this as an option (probably defaulted to false). It might still be a good idea to have this behind a flag anyway

Good question. So this feature is implemented using Beam metrics that are always emitted by Beam (alongside any number of other metrics) and sent over to Dataflow backend. If lineage is not enabled Dataflow drops those metrics in backend. So it should be fine to always report these. Other systems/IOs are doing a similar thing.

[andre-sampaio]

IIUC this would break existing workloads for projects without the lineage API enabled, is this correct?
If so we need to provide this as an option (probably defaulted to false). It might still be a good idea to have this behind a flag anyway

Good question. So this feature is implemented using Beam metrics that are always emitted by Beam (alongside any number of other metrics) and sent over to Dataflow backend. If lineage is not enabled Dataflow drops those metrics in backend. So it should be fine to always report these. Other systems/IOs are doing a similar thing.

Ahhh got it, thank you for explaining! lgtm