Skip to content

[pantsd] Add RunTracker stats.#5374

Merged
kwlzn merged 5 commits into
pantsbuild:masterfrom
twitter:kwlzn/pantsd_stats
Jan 23, 2018
Merged

[pantsd] Add RunTracker stats.#5374
kwlzn merged 5 commits into
pantsbuild:masterfrom
twitter:kwlzn/pantsd_stats

Conversation

@kwlzn

@kwlzn kwlzn commented Jan 23, 2018

Copy link
Copy Markdown
Member
[omerta pants (kwlzn/pantsd_stats)]$ ./pants -q list : --run-tracker-stats-local-json-file=test.json >/dev/null; cat test.json | jq '.pantsd_stats'

{
  "resident_graph_size": -1,
  "resulting_graph_size": 81
}
[omerta pants (kwlzn/pantsd_stats)]$ PANTS_ENABLE_PANTSD=True ./pants -q list : --run-tracker-stats-local-json-file=test.json >/dev/null; cat test.json | jq '.pantsd_stats'

{
  "resident_graph_size": 0,
  "resulting_graph_size": 81
}
[omerta pants (kwlzn/pantsd_stats)]$ PANTS_ENABLE_PANTSD=True ./pants -q list : --run-tracker-stats-local-json-file=test.json >/dev/null; cat test.json | jq '.pantsd_stats'

{
  "resident_graph_size": 15,
  "resulting_graph_size": 81
}
[omerta pants (kwlzn/pantsd_stats)]$ PANTS_ENABLE_PANTSD=True ./pants -q list :: --run-tracker-stats-local-json-file=test.json >/dev/null; cat test.json | jq '.pantsd_stats'

{
  "resident_graph_size": 15,
  "resulting_graph_size": 14634
}
[omerta pants (kwlzn/pantsd_stats)]$ PANTS_ENABLE_PANTSD=True ./pants -q list :: --run-tracker-stats-local-json-file=test.json >/dev/null; cat test.json | jq '.pantsd_stats'

{
  "resident_graph_size": 6198,
  "resulting_graph_size": 14634
}

Fixes #4644

@kwlzn kwlzn force-pushed the kwlzn/pantsd_stats branch from 2995776 to b8ada86 Compare January 23, 2018 01:49
@stuhood stuhood added this to the 1.4.x milestone Jan 23, 2018
@kwlzn kwlzn requested review from ity and stuhood January 23, 2018 07:12

@illicitonion illicitonion left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! :)

self._replace_targets(target_roots)
self._invalidation_report = invalidation_report
# TODO(#4769): This should not be exposed to anyone.
# Note that the Context created in unit tests by BaseTest uses a different codepath.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This note is probably still useful...

(I think @stuhood had plans to unify them? :))

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that work should already be tracked in #4401 fwict

Comment thread src/python/pants/goal/context.py Outdated
return self._scm

@property
def scheduler(self):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ity FYI, this may be useful :)

@stuhood stuhood Jan 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If possible, it would be good not to expose this as a public-seeming API, and continue to access the private field (or to expose a method that closes over the entire "get the graph len and record it in the runtracker" operation). The goal of #4769 is to give Tasks the ability to request that things run in the scheduler in a declarative way. Having imperative access is supposed to just be a stopgap.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sgtm - fixed.

Comment thread src/python/pants/goal/pantsd_stats.py Outdated
self.resident_graph_size = None
self.resulting_graph_size = None

def set_resident_graph_size(self, size):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/resident/already_resident/?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

renamed to preceding_graph_size

@stuhood stuhood left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Kris.

Comment thread src/python/pants/goal/context.py Outdated
return self._scm

@property
def scheduler(self):

@stuhood stuhood Jan 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If possible, it would be good not to expose this as a public-seeming API, and continue to access the private field (or to expose a method that closes over the entire "get the graph len and record it in the runtracker" operation). The goal of #4769 is to give Tasks the ability to request that things run in the scheduler in a declarative way. Having imperative access is supposed to just be a stopgap.

@kwlzn kwlzn merged commit 84c4de7 into pantsbuild:master Jan 23, 2018
stuhood pushed a commit that referenced this pull request Jan 24, 2018
[omerta pants (kwlzn/pantsd_stats)]$ ./pants -q list : --run-tracker-stats-local-json-file=test.json >/dev/null; cat test.json | jq '.pantsd_stats'

{
  "resident_graph_size": -1,
  "resulting_graph_size": 81
}
[omerta pants (kwlzn/pantsd_stats)]$ PANTS_ENABLE_PANTSD=True ./pants -q list : --run-tracker-stats-local-json-file=test.json >/dev/null; cat test.json | jq '.pantsd_stats'

{
  "resident_graph_size": 0,
  "resulting_graph_size": 81
}
[omerta pants (kwlzn/pantsd_stats)]$ PANTS_ENABLE_PANTSD=True ./pants -q list : --run-tracker-stats-local-json-file=test.json >/dev/null; cat test.json | jq '.pantsd_stats'

{
  "resident_graph_size": 15,
  "resulting_graph_size": 81
}
[omerta pants (kwlzn/pantsd_stats)]$ PANTS_ENABLE_PANTSD=True ./pants -q list :: --run-tracker-stats-local-json-file=test.json >/dev/null; cat test.json | jq '.pantsd_stats'

{
  "resident_graph_size": 15,
  "resulting_graph_size": 14634
}
[omerta pants (kwlzn/pantsd_stats)]$ PANTS_ENABLE_PANTSD=True ./pants -q list :: --run-tracker-stats-local-json-file=test.json >/dev/null; cat test.json | jq '.pantsd_stats'

{
  "resident_graph_size": 6198,
  "resulting_graph_size": 14634
}

Fixes #4644
@stuhood

stuhood commented Jan 24, 2018

Copy link
Copy Markdown
Member

Leaving a note here for posterity: in order to cherry-pick this one, I needed to cherry-pick #5239 as well.

@stuhood stuhood removed the needs-cherrypick [CI] label Jan 24, 2018
stuhood added a commit that referenced this pull request Jan 24, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants