HLS benchmarks by pepeiborra · Pull Request #3117 · haskell/haskell-language-server

pepeiborra · 2022-08-20T01:12:21Z

Port the ghcide benchmark suite to HLS, adding support to benchmark plugin "configurations" independently.

This includes the following changes to the ghcide benchmark suite:

Support for "configurations" which are defined as sets of plugin ids.
The benchmark will be run with only these plugins enabled and all others disabled
Support for configurable concurrency.

fendor · 2022-08-20T12:40:00Z

    cpp-options: -Dhls_brittany

-executable haskell-language-server
+library plugins


this might make it impossible to use stack for loading HLS into HLS. I.e. HLS needs to be configured to use cabal for development with this change due to commercialhaskell/stack#4564

Not a biggie, just fyi

Finally time to drop the stack descriptors and the Circle CI

michaelpj · 2022-08-21T08:55:45Z

Great!

Is there a reason not to delete the ghcide benchmarking code? Would make the diff simpler too, since I assume the benchmark runner fails are copies of the existing ghcide ones, and then it would just be a move.

pepeiborra · 2022-08-21T09:56:24Z

Great!

Is there a reason not to delete the ghcide benchmarking code? Would make the diff simpler too, since I assume the benchmark runner fails are copies of the existing ghcide ones, and then it would just be a move.

I am actively deleting it. But maybe I missed some code.

pepeiborra · 2022-08-21T13:49:07Z

This is ready for review. CI is setup to run benchmarks for the plugin configurations defined in bench/config.yaml:

None: no plugins
All
Ghcide: just the ghcide plugins
Core: Ghcide plugins + call hierarchy + code range + eval + pragmas. This is an arbitrary selection
: configurations for each individual plugin, excluding formatting plugins since none of the benchmark experiments exercise STextDocumentFormatting.

haskell-language-server/bench/config.yaml

Lines 83 to 161 in 78ffc49

    
           configurations: 
        
           - None: [] 
        
           - Core: 
        
             - callHierarchy 
        
             - codeRange 
        
             - eval 
        
             - ghcide-code-actions-bindings 
        
             - ghcide-code-actions-fill-holes 
        
             - ghcide-code-actions-imports-exports 
        
             - ghcide-code-actions-type-signatures 
        
             - ghcide-completions 
        
             - ghcide-type-lenses 
        
             - pragmas 
        
           - Ghcide: 
        
             - ghcide-code-actions-bindings 
        
             - ghcide-code-actions-fill-holes 
        
             - ghcide-code-actions-imports-exports 
        
             - ghcide-code-actions-type-signatures 
        
             - ghcide-completions 
        
             - ghcide-type-lenses 
        
           - All: 
        
             - alternateNumberFormat 
        
             - callHierarchy 
        
             - changeTypeSignature 
        
             - class 
        
             - codeRange 
        
             - eval 
        
             - explicitFixity 
        
             - floskell 
        
             - fourmolu 
        
             - gadt 
        
             - ghcide-code-actions-bindings 
        
             - ghcide-code-actions-fill-holes 
        
             - ghcide-code-actions-imports-exports 
        
             - ghcide-code-actions-type-signatures 
        
             - ghcide-completions 
        
             - ghcide-type-lenses 
        
             - hlint 
        
             - importLens 
        
             - moduleName 
        
             - ormolu 
        
             - pragmas 
        
             - qualifyImportedNames 
        
             - refineImports 
        
             - rename 
        
             - stylish-haskell 
        
           - alternateNumberFormat 
        
           - brittany 
        
           - callHierarchy 
        
           - changeTypeSignature 
        
           - class 
        
           - codeRange 
        
           - eval 
        
           - explicitFixity 
        
           - floskell 
        
           - fourmolu 
        
           - gadt 
        
           - ghcide-code-actions-bindings 
        
           - ghcide-code-actions-fill-holes 
        
           - ghcide-code-actions-imports-exports 
        
           - ghcide-code-actions-type-signatures 
        
           - ghcide-completions 
        
           # - ghcide-core                    # implicitly included in all configurations 
        
           # - ghcide-hover-and-symbols       # implicitly included in all configurations 
        
           - ghcide-type-lenses 
        
           - haddockComments 
        
           - hlint 
        
           - importLens 
        
           - moduleName 
        
           - ormolu 
        
           - pragmas 
        
           - qualifyImportedNames 
        
           - refineImports 
        
           - rename 
        
           - retrie 
        
           - splice 
        
           - stan 
        
           - stylish-haskell 
        
           - tactics

The results show that hls-hlint-plugin is, unsurprisingly, the most expensive plugin, followed closely by Tactics.

There is certainly margin for improvement:

Generated configurations: currently the list of configurations needs to be manually updated when new plugins are added (or removed).
Conditional configurations: the 9.2 run is benchmarking a lot of plugins that are not included in the 9.2 build of HLS. These configurations are equivalent to the None configuration.
Sharding: currently it takes CI ~4h to run all the configurations for the Cabal example, ideally we would split the work across multiple machines.

pepeiborra · 2022-08-21T13:58:09Z

Example successful runs:

michaelpj · 2022-08-21T14:03:03Z

I am actively deleting it. But maybe I missed some code.

Excellent. There are still some files in ghcide/bench, though?

pepeiborra · 2022-08-21T14:29:59Z

I am actively deleting it. But maybe I missed some code.

Excellent. There are still some files in ghcide/bench, though?

The ghcide test suite references them, so a bit more work is required to move them.

To be more precise, we have:

The benchmark experiment definitions, which are in a library for reuse in the ghcide test suite.
The ghcide-bench executable, ~~which could be renamed to lsp-bench since it's not that ghcide specific anymore.~~

Both things could be moved to ~~a standalone package lsp-bench with executable and test suite stanzas~~ the haskell-language-server package for clarity.

pepeiborra · 2022-08-21T18:38:59Z

I took a look and ghcide-bench uses code from the ghcide test suite (D.I.Test.Diagnostic) that is quite tricky to extract. The path would be:

move all the ghcide support test code to hls-test-utils,
move ghcide-bench to its own package.

Part 1 is a bit scary, I don't know how much work it will be.

michaelpj · 2022-08-22T08:58:21Z

The path would be:

Yeah, step 1 looks scary, especially because then the ghcide tests would depend on hls-test-utils which depends on ghcide? Not sure if that would work.

Would we need to do step 2? Wouldn't we just fold it into the HLS benchmark suite?

michaelpj

Haven't looked at the actual benchmarking code, I trust you on that, just superficial comments.

michaelpj · 2022-08-22T09:06:19Z

-    ]
+tests = testGroup "diagnostics providers" [ warningTests ]

-basicTests :: TestTree


seems marginally useful, are we just losing this?

We are losing the example plugins. There's hardly any point when we have a nice tutorial and 25? existing plugins that can serve as example

👍 Might as well delete their source too then?

michaelpj · 2022-08-22T09:06:54Z

Looks like you're going ahead with a ghcide-bench package.

pepeiborra · 2022-08-24T12:15:47Z

Looks like you're going ahead with a ghcide-bench package.

Yeah, but I avoided the scary step 1 by duplicating some things from the ghcide test suite. Can be deleted when/if they are extracted to a helper package

The main goal here is to move the Plugins module into an internal library so that it can be reused from the benchmark suite. In order to make that easier, and since they hardly serve a purpose in a repository with 25 plugins, I delete the Example and Example2 plugin descriptors and their dependencies.

Port the ghcide benchmark suite to HLS and benchmark plugin "configurations" independently. This includes the following changes to the ghcide benchmark suite and HLS: - Support for "configurations" which are defined as sets of plugin ids. The benchmark will be run with only these plugins enabled and all others disabled - Support for configurable concurrency. This relies on RTS -ol and -po flags to place the RTS traces in the target location rather than in the cwd This change requires two commits, the next one places ghcide/bench/hist/Main.hs into its final location to help 'git' recognize the change as a file move

Goal is to display the formatted CSV (via column) one row per line

We parse maxResidency and allocatedBytes from the RTS -S output, but runSessionWithHandles kills the server without waiting for it to exit and these stats don't get logged. The solution is to use runSessionWithHandles', but unfortunately it is internal and not exposed. I have raised a PR to expose it and in the meantime we need a source repo package.

* extract ghcide:experiments-types * extract haskell-language-server:plugins and let go of examples The main goal here is to move the Plugins module into an internal library so that it can be reused from the benchmark suite. In order to make that easier, and since they hardly serve a purpose in a repository with 25 plugins, I delete the Example and Example2 plugin descriptors and their dependencies. * HLS benchmark suite Port the ghcide benchmark suite to HLS and benchmark plugin "configurations" independently. This includes the following changes to the ghcide benchmark suite and HLS: - Support for "configurations" which are defined as sets of plugin ids. The benchmark will be run with only these plugins enabled and all others disabled - Support for configurable concurrency. This relies on RTS -ol and -po flags to place the RTS traces in the target location rather than in the cwd This change requires two commits, the next one places ghcide/bench/hist/Main.hs into its final location to help 'git' recognize the change as a file move * ghcide/bench/hist/Main.hs -> bench/Main.hs * CI - fix artifact names for uniqueness * disable shorten HLS step * Do not store eventlogs to avoid out of disk space * render durations up to milliseconds * shorten titles Goal is to display the formatted CSV (via column) one row per line * exclude formatting plugin configurations * Extract ghcide-bench to a standalone package * ghcide-bench: fix stderr capturing * Fix mem stats We parse maxResidency and allocatedBytes from the RTS -S output, but runSessionWithHandles kills the server without waiting for it to exit and these stats don't get logged. The solution is to use runSessionWithHandles', but unfortunately it is internal and not exposed. I have raised a PR to expose it and in the meantime we need a source repo package. * feedbacks * delete Example plugins

fendor · 2022-10-06T16:24:44Z

PR seems to have removed the example plugins, an accident?

edit: i see, not an accident

fendor reviewed Aug 20, 2022

View reviewed changes

pepeiborra force-pushed the hls-benchmarks branch 4 times, most recently from 3ca8f60 to bc907e9 Compare August 21, 2022 07:08

pepeiborra marked this pull request as ready for review August 21, 2022 08:08

pepeiborra requested review from Ailrun, Anton-Latukha, berberman, jneira and michaelpj as code owners August 21, 2022 08:08

pepeiborra force-pushed the hls-benchmarks branch from 78ffc49 to b664c0b Compare August 21, 2022 13:41

pepeiborra force-pushed the hls-benchmarks branch from 17baae3 to 228d6a4 Compare August 21, 2022 13:57

pepeiborra commented Aug 21, 2022

View reviewed changes

Comment thread bench/Main.hs

pepeiborra mentioned this pull request Aug 21, 2022

Expose runSessionWithHandles' haskell/lsp#450

Merged

pepeiborra force-pushed the hls-benchmarks branch from 6ce2ae4 to 80e8622 Compare August 21, 2022 23:32

michaelpj approved these changes Aug 22, 2022

View reviewed changes

pepeiborra added 3 commits August 24, 2022 13:17

extract ghcide:experiments-types

92d5cf2

pepeiborra added 11 commits August 24, 2022 13:17

ghcide/bench/hist/Main.hs -> bench/Main.hs

fab46c2

CI - fix artifact names for uniqueness

aaf6490

disable shorten HLS step

0f07ee0

Do not store eventlogs to avoid out of disk space

a8b51fc

render durations up to milliseconds

2645de3

shorten titles

72af303

Goal is to display the formatted CSV (via column) one row per line

exclude formatting plugin configurations

f290046

Extract ghcide-bench to a standalone package

c5b58dd

ghcide-bench: fix stderr capturing

36395db

feedbacks

5005793

pepeiborra force-pushed the hls-benchmarks branch from 80e8622 to 5005793 Compare August 24, 2022 12:17

delete Example plugins

5d1a7c0

pepeiborra merged commit d0e3e0f into master Aug 25, 2022

Uh oh!

Uh oh!

Conversation

pepeiborra commented Aug 20, 2022 • edited by gitpod-io Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fendor Aug 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pepeiborra Aug 20, 2022

Choose a reason for hiding this comment

Uh oh!

michaelpj commented Aug 21, 2022

Uh oh!

pepeiborra commented Aug 21, 2022

Uh oh!

pepeiborra commented Aug 21, 2022

Uh oh!

pepeiborra commented Aug 21, 2022

Uh oh!

Uh oh!

michaelpj commented Aug 21, 2022

Uh oh!

pepeiborra commented Aug 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pepeiborra commented Aug 21, 2022

Uh oh!

michaelpj commented Aug 22, 2022

Uh oh!

michaelpj left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

michaelpj Aug 22, 2022

Choose a reason for hiding this comment

Uh oh!

pepeiborra Aug 24, 2022

Choose a reason for hiding this comment

Uh oh!

michaelpj Aug 24, 2022

Choose a reason for hiding this comment

Uh oh!

michaelpj commented Aug 22, 2022

Uh oh!

pepeiborra commented Aug 24, 2022

Uh oh!

fendor commented Oct 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pepeiborra commented Aug 20, 2022 •

edited by gitpod-io Bot

Loading

fendor Aug 20, 2022 •

edited

Loading

pepeiborra commented Aug 21, 2022 •

edited

Loading

fendor commented Oct 6, 2022 •

edited

Loading