Styler extremely slow

[N2ITN]

Code Sample

def highlighter(col):
    '''
    Highlights rows in `highlight_map` keys where df[`highlight map` values]==1
    '''

    highlight_map = {
        'Service Type': 'ServiceType_Added',
        'Store #': 'Store_Added',
    }
    if col.name in highlight_map:
        return ['background-color: yellow' if final[highlight_map[col.name]][v] else '' for v in col.index.tolist()]
    else:
        return [''] * len(col)
    
styled = final.style.apply(highlighter)
styled.to_excel('highlights.xlsx', engine='openpyxl') 

Problem description

Here I have some conditional highlighting on a df with 18k rows. The issue is that despite preceding complex operations on the df (such as conditional merges and df.apply by row) taking ~300ms at the most, the Styler.apply part takes over two minutes. I realize this feature is in development but I am wondering if there is a way to make it faster or if this is a known issue.

[TomAugspurger]

Is it Styler.apply that's slow, or your function, or the writer? Have you profiled it to get a sense for where the time being spent?

Note that Styler builds up a list of tasks lazily. To actually run it you'll have to call _compute, so you'll want to profile something like

styled = final.style.apply(highlighter)._compute()

[N2ITN]

Good point.
Here are some times:

Function scanning the df and writing new values final.apply(highlighter) 1.42s

The Styler by itself final.style.apply(highlighter)._compute() 1m 28.3s

The Styler and openpyxl writer final.style.apply(highlighter).to_excel('highlights.xlsx', engine='openpyxl')
2m 31s

[TomAugspurger]

final.style.apply(highlighter)._compute()

If you want, you can break that down into how much time is spent in pandas' Styler._apply vs. your function. I suspect the majority will be in your function, in which case there's nothing pandas can do to hep ;)

For the to_excel part, I'd be curious to see a breakdown between pandas and whatever engine is doing the writing. I'm not familiar with that bit of the code though I'd guess most of the time will be spent in the writer engine.

[N2ITN]

As stated above, unless I misunderstand, calling the function independently of the styler using df.apply(highlight) takes 1.42s as opposed to with the styler df.style.apply(highlight) which takes 1m 28s.

Both calls have the same amount of iteration over the original dataframe, the only difference being the latter creates a Style object.
Therefore I don't believe it is my function that is slowing the process.

[TomAugspurger]

Therefore I don't believe it is my function that is slowing the process.

Could you do some line profiling then? Or post a reproducible example? _update_context could be to blame, but it's hard to say for sure.

[TomAugspurger]

@N2ITN find any time to profile this?

[TomAugspurger]

Closing, let us know if you're able to profile things.

[itssimon]

I'm experiencing the same issue and did some profiling.

I have a very small DataFrame:

In [4]: df.shape
Out[4]: (78, 4)

Just adding a text-align: left property takes almost 1 second:

In [5]: %%time
    ...: s = df.style.set_properties(**{'text-align': 'left'})
    ...: s.render();
    ...: 
CPU times: user 844 ms, sys: 68 ms, total: 912 ms
Wall time: 833 ms

I ran the line profiler and pasted the output here. As suspected most time is spent in _update_ctx().

@TomAugspurger Could you please reopen the issue? I'm happy to provide more info, just let me know what is helpful!

[TomAugspurger]

Profiling would be helpful.

…

On Sat, Mar 30, 2019 at 5:52 AM Simon Gurcke ***@***.***> wrote: I'm experiencing the same issue and did some profiling. I have a very small DataFrame: In [4]: df.shape Out[4]: (78, 4) Just adding a text-align: left property takes almost 1 second: In [5]: %%time ...: s = df.style.set_properties(**{'text-align': 'left'}) ...: s.render(); ...: CPU times: user 844 ms, sys: 68 ms, total: 912 ms Wall time: 833 ms I ran the line profiler and pasted the output here <https://pastebin.com/raw/ShfAcVGx>. As suspected most time is spent in _update_ctx(). @TomAugspurger <https://github.com/TomAugspurger> Could you please reopen the issue? I'm happy to provide more info, just let me know what is helpful! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#19917 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIpgucxErZ7xaqt2nSz2arz6AAxpRks5vb0H_gaJpZM4ST6vu> .

[TomAugspurger]

Ah, I missed that you linked to some. Does set_properties update every cell in the table? It'd be nice to avoid the calls to get_indexer in those cases. ``` 484 312 1351736.0 4332.5 86.9 i = self.index.get_indexer([row_label])[0] 485 312 170846.0 547.6 11.0 j = self.columns.get_indexer([col_label])[0] ``` On Sat, Mar 30, 2019 at 6:33 AM Tom Augspurger <tom.augspurger88@gmail.com> wrote:

…

Profiling would be helpful. On Sat, Mar 30, 2019 at 5:52 AM Simon Gurcke ***@***.***> wrote: > I'm experiencing the same issue and did some profiling. > > I have a very small DataFrame: > > In [4]: df.shape > Out[4]: (78, 4) > > Just adding a text-align: left property takes almost 1 second: > > In [5]: %%time > ...: s = df.style.set_properties(**{'text-align': 'left'}) > ...: s.render(); > ...: > CPU times: user 844 ms, sys: 68 ms, total: 912 ms > Wall time: 833 ms > > I ran the line profiler and pasted the output here > <https://pastebin.com/raw/ShfAcVGx>. As suspected most time is spent in > _update_ctx(). > > @TomAugspurger <https://github.com/TomAugspurger> Could you please > reopen the issue? I'm happy to provide more info, just let me know what is > helpful! > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#19917 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/ABQHIpgucxErZ7xaqt2nSz2arz6AAxpRks5vb0H_gaJpZM4ST6vu> > . >

[itssimon]

Yes, it seems that way. Unless you specify the subset parameter.

[TomAugspurger]

I could imagine an alternative updater for special cases like updating every value. Alternatively we could change the per cell dict to a chain map and apply the table properties to a dict shared by each cell, if that is indeed the issue.

…

On Mar 30, 2019, at 06:53, Simon Gurcke ***@***.***> wrote: Yes, it seems that way. Unless you specify the subset parameter. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[mjh7]

Hi there,

I am still relatively new to this toolset (I'm loving pandas overall!), but I seem to be having the same issue with slow Stylers as other users in this thread... Just checking if there are any new developments?

In my medium-small df (6K rows by 10 cols), it takes almost 4 minutes to process a function that assigns css font-size using df.style.applymap(my_css_function), while df.applymap(my_css_function) is practically instantaneous. I can process particular slices faster, of course, but it's awkward that I can't re-use the result - it has to get re-computed every time and prints the entire structure with no 'head' method, etc.

For my purposes, the style idea is very relevant, but I'd prefer to define rules to control how whatever I'm looking at in my jupyter notebooks are rendered all the time, as opposed to working most of the time with one structure with limited readability and creating a separate structure (Styler) every time I want to see it more clearly/richly. I don't know how reasonable of a request that is, but unless I'm missing something or it gets a lot faster, the usefulness will be fairly low.

Thanks for your time!

Edit: Just found the "Limitations" section near the bottom of the documentation, it says: "No large repr, and performance isn’t great; this is intended for summary DataFrames". So maybe it's working as intended, formatting control and basic visualizations as a bonus as opposed to core functionality. I can support that. But maybe this "Limitations" section could be moved close to the top of the documentation page so people don't get the wrong idea?