You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This is seemingly back to how it was with countsFrac being normalized but countsFracCurve not with Scottplot 4.1.11-beta?
Unless I'm being very stupid I'm getting the same behavior I described in issue #624, code below updated to 4.1.11
varplt=newScottPlot.Plot(600,400);Randomrand=newRandom(0);double[]values=DataGen.RandomNormal(rand,pointCount:1000,mean:50,stdDev:20);varhist=newScottPlot.Statistics.Histogram(values,min:0,max:100);varbar=plt.AddBar(hist.countsFrac,hist.bins);bar.BarWidth=hist.binSize*1.2;// oversize to reduce render artifactsbar.BorderLineWidth=0;plt.AddScatter(hist.bins,hist.countsFracCurve,lineWidth:2,markerSize:0);// decorate the plotplt.XLabel("Value (-)");plt.YLabel("Fraction (-)");plt.SetAxisLimits(yMin:0);plt.Grid(lineStyle:LineStyle.Dot);plt.SaveFig("stats_histogram.png");
I think the best solution is not to enable normalization on all histograms so much as make it toggleable, as quite often one does want a curve on the interval [0, 1].
Adding something like a countsFracNormalized as well as a countsCurve from #932 may be getting unwieldy. I wonder if there's a cleaner way to do this.
Perhaps we keep the behaviour where the curve is from [0, 1], and we add a function like ScottPlot.Tools.NormalizeTo() such that countsFracCurve becomes Tools.NormalizeTo(hist.countsFrac, 0, h) where h is the desired height of the mode of the normal distribution.
The height of the mode of the normal distribution is given by
which is approximately 0.4/sigma
Knowing this, one can find a multiplier such that we can arbitrarily rescale the curve.
However, this is all leading me to think that the best thing to do might be to move the distribution outside of histograms. We already have functions that can get summary statistics, and I think there's a lot of benefit in being able to plot distributions without having to make a histogram first. And I think if we have something like that we can add features like countsCurve and normalized countsFracCurve as well as other distributions like Students-T or Chi-square without cluttering the histogram code.
best thing to do might be to move the distribution outside of histograms
I definitely agree! There are lots of methods in the Tools module that should be in independent better-named modules. Similarly, statistic logic buried inside the Histogram and Population classes should be easier to access. Refactoring these classes and improving documentation has been added to the triage list (#716)
make it toggleable
Regardless of what the default behavior is (normalized or not), an ideal solution would be one where the user can define the intended behavior, and this should be in a way that is very easy to discover.
[Regarding a population statistics module] if we have something like that we can add features like countsCurve and normalized countsFracCurve as well as other distributions like Students-T or Chi-square without cluttering the histogram code
@bclehmann what level of interest do you have in creating such a module? I'm going to spend the next few days trying to push ScottPlot 4.1 out of beta and working on documentation, so I probably wouldn't get to this for some time. I'd love to model function arguments and documentation after numpy (e.g., numpy.histogram)
All these ideas are great and all but I was just pointing out that countsFrac and countsFracCurve should either both be normalized or both not normalized for consistency, just like it is in Scottplot 4.0, unless there's a good reason for the different behavior?
The cookbook example you linked uses 2 Y axes and plots counts not countsFrac, so it's kinda beside the point.
My only hesitation on creating a lot of code around distributions is that a lot of distributions have non-trivially implemented functions. For example the CDF of the normal distribution. And for all distributions the inverse of the CDF is very useful for getting random variables from that distribution (see inverse transform sampling), but it is almost always difficult to compute.
Now, all these functions have well known approximations, but my issue is if users are led to believe that they should use our implementation in performance or precision critical situations.
For example, I tried my hand at the inverse of the error function. I could get to accuracy within about 10^-5 with a weak approximation of erf(x) and a lot of Newton's method. mathnet-numerics has a lookup table of different polynomial approximations (presumably from the Taylor Series) and is accurate to about 10^-21 on an infinite precision datatype. Not to mention their version will be much much faster. It would be a massive increase in scope to attempt to provide high accuracy or high performance implementations.
So I guess I'm asking what the middle ground is. Just functions to provide random numbers from a distribution? Or a little more, as the PDF and CDF are useful for histograms.
The purpose of this comment is to document the existing functionality/language of the ScottPlot.Statistics.Histogram module.
In reviewing this topic I got confused myself, which is a strong indication that this module needs some work. It would greatly benefit from improved public field/property/method names and better XML documentation, and I've added this task to the triage list (#716)
Random Data
// create and plot normally-distributed random valuesRandomrand=new(0);intpointCount=10_000;double[]values=ScottPlot.DataGen.RandomNormal(rand,pointCount,mean:123,stdDev:12);varplt=newScottPlot.Plot(600,400);plt.AddScatterPoints(xs:ScottPlot.DataGen.Consecutive(pointCount),ys:values);
⚠️ These variable names are pretty bad - they don't clearly describe what they hold, and they don't have good XML documentation. It's not obvious which curves are ideal and which are based on actual data. I'll work to improve this, probably mimicking what matplotlib's histogram module does:
I made this a little better by adding a Histogram.probability field, but a true fix is to greatly refactor the Histogram module to give it more logical naming (and capitalization) across the board.
Activity
bclehmann commentedon Mar 31, 2021
I think the best solution is not to enable normalization on all histograms so much as make it toggleable, as quite often one does want a curve on the interval [0, 1].
Adding something like a
countsFracNormalizedas well as acountsCurvefrom #932 may be getting unwieldy. I wonder if there's a cleaner way to do this.Perhaps we keep the behaviour where the curve is from [0, 1], and we add a function like
ScottPlot.Tools.NormalizeTo()such thatcountsFracCurvebecomesTools.NormalizeTo(hist.countsFrac, 0, h)where h is the desired height of the mode of the normal distribution.The height of the mode of the normal distribution is given by

which is approximately 0.4/sigma
Knowing this, one can find a multiplier such that we can arbitrarily rescale the curve.
However, this is all leading me to think that the best thing to do might be to move the distribution outside of histograms. We already have functions that can get summary statistics, and I think there's a lot of benefit in being able to plot distributions without having to make a histogram first. And I think if we have something like that we can add features like
countsCurveand normalizedcountsFracCurveas well as other distributions like Students-T or Chi-square without cluttering the histogram code.swharden commentedon Apr 4, 2021
Hi @LB767, thanks for raising this issue! Connecting the dots here, the present behavior is demonstrated by this cookbook example: https://swharden.com/scottplot/cookbooks/4.1.11-beta/category/statistics/#histogram
I definitely agree! There are lots of methods in the
Toolsmodule that should be in independent better-named modules. Similarly, statistic logic buried inside theHistogramandPopulationclasses should be easier to access. Refactoring these classes and improving documentation has been added to the triage list (#716)Regardless of what the default behavior is (normalized or not), an ideal solution would be one where the user can define the intended behavior, and this should be in a way that is very easy to discover.
@bclehmann what level of interest do you have in creating such a module? I'm going to spend the next few days trying to push ScottPlot 4.1 out of beta and working on documentation, so I probably wouldn't get to this for some time. I'd love to model function arguments and documentation after numpy (e.g., numpy.histogram)
[-]Histogram countsFracCurve & countsFrac normalization[/-][+]Histogram normalization[/+]LB767 commentedon Apr 4, 2021
All these ideas are great and all but I was just pointing out that
countsFracandcountsFracCurveshould either both be normalized or both not normalized for consistency, just like it is in Scottplot 4.0, unless there's a good reason for the different behavior?The cookbook example you linked uses 2 Y axes and plots
countsnotcountsFrac, so it's kinda beside the point.See these lines for the inconsistency:
ScottPlot/src/ScottPlot/Statistics/Histogram.cs
Lines 70 to 72 in 1cc87b8
bclehmann commentedon Apr 5, 2021
My only hesitation on creating a lot of code around distributions is that a lot of distributions have non-trivially implemented functions. For example the CDF of the normal distribution. And for all distributions the inverse of the CDF is very useful for getting random variables from that distribution (see inverse transform sampling), but it is almost always difficult to compute.
Now, all these functions have well known approximations, but my issue is if users are led to believe that they should use our implementation in performance or precision critical situations.
For example, I tried my hand at the inverse of the error function. I could get to accuracy within about 10^-5 with a weak approximation of erf(x) and a lot of Newton's method. mathnet-numerics has a lookup table of different polynomial approximations (presumably from the Taylor Series) and is accurate to about 10^-21 on an infinite precision datatype. Not to mention their version will be much much faster. It would be a massive increase in scope to attempt to provide high accuracy or high performance implementations.
So I guess I'm asking what the middle ground is. Just functions to provide random numbers from a distribution? Or a little more, as the PDF and CDF are useful for histograms.
test to demonstrate histogram #930
more histogram demonstrations #930
swharden commentedon Apr 11, 2021
The purpose of this comment is to document the existing functionality/language of the
ScottPlot.Statistics.Histogrammodule.In reviewing this topic I got confused myself, which is a strong indication that this module needs some work. It would greatly benefit from improved public field/property/method names and better XML documentation, and I've added this task to the triage list (#716)
Random Data
Create a Histogram
Histogram Counts
Curves
Next Steps
improve Statistics.Histogram documentation
improve Histogram cookbook recipes
swharden commentedon Apr 12, 2021
I made this a little better by adding a
Histogram.probabilityfield, but a true fix is to greatly refactor the Histogram module to give it more logical naming (and capitalization) across the board.I also improved the cookbook recipes to make this more obvious: https://swharden.com/scottplot/cookbooks/4.1.12-beta/#statistics