.NET Fest 2017. Игорь Кочетов. Классификация результатов тестирования производительности с помощью Machine Learning

Editor's Notes

• #5: Instead of programming some rules we feed training data (learning examples) into algorithm and access results
• #7: Web data (click-stream or click through data) Mine to understand users better Huge segment of silicon valley Self customizing programs Netflix Amazon iTunes genius Take users info Learn based on your behavior Next - types of learning tasks
• #8: Unsupervised - unlabeled data. Given the data find patterns and structure in the data Anomaly Detection (Fraud detection, Manufacturing, DataCenter monitoring) Anomaly detection vs. supervised learning: very small number of positive examples Content based recommendation and Collaborative filtering (if we have a set of features for movie rating you can learn a user's preferences, and vice versa, If you have your users preferences you can therefore determine a film's features) More examples: cocktail party algorithmMore details on Recommender Systems:Recommender systems typically produce a list of recommendations in one of two ways – through collaborative and content-based filtering or the personality-based approach.[7] Collaborative filtering approaches build a model from a user's past behaviour (items previously purchased or selected and/or numerical ratings given to those items) as well as similar decisions made by other users. This model is then used to predict items (or ratings for items) that the user may have an interest in.[8] Content-based filtering approaches utilize a series of discrete characteristics of an item in order to recommend additional items with similar properties.[9] These approaches are often combined (see Hybrid Recommender Systems).
• #10: Each test run provides us with decimal value as a result: milliseconds needed to complete. So we have a historic data for every measured feature and what to know if it increases, decreases, stays the same or jumps all around.
• #11: Our problem could be modeled as Handwriting recognition one
• #12: Every image is just an array of numbersWhich we feed into an algorithm (i.e. input) And the output is one of 10 digits Which brings us back to our problem:
• #13: Brain Does loads of crazy things Hypothesis is that the brain has a single learning algorithm Neuron: Three things to notice Cell body Number of input wires (dendrites) Output wire (axon) Simple level Neuron gets one or more inputs through dendrites Does processing Sends output down axon
• #14: a neuron is a logistic unit That logistic computation is just like logistic regression hypothesis calculation X vector is our input (X0 is a constant, known as bias) Ɵ vector is our parameters which may also be called the weights of a model (that’s what we want to learn)
• #15: This is the sigmoid function, or the logistic function Crosses 0.5 at the origin, then flattens out, Asymptotes at 0 and 1 Which gives us DECISION BOUNDARY When using linear regression we did hθ(x) = (θT x) For classification hypothesis representation we do hθ(x) = g((θT x)) Where we define g(z) z is a real number g(z) = 1/(1 + e-z)
• #16: It could be more than a line, actually
• #17: In order to achieve that we can apply higher order polynomial or use NN
• #18: First layer is the input layer Final layer is the output layer - produces value computed by a hypothesis Middle layer(s) are called the hidden layers ai(j) - activation of unit i in layer j Ɵ(j) - matrix of parameters controlling the function mapping from layer j to layer j + 1 Every input/activation goes to every node in following layer
• #19: NN is a logistic regression at scale Neural networks learning its own features!!!!! ai(j) - activation of unit i in layer j Ɵ(j) - matrix of parameters controlling the function mapping from layer j to layer j + 1 Every input/activation goes to every node in following layer Next - multiclass
• #20: Recognizing stable, unstable, regression or progression Build a neural network with four output units Output a vector of four numbers 1 is 0/1 stable 2 is 0/1 unstable 3 is 0/1 regression 4 is 0/1 progression
• #21: Inputs = features Outputs = number of classification categories Flip back to explain forward and back propagation
• #23: We will use AForge.NET library. We have to prepare Inputs and Outputs, choose Activation function and Network Structure (number of nodes, layers) And train the network until error is small enough
• #24: Having single value to measure performance of the algorithm is really important So the first step is to compare labeled inputs with algorithm outputs and calculate %% of correct results
• #26: Jtrain Error on smaller sample sizes is smaller (as less variance to accommodate) So as m grows error grows Jcv Error on cross validation set When you have a tiny training set your generalize badly But as training set grows your hypothesis generalize better So cv error will decrease as m increases High bias e.g. setting straight line to data Jtrain Training error is small at first and grows Training error becomes close to cross validation So the performance of the cross validation and training set end up being similar (but very poor) Jcv Straight line fit is similar for a few vs. a lot of data So it doesn't generalize any better with lots of data because the function just doesn't fit the data The problem with high bias is because cross validation and training error are both high Also implies that if a learning algorithm as high bias as we get more examples the cross validation error doesn't decrease So if an algorithm is already suffering from high bias, more data does not help High variance e.g. high order polynomial Jtrain When set is small, training error is small too As training set sizes increases, value is still small But slowly increases (in a near linear fashion) Error is still low Jcv Error remains high, even when you have a moderate number of examples Because the problem with high variance (overfitting) is your model doesn't generalize An indicative diagnostic that you have high variance is that there's a big gap between training error and cross validation error If a learning algorithm is suffering from high variance, more data is probably going to help
• #27: Applying higher order polynomial (or complex NN)
• #30: Precision How often does our algorithm cause a false alarm? Of all patients we predicted have cancer, what fraction of them actually have cancer = true positives / # predicted positive = true positives / (true positive + false positive) High precision is good (i.e. closer to 1) You want a big number, because you want false positive to be as close to 0 as possible Recall How sensitive is our algorithm? Of all patients in set that actually have cancer, what fraction did we correctly detect = true positives / # actual positives = true positive / (true positive + false negative) High recall is good (i.e. closer to 1) You want a big number, because you want false negative to be as close to 0 as possible F1Score (fscore) = 2 * (PR/ [P + R]) Fscore is like taking the average of precision and recall giving a higher weight to the lower value Many formulas for computing comparable precision/accuracy values If P = 0 or R = 0 the Fscore = 0 If P = 1 and R = 1 then Fscore = 1 The remaining values lie between 0 and 1
• #31: Find average value (mean) and subtract and, then divide by the range (st deviation)
• #33: Don’t be afraid to try, even small projects could be fun and useful