Machine Learning 1
- 1. MuhammadGulraj BS GIKI,Pakistan MS UET Peshawar,Pakistan 1 Machine Learning Name: Muhammad Gulraj [email protected]
- 2. MuhammadGulraj BS GIKI,Pakistan MS UET Peshawar,Pakistan 2 K–Nearest neighbor (KNN) In machine learning and pattern recognition K - Nearest neighbor KNN is an algorithm (non- parametric) that can be used for classification and regression. It is among the simplest algorithms in machine learning. The output of the KNN depends on whether the algorithm is used for regression or classification. KNN classify an object using the majority vote/neighbor. The object is assigned to the class which has the most members near to the object.
- 3. MuhammadGulraj BS GIKI,Pakistan MS UET Peshawar,Pakistan 3 KNN is instance based learning or lazy learning. KNN is simple but powerful algorithm in which no training is required and new training example can be added easily. However it is slow and expensive having computation complexity of O (md). KNN has a slow run time performance but it can be improved significantly by removing redundant data, computing approximate distance or using pre-sort training examples into fast data structures. KNN can be used in handwritten character classification, content based image retrieval, intrusion detection and fault detection. Let’s consider a simple example of an object which have many sea bass and salmon as neighbors. Let’s assume that K = 3 which means that we will consider 3 nearest neighbors of the object. From the below image it is clear that the object have 2 sea bass and 1 salmon, so the KNN algorithm will classify the object as sea bass.
- 4. MuhammadGulraj BS GIKI,Pakistan MS UET Peshawar,Pakistan 4 KNN Algorithm is very simple. The training period consist of storing all instances and labels (Class labels). If feature selection has to be performed then n-fold cross validation should be used on training set. If we want to test a new instance X given a set Y, the following steps are needed. i. Compute the distance of X from each instance in set Y ii. Sort the distance in increasing order and pick the highest K elements. iii. Find the most repeated class in K nearest neighbor. We can implement the KNN algorithm in matlab for IRIS dataset. Summary of the script is as follows. i. Load iris data in matlab ii. Randomize the data for new iteration for new sets of data and training set iii. For every observation we compute the Euclidean distance iv. We compute the K nearest neighbor and store it in an array v. We assign the label for the lowest distance vi. In case there is a tie we will randomly assign a class label vii. Return the label of the class viii. Find confusion matrix Please find the code knniris.m in the assignment folder. KNN shows very good results with less number of classes and features.
- 5. MuhammadGulraj BS GIKI,Pakistan MS UET Peshawar,Pakistan 5 Correctly classified 149 Incorrectly classified 1 Mean error 0.008 Relative absolute error 1.9% Root mean squared error 0.009 Total instances 150 Confusion matrix a b c setosa 50 0 0 vericolor 0 50 0 virginica 0 0 49 The detailed analysis shows that the KNN classifier makes very few mistakes in a dataset that is simple, although not linearly separable.
- 6. MuhammadGulraj BS GIKI,Pakistan MS UET Peshawar,Pakistan 6 Bayes Classifier Bayesian Theorem (Bayes Rule, Bayes Law) is the result of mathematical manipulation of conditional probabilities in probability and statistics. Bayesian classification rule provides a mathematical rule for updating the existing belief when find new evidence. Mathematically we can show it as: This rule can be explained by a simple example of a new born who observe a sun set and wonders if the sun will rise again tomorrow. The new born will assign equal probabilities (0.5, 0.5) to both outcomes. When the sun rise next day, the probability of sun rise will increase from 0.5 to 0.66 and thus the child’s belief that the sun will rise again increases. This process will continue and the child’s belief that the sun will rise again, increases from fifty percent probability to a universal truth. Another example is, assume someone told you that he had a nice conversation with someone on a bus. Knowing nothing about the conversation, the probability that the person had a conversation with a woman is 50% and the probability that the conversation was with a man is 50%. Suppose that the person also told you that the conversational partner had long hair. Thus the probability that the conversational partner is a woman increases because most of women (75%) have long hair. Similarly more features or evidences can increase the existing belief and can help you in deciding whether the conversational someone was ‘man’ or a ‘woman’.
- 7. MuhammadGulraj BS GIKI,Pakistan MS UET Peshawar,Pakistan 7 We can apply this rule using the above mentioned mathematical formula. Let’s suppose A represent an event ‘person has cancer’ and even B represent an event ‘person is smoker’. Let’s suppose the probability of event A, i.e. ‘person has cancer’ P (A) = 0.1 (which means that 10 percent of patients entering the clinic have cancer) and the probability of a patient having cancer is a smoker is P (B) = 0.5, Using the previous data of patients, we can determined the probability of smokers having cancer P (B|A) = 0.8, Using these numbers the probability that a person has cancer and is a smoker increases from 0.1 to 0.16 which is a significant increase. This shows that after finding new evidence the probability varies significantly. The given dataset contains 150 instances, corresponding to three equally-recurrent species of iris plant (setosa, versicolour, virginica).
- 8. MuhammadGulraj BS GIKI,Pakistan MS UET Peshawar,Pakistan 8 The execution and result of bayes classifier is as follows which shows that a Naïve Bayes classifier makes less mistakes in a dataset that, although simple, is not linearly separable.
- 9. MuhammadGulraj BS GIKI,Pakistan MS UET Peshawar,Pakistan 9 K–Means Clustering K means clustering is a popular method used for vector quantization. K means clustering is basically the partition of X observations into K clusters. In K means clustering each observation belongs to the cluster having the nearest mean. K means clustering is an NP hard problem (Computationally difficult). Efficient heuristic algorithms can be used which converge rapidly to a local minimum. Assume a set of observations x1, x2, x3, x4 …. Xn, every observation is a vector of d- dimensions. The main aim of K means clustering is to partition the n observations into k sets (clusters) where (k<=n). S = S1, S2 , … Sk (to minimize the sum of squares within-cluster). μi is mean of points in Si.
- 10. MuhammadGulraj BS GIKI,Pakistan MS UET Peshawar,Pakistan 10 The algorithm of K means clustering is as follows. i. Specify K (the number of clusters) ii. Select K points randomly as cluster centers. iii. Assign each object/instance to closest cluster center. iv. Find the centroid or mean for every cluster and use it as new cluster center. v. Now reassign all the objects/instances to the closest cluster center. vi. Iterate until the center of the clusters doesn’t change any more.
- 11. MuhammadGulraj BS GIKI,Pakistan MS UET Peshawar,Pakistan 11 If we use the K means clustering on Iris data set, it will find the natural grouping between iris specimens based on the features given in the data. Using the K means algorithm we must first specify the number clusters we want to create. The Matlab implementation of K means algorithm is present in the assignment folder. The different results that we found using K means algorithm are as follows.
- 12. MuhammadGulraj BS GIKI,Pakistan MS UET Peshawar,Pakistan 12 Every Iteration that is performed by the K means algorithm, it reassigns the points between clusters to decrease the distance between centroids and recompute it for new clusters. In each iteration the re-assignment decreases until the algorithm reaches a minimum.
- 13. MuhammadGulraj BS GIKI,Pakistan MS UET Peshawar,Pakistan 13 The new clusters formed will be shown like the following figure. References
- 14. MuhammadGulraj BS GIKI,Pakistan MS UET Peshawar,Pakistan 14 1. http://documentation.statsoft.com/STATISTICAHelp.aspx?path=MachineLearning/MachineL earning/NaiveBayes/NaiveBayesClassifierExample1Classification 2. http://www.mathworks.com/help/stats/examples/classification.html 3. Andrew Ng (2013), an online course for Machine learning, Stanford University, Stanford, https://class.coursera.org/ml-004/class. 4. Duda and Hart, Pattern Classification (2001-2002), Wiley, New York. 5. Ying Cui and Zhong Jin, Facial feature points (2012), http://www.jprr.org/index.php/jprr 6. Ioannis Dimou and Michalis Zervakis, On the analogy of classifier ensembles, http://www.jprr.org/index.php/jprr