SNMP Project: SNMP-based Network Anomaly Detection Using Clustering

EP2300 SNMP Project Report Amy Skinner (skinner@kth.se) - Laili Aidi (aidi@kthse)

1. Summary
This project aims to design and implement a system that is able to monitor the network using SNMP and
identify the specific possible attacks (DoS and port scan) using a cluster analysis. In the first task, the
program discovers the topology of the network. After successful discovery phase, it will be able to
monitor the link utilization (network link-states) for a specified period of time, and then detect the
anomaly, using k-means clustering scheme [1]. These anomalies will be analyzed to recognize the attack.
Moreover, this program also provides an advance feature, which is defined as optional task, as it
executes online monitoring and detects the attacks using Davies-Bouldin Index as quality scoring
measurement [2].

2. Software Design and MIB objects
A. The MIB objects which are used in this system are:
i. During network crawling System Group and Interface Group (Interfaces table), as listed below:
 sysName, OID 1.3.6.1.2.1.1.5. This MIB object is used to get the administratively assigned name
of the router
 ifIndex, OID 1.3.6.1.2.1.2.2.1.1. This MIB object is used to get the interface value of the router
 ifDescr, OID 1.3.6.1.2.1.2.2.1.2. This MIB object is used to get the description of the specific
interface that is discovered previously from the ifIndex MIB object request.
 ipAdEntIfIndex, OID 1.3.6.1.2.1.4.20.1.2. This MIB object represent the index that identifies the
interface to which it is applicable in the value ifIndex MIB object. Using this MIB Object, we can
identify the Interfaces that exist in the IP routing table of the Router.
 ipAdEntAddr, OID 1.3.6.1.2.1.4.20.1.1. This MIB object represents the IP address of the specific
interface of the Router.
ii. To discover the network topology, we identified the link level neighbor of each of the identified
Router using MIB Objects in Interface Group (IP Routing tables), which is the ipRouteNextHop, OID
1.3.6.1.2.1.4.21.1.7. This MIB object represents the next hop IP address of a route in the router.
iii. To identify the attacks, we used two MIB Objects in the Interface Group (Interfaces table) that relate
to interface utilization of a route, thus it able to represent the link-states of the network, as listed
below:
 ifInOctets, OID 1.3.6.1.2.1.2.2.1.10. This MIB object represents the total number of octets
received on the specific interface of the Router.
 ifInUcastPkts, OID 1.3.6.1.2.1.2.2.1.11. This MIB object represents the amount of unicast packets
delivered to a higher-layer protocol.

B. Below is the design of the software in this SNMP-based network management system, including the
classes, key data structures and operations. A full-size class diagram is given in Appendix 5A.
i. Class Start, the starting point to running the program. It contains the constant variables, used as
default parameters to run the specific task, if user has not specified with command line arguments.
ii. Class Router, represents the Managed node (Router), which contains:
 hostname, which is String data type containing the hostname of the node
 interfaces, which is Map of Integer (interface index) to RouterInf data structure containing
the interfaces of a router
 localIps, which is List of Strings containing the local IP addresses of a router
 neighborIps, which is List of Strings containing the neighbor (next-hop) IP addresses of a router

1


iii. Class RouterInf, represents the network interface of the router, which contains:
 The IP Address, called ip, which is String data type.
 The description, called desc, which is String data type.
iv. Class SNMPUtils, is static class that provide the SNMP values and operations that are needed to
accomplish the task, which are:
 OID, which is Map of Strings from the human readable OID names data type to Strings of OID
numeric values, for the MIB objects of which are requested during the run of this program
 open() and close(), opens and closes the SNMP session
 getVarBind(), returns SNMP getNext MIB variable-value binding (value with its OID)
 getVar(), returns SNMP getNext MIB value
v. Class SNMPCrawler, responsible for the node and link discovery task of the test-bed network:
 createRouter(), creates a router and add to global list of routers operation
 addInterfaces(), discovers and adds a list of the Interfaces of a router
 addNeigbors(), discovers and adds a list of the link level Neighbors of a router
vi. Class SNMPPoller, provides polling operation to capture the link-states of routers:
 poll and onlinePoll, operations used in Task 2, respective Task 3, to poll all routers for a
specified period of time and quit, or to continuously poll and call Clusterer after w polling
rounds.
 xRounds and yRounds, which is Hashtable of integer data type to List of Long data structure.
This Integer represent the round number, and the List of Long data structure contains the sum
of ifInOctets, respective ifInUcastPkts, from every interface of each router in each round
vii. Class PollingTread
This class has composition relationship with Class SNMPPoller, which polls the information of the
routers simultaneously in every round.
vii. Class Clusterer, is a Thread that provides clustering calculation based on k-means clustering
method and/or Davies-Bouldin Index and show the result. This class contains 2 data structures which
represent the global-state of the network in every round, and operations which are:
 deltXt and deltYt, the delta values of MIB object ifInOctets and MIB object ifInUcastPkts from
all routers in every round, calculated from the average value of the sum of MIB object from all
interfaces from all routers in round t
 cluster(), the cluster formation operation, which is used to perform clustering until the it is
convergence or reach the maximum iteration for convergence (10 iterations)
 getNewCentroids(), calculates the centroids from a list of type Cluster
 calcDbi(), DBI value operation, used to get the Davis-Boulman Index of the clusters in each
calculation for the same dataset.
 findAttacks(), identifies the DoS and port scan attacks.
vii. Class Cluster
This class has composition relationship with Class Clusterer, represent the cluster object, containing
 CentroidX and CentroidY, the X, respective Y values of the centroid
 Xs and Ys, holds all the X, respective Y values of all the points in this cluster
 getNumPoints, returns the number of points in this cluster

2


3. Clustering Algorithm and Anomaly Detection Scheme
/** Anomaly detection **/
delXt = createDeltas(createAverages(xRounds, "X"));
delYt = createDeltas(createAverages(yRounds, "Y"));
numDeltas = delXt.size();

List<Cluster> initClusters = new ArrayList<Cluster>();
while(initCentroids.size() < k) {
Cluster c = new Cluster(delXt.get(p),delYt.get(p));
initClusters.add(c);
} }

List<Cluster> clusters = cluster(initClusters);
List<Cluster> newCentroids = getNewCentroids(clusters);
int numIterations = 0;
while (!haveSameCentroids(clusters, newCentroids)) {
clusters = cluster(newCentroids);
newCentroids = getNewCentroids(clusters);
if (numIterations++ > MAX_CLUSTERING_ITERATIONS)
break;
} }
double dbi = computeDbi(clusters);

Our clustering algorithm is based off the instructions in the project description in sections 2.3b and 2.3c
[3]. For all of our calculations we have kept track of the values for the x value (the sum of the ifInOctets
MIB values for every interfaces on a given router) and the y value (the sum of the ifInUcastPkts MIB
values for every interface on a router) as separate variables, to the data structures as simple as possible,
since they both change and are operated on independent of each other. At the end of the polling phase
we have two tables that hold all of the polled values, xRounds and yRounds. These tables have the
polling round number as keys, and the values are lists of the x or y values from all routers that
responded with valid results for the corresponding polling round. This data, along with an integer
interval, specifying how often polling should occur, an integer k, to indicate the number of clusters
that should be created, and repeats, to indicate how many times we should recalculate the clusters for
a different time period, are the inputs to the clustering function.

The clustering algorithm begins by determining the average global state for each round, by summing up
all values the list for that round, and then dividing by the number of responses in the list. This number
can vary, if we have received a timeout when requesting a MIB value from a router. One of the biggest
design choices for this project was to decide how to handle these timeouts. If we receive a timeout
from a router while trying to get information about one of the interfaces, we do not add the information
received from the other interfaces to the list for the round, so there will be one less entry in the list from
this round. We have chosen to do this, because we assume that the null responses from the routers
occur independently of when an attack occurs (only as a result of too many students executing at one
time), so we do not want to bring down the global state average for that round and create something
that may look anomalous, but not because of an attack. Another facet of this decision was if we
received a null during a poll of the x value from a router, but not during a poll of the y value, should we
add the y sum value to the y list, even though we are not adding the x value to the x list? We have
decided that because we are only dealing with global averages in this project, and not with the
information from specific routers, that there is no reason why we cannot return one MIB sum value to
help calculate the average. All of these conditions can be seen in the run() method of the
PollingThread class, in SNMPPoller.java.

3


After averaging, we determine the changes in global state. This is done by finding the differences
between the averages, stored as deltXt and deltYt. Since we are storing all data for x and y
separately, there is the possibility that this lists could become different lengths. However, this could
only occur if we were to get null responses from all routers for one of the values, but not the other, in a
given round. We consider this to be incredibly unlikely, so we assume the size of x list to be the same as
the size of the y list.

In the next step, we run a loop to pick a new random points in the data set to be initial centroids (x(p),
y(p)). Then we make a loop though all of the rounds, and for each point (x(t),y(t)), we record the
Euclidean distance to all of the initial centroids. After calculating the distances, the point is then added
to a cluster with the centroid which is closest to itself. The clusters are stored as lists of type Cluster.
Then we compute the new centroid for each cluster. This step is repeated until we get the same
centroids after an iteration (convergence), or until a maximum number of iterations (10) is met. After
the clusters are created, the Davies-Bouldin index (DBI) is computed for each clustering round. This
entire clustering process is repeated repeats number of times, to compare values discovered from
clusterings with different random initial centroids.

/** Anomaly detection **/
int largestCentroidCluster = clusters.getClusterWithLargestCentroid();
long largestCentroid = clusters.get(LargestCentroidCluster).getCentroid();

int secondLargestCentroidCluster = clusters.getClusterWithLargestCentroid();
long secondLargestCentroid = clusters.get(LargestCentroidCluster).getCentroid();

if(clusters.get(largestCentroidCluster).getNumPoints() <
clusters.get(secondLargestCentroidCluster).getNumPoints()) {
print ("There was a DoS attack in cluster: t" + largestCentroidCluster + 1) -
Rounds: ");
for(int i: clusters.get(largestCentroidCluster).getRounds()) {
print(clustersRounds.get(largestCentroidCluster).get(i) + " "); }

print ("nThere was a port scan attack in cluster: " +
(secondLargestCentroidCluster + 1) - Rounds: ");
for(int i=0; i<clustersRounds.get(secondLargestCentroidCluster).size(); i++) {
print(clustersRounds.get(secondLargestCentroidCluster).get(i) + " "); }
}

The anomaly detection scheme (ADS) works by using the qualities listed in section 2.3d of the project
description. First, the top two clusters are picked by their centroid value, which means the two clusters
whose centroid is furthest from the origin. After that, we determine if attacks have happened, by
testing if the qualities of these two clusters agree with qualities laid out in the project description, that is
to say, that if the largest of the two centroids has a smaller size, we call this a DoS Attack, and can
therefore call the other cluster a port scan attack. If these two clusters do not share these qualities, we
consider that it is indeterminate whether there was an attack or not. This can happen due to poor
choice of random initial centroids which prohibit the clusters from forming in predictable ways. Our
clusterer takes a variable integer repeats, which controls how many times we repeat the calculations
with different initial random centroids, that we are more accurately able to say during which rounds
there may have been an attack. We have decided to run the ADS on all clusterings, rather than just the
clusterings with the lowest DBI, because we have found that the clusterings with the lowest DBI do not
always show the most accurate attack detection (see Section 4A). We do however determine and
output which clustering has the lowest DBI, to conform to the requirements of Task 3.

4


4. Analysis of Results
In this section we present plots of data produced from a run of the program in Task 2. Similar data is
created from every in Task 3, with the difference that the lowest DBI is identified, and only that data is
output to a file. For this run of the program , we have selected to do 15 clusterings to ensure a breadth
of different results, and selected 3 unique clusterings to discuss here. For the plots we output data to
files from our program and then use GnuPlot to create the images [4].
In section A, we see clustering round number 12. This round had the lowest DBI. However, we can see
that the clusters were not very evenly distributed. We can tell that Cluster 1 (only one point, the
centroid, so the red cross is covered by the light blue square), and Cluster 2 are likely anomalous, but
maybe shouldn’t be clustered as they are. This is due to poor random initialization of the centroids. As
the initial centroid became the only member of the Cluster 1 in the first iteration, even after subsequent
iterations, it remained the only member of the cluster. In section B, we see clustering round number 13,
with a somewhat higher DBI. In this round we still identify the attacks by the criteria given in section
2.3d of the project description, however by looking at the points, it appears that most of the anomalous
points are clustered into Cluster 2, causing the algorithm to identify Cluster 1 as another anomalous
cluster, even though many of its points appear to be in the normal range. In section C, we see clustering
round 15, with a moderately low DBI. In this clustering we have identified two anomalous looking
clusters, but were unable to identify either as a specific attack, because the cluster with the largest
centroid also had the greatest size (contrary to the criteria).

A. Output from clustering 12:

Calculation number: 12
Cluster 1: INITIAL CENTROID: (2441728310,207925532) CENTROID: (2441728310,207925532)
Distance to origin: 2147483647 size:1
DBI: 0.51
There was a DoS attack in cluster: 1 -Rounds: 9
There was a port scan attack in cluster: 3 -Rounds: 4 11 17 18

5


B. Output from clustering 13:

DBI: 0.76
There was a DoS attack in cluster: 2 -Rounds: 4 9 11 17 18
There was a port scan attack in cluster: 1 -Rounds: 0 2 3 5 6 7 8 10 12 14 16 19 20 30 35 36 38 58 72

C. Output from clustering 15:

DBI: 0.6
Unable to positively identify attacks due to cluster sizes and centroid values.

6


5. Appendix
A. UML Class diagram of the project

Figure 1. Class Diagram of the designed software
7


B. Console Output from the run of the program discussed in Section 4
[aidl@brooklyn src]$ java Start -t 2 -r 15 –o 1
Starting EP2300 SNMP assignment, Task 2 (Clustering Global States)
Beginning crawl at IP: 192.168.1.10 (default)
Crawled Router: R9
Interface (1): 192.168.1.10 FastEthernet0/0
Interface (3): null Null0
Neighbor: 192.168.4.14
Neighbor: 192.168.1.15
Crawled Router: R13
Neighbor: 192.168.14.1
Neighbor: 192.168.4.10
Crawled Router: R14
Neighbor: 192.168.1.10
Neighbor: 192.168.13.3
com.adventnet.snmp.snmp2.SnmpException: Time Synchronization has failed.
at com.adventnet.snmp.snmp2.usm.USMUserEntry.timeSynchronize(USMUserEntry.java:1185)
at com.adventnet.snmp.snmp2.usm.USMUtils.doTimeSync(USMUtils.java:2028)
at com.adventnet.snmp.snmp2.usm.USMUtils.doTimeSync(USMUtils.java:1927)
at com.adventnet.snmp.snmp2.usm.USMUtils.init_v3_parameters(USMUtils.java:1414)
at SNMPUtils.getVarBind(SNMPUtils.java:92)
at SNMPCrawler.addNeighbors(SNMPCrawler.java:112)
at SNMPCrawler.createRouter(SNMPCrawler.java:68)
at SNMPCrawler.start(SNMPCrawler.java:44)
at SNMPCrawler.<init>(SNMPCrawler.java:25)
at Start.main(Start.java:133)
Crawled Router: R0
Neighbor: 192.168.8.2
Neighbor: 192.168.14.14
Crawled Router: R2
Neighbor: 192.168.13.15
Neighbor: 192.168.12.4
Crawled Router: R1
Neighbor: 192.168.8.1
Neighbor: 192.168.0.11
Crawled Router: R3
Neighbor: 192.168.12.3
Neighbor: 192.168.9.9
Crawled Router: R10

8


Neighbor: 192.168.0.2
Neighbor: 192.168.7.8
Crawled Router: R8
Neighbor: 192.168.9.4
Neighbor: 192.168.10.12
Crawled Router: R7
Neighbor: 192.168.7.11
Neighbor: 192.168.3.5
Crawled Router: R11
Neighbor: 192.168.11.13
Neighbor: 192.168.10.9
Crawled Router: R4
Neighbor: 192.168.3.8
Neighbor: 192.168.5.6
Crawled Router: R12
Neighbor: 192.168.2.16
Neighbor: 192.168.11.12
Crawled Router: R5
Interface (4): 192.168.100.100 Loopback0
Neighbor: 192.168.5.5
Neighbor: 192.168.15.7
Crawled Router: R15
Neighbor: 192.168.6.7
Neighbor: 192.168.2.13
Crawled Router: R6
Neighbor: 192.168.15.6
Neighbor: 192.168.6.16
Crawling completed: 40s

[The network discovery of Task 1 is now completed and moving on to the polling of Task 2…]

Beginning polling...
Using interval : 2 seconds (default)
Using timespan : 180 seconds (default)
Using K-value : 4 (default)
Will repeat calculations : 15 times

9


[During the polling of Task 2 we frequently receive TimeSynchronizationExceptions outputted from the
SNMP class that we do not control. Also, we output every time we get a null response from a router.
We handle these nulls as discussed in Section 3. We receive many of these throughout the polling, so
we have edited away most of this output for simplicity, but here is a sample]

com.adventnet.snmp.snmp2.SnmpException: Discovery Failed
at com.adventnet.snmp.snmp2.SnmpEngineEntry.discoverSnmpEngineID(SnmpEngineEntry.java:698)
at com.adventnet.snmp.snmp2.usm.USMUtils.doDiscovery(USMUtils.java:1871)
at com.adventnet.snmp.snmp2.usm.USMUtils.init_v3_parameters(USMUtils.java:1413)
at SNMPUtils.getVar(SNMPUtils.java:142)
at PollingThread.run(SNMPPoller.java:197)
com.adventnet.snmp.snmp2.SnmpException: Failed to authenticate the SecurityParameters for user
2G1332_student SnmpEngineEntry not found for address 192.168.10.12 port 161
at com.adventnet.snmp.snmp2.Snmp3Message.processMessage(Snmp3Message.java:1132)
at com.adventnet.snmp.snmp2.SnmpSession.processPDUForVersion3(SnmpSession.java:2297)
at com.adventnet.snmp.snmp2.SnmpSession.setPDUParams(SnmpSession.java:2134)
at com.adventnet.snmp.snmp2.SnmpSession.send(SnmpSession.java:1974)
at com.adventnet.snmp.snmp2.SnmpSession.syncSend(SnmpSession.java:2558)
at SNMPUtils.getVar(SNMPUtils.java:142)
at PollingThread.run(SNMPPoller.java:197)
SNMP EXCEPTION ON IP: 192.168.10.12 OID: .1.3.6.1.2.1.2.2.1.10.1
NPE in Polling Thread.run() - X - Router: R11 round:0
RESULT IS NULL!!! TIMEOUT!!! IP: 192.168.0.2 OID: .1.3.6.1.2.1.2.2.1.10.1
NPE in Polling Thread.run() - Y - Router: R8 round:0

[End of sample of the errors… We now jump to poll completion, which begins with outputting the delta values
created from the global link-states we discovered during polling]

Done polling (74 rounds): 216s
Round: 1 Delta values (676248289,25047813)

10



[ After outputting the delta values, the clustering algorithm begins]

Cluster 1: INITIAL CENTROID: (75268897,148930723) CENTROID: (531160408,681158721) Distance to
origin: 863775770 size:2

11


Cluster 3: INITIAL CENTROID: (518732205,69707029) CENTROID: (1580738673,176220052) Distance
to origin: 1590530810 size:9
DBI: 0.6

DBI: 0.71
There was a DoS attack in cluster: 3
-Rounds: 4 9 11 17 18
There was a port scan attack in cluster: 2
-Rounds: 5 6 7 12 14 16 19 35 72

DBI: 0.6

DBI: 0.88

DBI: 0.59


12


DBI: 0.59

DBI: 0.72
-Rounds: 4 9 11 17 18
-Rounds: 0 2 3 5 6 7 8 10 12 14 16 19 20 30 35 36 38 58 72

DBI: 0.59

DBI: 0.69
-Rounds: 4 9 11 17 18
-Rounds: 5 6 7 12 14 16 19 35 72


13


DBI: 0.89
-Rounds: 4 9 11 17 18
-Rounds: 0 1 2 3 5 6 7 8 10 12 13 14 16 19 20 21 23 30 31 35 36 38 42 44 57 58 63 72

DBI: 0.63
-Rounds: 4 9 11 17 18
-Rounds: 5 6 7 12 16 35

DBI: 0.51
-Rounds: 9
-Rounds: 4 11 17 18

DBI: 0.76
-Rounds: 4 9 11 17 18
-Rounds: 0 2 3 5 6 7 8 10 12 14 16 19 20 30 35 36 38 58 72

DBI: 0.76

14


-Rounds: 4 9 11 17 18
-Rounds: 0 2 3 5 6 7 8 10 12 14 16 19 20 30 35 36 38 58 72

DBI: 0.6

The calculation with the mininum Davies-Bouldin Index occurred in clustering round: 12 with DBI: 0.51

[As mentioned in Section 4, we ran 15 calculations to ensure that we could find significantly unique clusterings
to discuss – ordinarily this is set to the default of 3 clusterings. In Task 3, we would have already begun the next
round of polling before clustering, and this would repeat every w rounds]

C. References
[1] k-means clustering http://en.wikipedia.org/wiki/K-means_clustering
[2] Davies–Bouldin index http://en.wikipedia.org/wiki/Davies–Bouldin_index
[3] EP2300 SNMP Project Description
http://www.s3.kth.se/lcn/courses/EP2300/snmp_project_2011.pdf
[4] GnuPlot http://www.gnuplot.info/

15

SNMP Project: SNMP-based Network Anomaly Detection Using Clustering

More Related Content

What's hot (17)

Viewers also liked (13)

Similar to SNMP Project: SNMP-based Network Anomaly Detection Using Clustering (20)

More from Laili Aidi (20)

Recently uploaded (20)

SNMP Project: SNMP-based Network Anomaly Detection Using Clustering