Linux System Programming - File I/O

Systems Programming
• Systems programming involves the development of the
individual pieces of software that allow the entire
system to function as a single unit.
• Systems programming involves many layers such as the
operating system (OS), firmware, and the development
environment.
• System programming concepts are varies based on
operating systems.
• Example:
– Windows OSWindows System programming
– Linux OSLinux System programming
– MAC OSMac System Programming

Cornerstones of System Programming
in LINUX
• system calls:
– System programming starts and ends with system
calls
– System calls are function invocations made from
user space—your text editor, favorite game, and
so on to requests a service from the kernel( core
part of OS)

in LINUX
• the C library:
– On modern Linux systems, the C library is provided
by GNU libc, abbreviated glibc.
– The GNU C Library project provides the core
libraries for the GNU system
– GNU is an operating system that is free software

in LINUX
• The C Compiler
– In Linux, the standard C compiler is provided by
the GNU Compiler Collection (gcc).
– A compiler is a special program that processes
statements written in a particular programming
language and turns them into machine language
or "code" that a computer's processor uses.

APIs & ABIs
• API
– It stands for Application Programming Interface
– It is a software intermediary that allows two applications to talk
to each other.
– It is based on source code
• ABI
– It stands for Application binary interface
– an ABI defines the binary interface between two or more pieces
of software on a particular architecture
– It defines how an application interacts with itself, how an
application interacts with the kernel, and how an application
interacts with libraries.
– It is based on machine code/object code

Opening Files
• A file is opened and a file descriptor is obtained
with the open() system call.
• Syntax:
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
int open (const char *name, int flags);/opens existing file
int open (const char *name, int flags, mode_t mode);/it
will create a new file if it does not exist/
Header Files

Opening Files
• int open (const char *name, int flags);
File name with path Specifies mode
int open (const char *name, int flags, mode_t mode);
File name with path O_CREAT Specifies mode

Opening Files
• The following are some of values for flag and
mode parameters:
1.O_RDONLY-----Open for reading only.
2.O_WRONLY-----Open for writing only.
3.O_RDWR---------Open for reading and writing.
4. O_CREAT---------Create the file if it doesn't exist.
5.O_EXCL----When used with O_CREAT, if the file already
exists it is an error and the open() will fail
6. O_APPEND------Open the file in append mode

Opening Files
• Example program:
#include<stdio.h>
#include<fcntl.h>
#include<errno.h>
int main()
{
// if file does not have in directory
// then file foo.txt is created.
int errno;
int fd = open("foo.txt", O_RDONLY);
printf("fd = %d/n", fd);
if (fd ==-1)
{
// print which type of error have in a code
printf("Error Number % dn", errno);
// print program detail "Success or failure"
perror("Program");
}
return 0;
}
Header files
Main function

Opening Files
• Fd:
– It stands for file descriptor
– File descriptor is integer that uniquely identifies an opened
file.
– A file descriptor is a non-negative integer, generally
represented in the C programming language as the type int
• Most of the functionsdeclared are in
the <fcntl.h> header ,<stdio.h> and <errno.h>
• Return 0 is the statement which returns 0 on the
success after finishing the program execution
and returning a non-zero number means failure

Opening Files
• Linux is the multi-user operating system which
can be accessed by many users simultaneously.
• But this raises security concerns as an unsolicited
user can corrupt, change or remove crucial data.
• For effective security, Linux divides authorization
into 2 levels:
– Ownership
– Permission

Opening Files
• Ownership:
– Every file and directory on your Unix/Linux system is
assigned 3 types of owner, given below:
• User
– A user is the owner of the file. By default, the person who created
a file becomes its owner. Hence, a user is also sometimes called
an owner.
• Group
– A user- group can contain multiple users. All users belonging to a
group will have the same access permissions to the file.
– Suppose you have a project where a number of people require
access to a file.
– Instead of manually assigning permissions to each user, you could
add all users to a group, and assign group permission to file such
that only this group members and no one else can read or modify
the files.
• Other
– Any other user who has access to a file
– Practically, it means everybody else.

Opening Files
• Permissions
– Every file and directory in your UNIX/Linux system
has following 3 permissions defined for all the 3
owners discussed above.
– The three permissions are
• Read:This permission give you the authority to open
and read a file
• Write:The write permission gives you the authority to
modify the contents of a file.
• Execute:This permission give you the authority to
execute a file

Opening Files
• Example for file permission:

Opening Files
• The creat() Function/System call:
– Create and open a file
• Syntax:
#include <sys/stat.h>
#include <fcntl.h>
int creat (const char *name, mode_t mode);
Header files
Name of file
Mode

Opening Files
• Return Values and Error Codes
– Both open() and creat() return a file descriptor on
success.
– On error, both return −1, and set errno to an
appropriate error value

Reading via read()
• The mechanism used for reading is the read()
system call.
• Syntax:
#include <unistd.h>
ssize_t read (int fd, void *buf, size_t len);
File descriptor Address of
first Byte
Length or count

Reading via read()
• When read() returns successfully, its return
value is the number of bytes actually read and
placed in the buffer.
• If len(third parameter) is zero, read returns
zero and has no other results.
• On success a non-negative integer is returned
indicating the number of bytes actually read.
• Otherwise, a -1 is returned.

Reading via read()
• A call to read() can result in many possibilities:
– The call returns a value equal to len
– The call returns a value less than len, but greater
than zero(if any interrupt occurs during read)
– The call returns 0. This indicates EOF. There is
nothing to read.
– The call blocks because no data is currently
available
– The call returns −1, and errno is set to EINTR or
EAGAIN.

Reading via read()
Nonblocking Reads
• By default, read() waits until at least one byte is
available to return to the application; this default
is called "blocking" mode.
• Alternatively, individual file descriptors can be
switched to "non-blocking" mode, which means
that a read() on a slow file will return
immediately, even if no bytes are available.
• This is called nonblocking I/O; it allows
applications to perform I/O, potentially on
multiple files, without ever blocking

Reading via read()
Other Error Values
• Possible errno values after a failure on read() include:
– EBADF
• The given file descriptor is invalid or is not open for reading.
– EFAULT
• The pointer provided by buf is not inside the calling process’s
address space.
– EINVAL
• The file descriptor is mapped to an object that does not allow
reading.
– EIO
• A low-level I/O error occurred.

Reading via read()
Size Limits on read()
• The size_t and ssize_t types are used for read()
• The size_t type is used for storing values used to
measure size in bytes.
• The ssize_t type is a signed version of size_t (the
negative values are used to connote errors).
• the maximum value of an ssize_t is SSIZE_MAX
which is 2,147,483,647 bytes on a 32- bit
machine

Writing with write()
• The most basic and common system call used
for writing is write().
• It writes data from a buffer declared by the
user to a given device, such as a file
• This is the primary way to output data from a
program by directly using a system call
• Syntax:
#include <unistd.h>
ssize_t write (int fd, const void *buf, size_t count);
File descriptor
Destination
Source Length of the data to
be written

• On success, the number of bytes written is
returned.
• On error, −1 is returned and errno is set
appropriately
• Partial writes:
– a successful write() may transfer fewer than count
bytes.
– Such partial writes can occur for various reasons
• If there was insufficient space on the disk device to write all
of the requested bytes
• Because of interrupt

• Append Mode:
– When fd is opened in append mode (via O_APPEND),
writes do not occur at the file descriptor’s current file
position. Instead, they occur at the current end of the
file.
• Nonblocking Writes
– When fd is opened in nonblocking mode (via
O_NONBLOCK), and the write as issued would
normally block, the write() system call returns −1 and
sets errno to EAGAIN
– The request should be reissued later

• Other Error Codes
– EBADF:The given file descriptor is not valid or is
not open for writing
– EFAULT:The pointer provided by buf points outside
of the process’s address space
– EINVAL:The given file descriptor is mapped to an
object that is not suitable for writing
– ENOSPC:The filesystem backing the given file
descriptor does not have sufficient space.

• Size Limits on write()
– If count is larger than SSIZE_MAX, the results of the call to
write() are undefined
– A call to write() with a count of zero results in the call
returning immediately with a return value of 0.
• Behavior of write()
– when a user-space application issues a write() system call,
the Linux kernel performs a few checks and then simply
copies the data into a buffer.
– Later, in the background,the kernel gathers up all of the
dirty buffers, which are buffers that contain data newer
than what is on disk, sorts them optimally, and writes
them out to disk

Synchronized I/O
User issues write() to write some
data which contains three lines to file
called Sample.txt located in your z
drive
Kernel or OS accept the request and collect data from
write
Time frame1:data is collected to buffer
Time Frame2:data is collected to buffer
Time Frame 3:data is Collected to buffer
And then
It will sort the data in buffers and in optimal time the
consolidated data will be moved to sample.txt located
in your z drive
request
Process

Synchronized I/O
• By default, the Linux kernel writes data to disk
asynchronously.
• Writes are buffered (cached) in memory, and
written to the storage device at the optimal
time.
• The Synchronous I/O provides some functions
to ensure that all operations are finish before
they return.

Synchronized I/O
• Sync System call:
– The sync system call forces an immediate write of all
cached data to disk but it doesn’t wait to complete.
– This call initiates the process of committing all buffers
to disk.
• Syntax:
sync [OPTION] [FILE]...
• Examples:
1. sync -d: This option sync only file data not
metadata
2. sync -f: This option will sync the file systems
which contains the files.

Synchronized I/O
fsync() and fdatasync()
• Syntax of fsync()
#include <unistd.h>
int fsync (int fd);
• The call to above function ensures that all dirty data
associated with the file mapped by the file descriptor
fd are written back to disk.
• The file descriptor fd must be open for writing.
• The call writes back both data and metadata, such as
creation timestamps and other attributes
• It will not return until the hard drive says that the data
and metadata are on the disk
• Metadata summarizes basic information about data.

Synchronized I/O
fsync() and fdatasync()
• Syntax for fdatasync()
#include <unistd.h>
int fdatasync (int fd);
• A call to fdata sync() will flush a file’s size,
since you need that to read the file correctly.
• The call does not guarantee that nonessential
metadata is synchronized to disk, and is
therefore potentially faster

Synchronized I/O
O_SYNC Flag and O_RSYNC
• O_SYNC requires that any write operations
block until all data and all metadata have been
written to persistent storage.
• The O_RSYNC flag specifies that only normal
data be synchronized after each write
operation, not metadata.

Direct I/O
• The Linux kernel, like any modern operating system
kernel, implements a complex layer of caching,
buffering, and I/O management between devices and
applications.
• A high-performance application may wish to bypass
this layer of complexity and perform its own I/O
management.
• Providing the O_DIRECT flag to open() instructs the
kernel to minimize the presence of I/O management.
• When this flag is provided, I/O will initiate directly from
user-space buffers to the device, bypassing the page
cache

Direct I/O
• All I/O will be synchronous; operations will not
return until completed.
• For an I/O operation to be performed as direct
I/O, it must meet certain alignment criteria.
• The alignment constraints are usually determined
by the disk driver, the disk controller, and the
system memory management hardware and
software.
• If a request fails to meet the alignment
constraints for direct I/O, the request is
performed as data synchronous I/O.

Closing Files
• After a program has finished working with a file
descriptor, it can unmap the file descriptor from the
associated file via the close() system call:
#include <unistd.h>
int close (int fd);
• A call to close() unmaps the open file descriptor fd and
disassociates the file from the process.
• It is a common mistake to not check the return value of
close().
• There are a handful of possible errno values on failure.
Other than EBADF the most important error value is
EIO, indicating a low-level I /O error probably unrelated
to the actual close

Seeking with lseek()
• lseek is a system call that is used to change
the location of the read/write pointer of a file
descriptor.
• Syntax:
#include <unistd.h>
off_t lseek (int fd, off_t pos, int origin);

• SEEK_CUR
– The current file position of fd is set to its current value
plus pos, which can be negative, zero, or positive. A
pos of zero returns the current file position value.
• SEEK_END
– The current file position of fd is set to the current
length of the file plus pos, which can be negative,
zero, or positive. A pos of zero sets the offset to the
end of the file.
• SEEK_SET
– The current file position of fd is set to pos. A pos of
zero sets the offset to the beginning of the file.

Error Values
• EBADF:The given file descriptor does not refer to
an open file descriptor
• EINVAL:The value given for origin is not one of
SEEK_SET, SEEK_CUR, or SEEK_END, or the
resulting file position would be negative.
• EOVERFLOW:The resulting file offset cannot be
represented in an off_t.
• ESPIPE:The given file descriptor is associated with
an unseekable object, such as a pipe,FIFO, or
socket.

Positional Reads and Writes
• In lieu of using lseek(), Linux provides two variants of
the read() and write() system calls
• Both receive the file position from which to read or write
• Upon completion, they do not update the file position
• The read form is called pread():
#define _XOPEN_SOURCE 500
#include <unistd.h>
ssize_t pread (int fd, void *buf, size_t count, off_t pos);
• This call reads up to count bytes into buf from the file
descriptor fd at file position pos.

Positional Reads and Writes
• The write form is called pwrite():
#define _XOPEN_SOURCE 500
#include <unistd.h>
ssize_t pwrite (int fd, const void *buf, size_t count, off_t pos);
• This call writes up to count bytes from buf to the file
descriptor fd at file position pos.

Truncating Files
• Linux provides two system calls for truncating the
length of a file, both of which are defined and required
(to varying degrees) by various POSIX standards.
• They are:
#include <unistd.h>
int ftruncate (int fd, off_t len);
• and:
#include <unistd.h>
int truncate (const char *path, off_t len);

Truncating Files
• The most common use of these system calls is to
truncate a file to a size smaller than its current
length
• The ftruncate() system call operates on the file
descriptor given by fd, which must be open for
writing
• The truncate() system call operates on the
filename given by path, which must be writable.
• Both return 0 on success. On error, they return −1
and set errno as appropriate

Multiplexed I/O
• Multiplexed I/O allows an application to
concurrently block on multiple file descriptors and
receive notification when any one of them
becomes ready to read or write without blocking.
• With I/O multiplexing, we call select or poll and
block in one of these two system calls, instead of
blocking in the actual I/O system call.

Multiplexed I/O
• Select is a system call and application programming
interface (API) in Unix-like and POSIX-compliant
operating systems for examining the status of file
descriptors of open input/output channels
• Syntax:
int select (int n,
fd_set *readfds,
fd_set *writefds,
fd_set *exceptfds,
struct timeval *timeout);

Multiplexed I/O
Argument Description
n This is an integer one more than the maximum of any file
descriptor in any of the sets
*readfds file descriptors to be checked for being ready to read
*writefds file descriptors to be checked for being ready to Write
*exceptfds file descriptors to be checked for error conditions
*timeout specifies a maximum interval to wait for the selection to
complete

Multiplexed I/O
• Poll system call:
– Unlike select(), with its inefficient three bitmask-based
sets of file descriptors, poll() employs a single array of
nfds pollfd structures, pointed to by fds
• Syntax:
#include <poll.h>
int poll (struct pollfd *fds, nfds_t nfds, int timeout);
It contains the set events
which have to checked for
the availbility
Number of fds
Time limit

Multiplexed I/O
• Structure of pollfd *fds
#include <poll.h>
struct pollfd {
int fd; /* file descriptor */
short events; /* requested events to watch */
short revents; /* returned events witnessed */
};

Kernel Internals
• The kernel subsystem consists of :
– the virtual filesystem (VFS),
– the page cache, and
– page writeback
• The virtual filesystem, occasionally also called a virtual
file switch, is a mechanism of abstraction that allows
the Linux kernel to call filesystem functions and
manipulate filesystem data without knowing—or even
caring about—the specific type of filesystem being
used.
• Linux file system is generally a built-in layer of a Linux
operating system used to handle the data management
of the storage. It helps to arrange the file on the disk
storage. It manages the file name, file size, creation
date, and much more information about a file.

Kernel Internals
• The Page Cache:
– The page cache is an in-memory store of recently
accessed data from an on-disk filesystem.
– Disk access is painfully slow, particularly relative to
today’s processor speeds.
– Storing requested data in memory allows the
kernel to fulfill subsequent requests for the same
data from memory, avoiding repeated disk access.

Kernel Internals
• Page Writeback
– Eventually the dirty buffers need to be committed to
disk, synchronizing the on-disk files with the data in
memory. This is known as writeback. It occurs in two
situations:
• When free memory shrinks below a configurable threshold,
dirty buffers are written back to disk so that the now-clean
buffers may be removed, freeing memory.
• When a dirty buffer ages beyond a configurable threshold,
the buffer is written back to disk. This prevents data from
remaining dirty indefinitely.

Linux System Programming - File I/O

More Related Content

What's hot (20)

Similar to Linux System Programming - File I/O (20)

Recently uploaded (20)

Linux System Programming - File I/O