Basic VI Commands Summary

How to create Shared Disk “between” two Solaris Machine on VMWare

Step 1 Create first Machine

Create the windows folders to house the first virtual machines and the shared storage.

D:\>mkdir SUNOS-1

F:\>mkdir SHARED-STORAGE
Install Solaris virtual machine in SUNOS-1 Directory.

Step 2 Create virtual disks for storage usage, which is shared by Both Machine.

 Down the virtual Machine

 Go to VMware Server Console. Click on Edit virtual machine settings.

 Virtual Machine Settings: Click on Add.

 Add Hardware Wizard: Click on Next.

 Hardware types: Select Hard Disk.

 Select a Disk:

 Disk: Select create a new virtual disk.

 Select a Disk Type:

 Virtual Disk Type: Select SCSI (Recommended).

 Specify Disk Capacity:

 Disk capacity: Enter “10GB.”

 Select Allocate all disk space now.

 Specify Disk File:

 Disk file: Enter “F:\>SHARED-STORAGE\DISK1.vmdk.”

 Click on Advanced

 Add Hardware Wizard:

 Virtual device node: Select IDE0.1.

 Mode: Select Independent, Persistent for all shared disks.

 Click on Finish.

Step 3 Modify virtual machine configuration file.

Additional parameters are required to enable disk sharing between the two virtual RAC nodes. Open

the configuration file, D:\>SUNOS-1\ Solaris 10.vmx.vmx and add the bold parameters listed below.

.
.

priority.grabbed = "normal"

priority.ungrabbed = "normal"

disk.locking = "FALSE"

diskLib.dataCacheMaxSize = "0"

ide0:1.sharedBus = "virtual"

ide0:0.redo = ""

ethernet0.addressType = "generated

ethernet0.connectionType = "hostonly"

ide0:1.present = "TRUE"

ide0:1.fileName = "E:\SHARED-DISK.vmdk"

ide0:1.redo = ""

checkpoint.vmState = ""

ide0:1.mode = "independent-persistent"

ide0:1.deviceType= "disk"
floppy0.present = "FALSE"

Step 4 Create and Configure the Second Virtual Machine

Create the windows folders to house the second virtual machines.

E:\>mkdir SUNOS-2

 Shutdown the First Virtual Machine

 Copy all the files from D:\SUNOS-1 to E:\SUNOS-2

 Open your VMware Server Console, press CTRL-O to open the second virtual machine,

 E:\SUNOS-2\Solaris 10.vmx.

 Rename the second virtual machine name from SUN1 to SUN2.

 Click on Start this virtual machine to start SUN2, leaving SUN1 powered off.

 SUN2 – Virtual Machine: Select create a new identifier.

 Log in as the root user and modify the network configuration.

Follow below step for modifying Host Name and IP

 $ ifconfig <Ethernat> <new IP>

 $ ifconfig <Ethernet> up

 $ go to /etc/hosts file and change IP and host

 $ go to /etc/nodenames file and change host anme

 $ go to /etc/hostname.<Ethernet> and change host name

 Restart the Second Virtual Machine.

 Start the First virtual Machine

 Verify all changes and Enjoy. Your Shared Storage is ready for fun.

If you want to add disk with Sun Solaris server. You just shut down the Machine and attached the

disk. Start the machine and execute the following command (#devfsadm). After executing this

command sun refresh the device list.

 I am describing here example for adding disk on VM (SUN SOLARIS)

Step: 1 shut down the VM

Step: 2 go to the Vmware setting

Step: 3 add a new hard disk device

Step: 4 Start Virtual machine and execute one of the following Method:

Method: 1 Open terminal and type following command:

# devfsadm

Method: 2 open terminal and type following command:

# touch /reconfigure

# reboot

After adding the physical disk. You must format the disk, partition the disk, create file system and

mount the file system.

We will discuss How to Format the Disk? in next session.

How to Format Disk?

You have learned about adding disk in my previous update. What do you think, only after adding the

disk, you are able to store data on disk?. NO…….NO….Never……you can not store data on disk.

if you want to store data on disk. You must complete the following task: Format the Disk, create disk

partition, create slice, create file system and mount the file system.

I will discuss all topics separately on my blog.

Here we will discuss only about How to Format the Disk?

Step 1 Invoke the format utility. This will display a numbered of list of disks.

Step 2 Type the number of the disk (Newelly Added Disk) on which to create a Solaris fdisk partition।

Step 3 Select the fdisk menu।format> fdisk

Step 4 The fdisk menu that is displayed depends upon whether the disk has existing fdisk partitions...
Type Yes for making 100% Solaris disk partation।

Step 5 Label the disk

format> label

Ready to label disk, continue? yes

format> Quit

Step 6 After creating fdisk partition, Next we will create slices on the disk(I will discuss How to create

Disk slice and label in next session...please wait coming soon........)

How to Create Disk Partation / Slices and Label a Disk?

Posted On Thursday, November 20, 2008 at at 9:12 AM by Anup Kumar Srivastav  

How to Create Disk Partation / Slices and Label a Disk?

After adding and formating disk we should create Disk Partation/Slice and Label the Disk.

Step 1 Invoke the format utility. A numbered list of disks is displayed.

Step 2 Type the number of the disk that you want to repartition.where disk-number is the number of

the disk that you want to repartition.

Step 3 Select the partition menu.

format> partition

Step 4 Display the current partition (slice) table.

partition> print

Step 5 Start the modification process.

partition> modify

Step 6 Set the disk to all free hog.

Choose base (enter number) [0]? 1

Step 7 Create a new partition table by answering yes when prompted to continue.

Do you wish to continue creating a new partition table based on above table[yes]? yes

Step 8. Make the displayed partition table the current partition table by answering yes when

prompted.

Okay to make this the current partition table[yes]? yes

If you don't want the current partition table and you want to change it, answer no.

Step 9. Name the partition table.

Enter table name (remember quotes): "partition-name"where partition-name is the name for the new

partition table.

Step 10. Label the disk with the new partition table after you have finished allocating slices on the new

disk.

Ready to label disk, continue? Yes

Step 11. Quit the partition menu.

Step 12 Verify the new disk label.

Step 13. Exit the format utility.

Step 14 After Labelind you can create file systems on the disk
How to Create File Systems?
How to Create File Systems?

After creating Disk Partition, we should create File System and Mount the file file system.

Step 1. Create a file system for each slice.

# newfs /dev/rdsk/cwtxdysz

where /dev/rdsk/cwtxdysz is the raw device for the file system to be created.

Step 2. Create a Directory

# mkdir

Step 3. Verify the new file system by mounting.

# mount /dev/dsk/cwtxdysz /

# ls

How to Add an Entry to the /etc/vfstab File?

How to Add an Entry to the /etc/vfstab File

Step 1 log in as a Root User

Step 2 Edit the /etc/vfstab file and add an entry.

Step 3 save the changes.

Solaris 10 OS Installation on VMWARE
Solaris 10 OS Installation on VMWARE

1. Open VMWARE Console and Click on New Virtual Machine and click next.

2. Select Virtual Machine Configurations: Typical

3. Select Gust Operation System: Sun Solaris and version solaris10

4. Type virtual Machine Name and Location.

5. Select Network Connection. Total 4 options.

 User Bridge connection

 Use network address translation (NAT)

 Use host only network (Select Host Only Network)

 Do not use a network

6. Specify Disk Capacities. (8 GB) and Click on Finish Button.

Now your virtual machine is ready to install Solaris 10 OS

7. Double Click on CD-ROM Devices

8. Select Use ISO image when you planning to install OS through ISO image otherwise Use physical

drive.

9. Click on Start the Virtual Machine.

10. Select Solaris Interactive (Default) installation

11. Configure keyboard Layout screen appear (press F2)

12 Select a language English

13 welcome screen appear (Click Next)

14 Select network connectivity and (Click Next).

15. DHCP Screen appears select No and (Click Next).

16 Type Host Nome and (Click Next)

17 Type IP Address and (Click Next)

18 Type netmask (Select default) and (Click Next)

19 Select No for Enable Ipv6 for pcn0 and (Click Next)

20 Select None for Default Route and (Click Next)

21 Select No for Enable Kerberos security and (Click Next)

22 Select None for Name Services and (Click Next)

23 NFSv4 Domain Name select Default and (Click Next)

24 Select Geographic Time Zones and (Click Next)

25 Continent and Country (INDIA) and (Click Next)

26 Accept the default date and time and (Click Next)

27 Enter ROOT password and (Click Next)

28 Select Yes for Enabling Remote Services and (Click Next)

29 Confirm Information and (Click Next)

30 Select default Install option and (Click Next)

After that the system is being analyzed. Please wait screen appear and select the Type of Installation

Now Installer Install to OS…

_________________________________________________________________________________

Fileystems
What is a file system?

Display all filesystems

We will start by issuing a df –k command to get a listing of all UNIX mounts points.  This example is
from the Solaris dislect of UNIX.

root> df -k

Filesystem            kbytes    used   avail capacity  Mounted on

/dev/dsk/c0t0d0s0    4032504  104381 3887798     3%    /
/dev/dsk/c0t0d0s4    4032504  992890 2999289    25%    /usr
/proc                      0       0       0     0%    /proc
fd                         0       0       0     0%    /dev/fd
mnttab                     0       0       0     0%    /etc/mnttab
/dev/dsk/c0t0d0s3    4032504  657034 3335145    17%    /var
swap                 4095176       8 4095168     1%    /var/run
swap                 4095192      24 4095168     1%    /tmp
/dev/dsk/c0t0d0s5    1984564  195871 1729157    11%    /opt
/dev/dsk/c0t0d0s7   14843673 1619568 13075669   12%    /helpdesk
/dev/oradg/u02vol   12582912 8717924 3744252    70%    /u02
/dev/oradg/u01vol    8796160 5562586 3132548    64%    /u01
/dev/oradg/u04vol   10035200 1247888 8519534    13%    /u04
/dev/oradg/u03vol   12582912 2524060 9744542    21%    /u03
/dev/dsk/c0t0d0s6    1984564  931591  993437    49%    /export/home
/vol/c0t/orcl901_3    270364  270364       0   100%   
/cdrom/orcl901_3

Here we see the following display columns:

1 – Filesystem name           

2 – The kbytes in the filesystem  

3 – Kbytes used in the filesystem  

4 – Kbytes available in the filesystem  

5 – File system capacity

6 – The mount point associated with the filesystem

Our goal is to filter this output to see the available space for the Oracle file systems. We also see rows

in this server that are not associated with Oracle files. 

Display Oracle filesystems

Our next step is to eliminate all file systems except the Oracle file systems.  In this system, we are

using the Oracle Optimal Flexible Architecture (OFA), and all Oracle filesystems begin with /u0. 

Hence, we can use the UNIX grep utility to eliminate all lines except for those containing the string

/u0:

root> df -k|grep /u0

/dev/vx/dsk/oradg/u02vol 12582912 8717924 3744252    70%    /u02

/dev/vx/dsk/oradg/u01vol 8796160 5563610 3131556     64%    /u01

/dev/vx/dsk/oradg/u04vol 10035200 1247888 8519534    13%    /u04

/dev/vx/dsk/oradg/u03vol 12582912 2524060 9744542    21%    /u03

Extract the available space for each filesystem

Now that we have the Oracle file systems, we can use the UNIX awk utility to extract the fourth

column, which is the available space in the filesystem.

root> df -k|grep /u0|awk '{ print $4 }'

3744252

3132546

8519534

9744542

Create the script to check space in all file systems

Now that we see the command, we can place this command inside a loop to evaluate the free space

for each filesystem.  Note that our command is placed inside the Korn shell for loop, and the command

is enclosed in back-ticks (the key immediately above the tab key).

check_filesystem_size.ksh

#!/bin/ksh 

for i in `df -k|grep /u0|awk '{ print $4 }'`

do

   # Convert the file size to a numeric value

   filesize=`expr i` 

   # If any filesystem has less than 100k, issue an alert

   if [ $filesize  -lt 100 ]

   then

      mailx -s "Oracle filesystem $i has less than 100k free."\

         don@burleson.cc\

         oracle_dba@my.company.com

   fi

done

This simple script will check every file system on the server and e-mail us as soon as any file system

has less than 100k of free space.

Scheduling the file alert

I generally place this type of script in a crontab file, and execute it every three minutes as shown in

the UNIX crontab entry below.

#****************************************************************

# This is the every 5 min. trace file alert report for the DBAs

#****************************************************************

1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,

47,49,51,53,5,57,59 * * * * /home/oracle/check_filesystem_size.ksh >

dev/null >&1

How do I find out what the largest files are in a file system?? In other words,
if I get the message that /tmp is 90% full and /tmp is not a raw slice but a
file system, how do I find out what the 20 largest files are and their
complete path??
Other people will probably be able to write some nice scripts for you, but
I've found the best solution on Linux to be fsv. I don't know much about HP-
UX, but I found what looks to be an HP-UX version of fsv.

Here's a one liner getting the 20 largest files in /tmp:

# all on one line

find /tmp -type f -print | xargs ls -l | sort -r -n -k 5,5 | head -20

You could also do the ls within the find command like this:

find /tmp -type f -exec ls -la {} \; | sort -r -n -k 5,5 | head -20

I tried your xargs command and that got me most of what I wanted...The
problem is that I got the 20 largest files under /tmp, but I didn't get the 20
largest files in the /tmp "filesystem"...

When I ran it on the 2 largest files I got this:

-rw-r--r-- 1 oracle dba

714559 Aug 20 2002 /tmp/admin/export.dmp

-rw-r----- 1 oracle dba 558828 May 16 2002 /tmp/files/dh_out.txt

If I go and do a "df -k ." from /tmp/admin and /tmp/files, I get two different
filesystems (see below):

/tmp/admin (/dev/vg00/lvol1) : 8733 total allocated Kb

1893621 free allocated Kb
6840322 used allocated Kb
78 % allocation used

/tmp/files (/dev/vg01/lvol1) : 4059 total allocated Kb

2145 free allocated Kb
3787 used allocated Kb
47 % allocation used

Do you have any ideas or suggestions??

It should work. You must have links in your /tmp file system. I don't know
about HP-UX, but the Solaris find has a -follow option which means "to follow
the link". This might help:

find /tmp -type f -follow -print | xargs ls -l | sort -r -n -k 5,5 | head -20

Differences between shells in UNIX

by admin    in AIX

sh csh ksh bash tcsh zsh rc es

Job control N Y Y Y Y Y N N
Aliases N Y Y Y Y Y N N
Shell functions Y(1) N Y Y N Y Y Y
"Sensible" Input/Output redirection Y N Y Y N Y Y Y
Directory stack N Y Y Y Y Y F F
Command history N Y Y Y Y Y L L
Command line editing N N Y Y Y Y L L
Vi Command line editing N N Y Y Y(3) Y L L
Emacs Command line editing N N Y Y Y Y L L
Rebindable Command line editing N N N Y Y Y L L
User name look up N Y Y Y Y Y L L
Login/Logout watching N N N N Y Y F F
Filename completion N Y(1) Y Y Y Y L L
Username completion N Y(2) Y Y Y Y L L
Hostname completion N Y(2) Y Y Y Y L L
History completion N N N Y Y Y L L
Fully programmable Completion N N N N Y Y N N
Mh Mailbox completion N N N N(4) N(6) N(6) N N
Co Processes N N Y N N Y N N
Builtin artithmetic evaluation N Y Y Y Y Y N N
Can follow symbolic links invisibly N N Y Y Y Y N N
Periodic command execution N N N N Y Y N N
Custom Prompt (easily) N N Y Y Y Y Y Y
Sun Keyboard Hack N N N N N Y N N
Spelling Correction N N N N Y Y N N
Process Substitution N N N Y(2) N Y Y Y
Underlying Syntax sh csh sh sh csh sh rc rc
Freely Available N N N(5) Y Y Y Y Y
Checks Mailbox N Y Y Y Y Y F F
Tty Sanity Checking N N N N Y Y N N
Can cope with large argument lists Y N Y Y Y Y Y Y
Has non-interactive startup file N Y Y(7) Y(7) Y Y N N
Has non-login startup file N Y Y(7) Y Y Y N N
Can avoid user startup files N Y N Y N Y Y Y
Can specify startup file N N Y Y N N N N
Low level command redefinition N N N N N N N Y
Has anonymous functions N N N N N N Y Y
List Variables N Y Y N Y Y Y Y
Full signal trap handling Y N Y Y N Y Y Y
File no clobber ability N Y Y Y Y Y N F
Local variables N N Y Y N Y Y Y
Lexically scoped variables N N N N N N N Y
Exceptions N N N N N N N Y

Key to the table above.

Y Feature can be done using this shell.

N Feature is not present in the shell.

F Feature can only be done by using the shells function

mechanism.

L The readline library must be linked into the shell to enable

this Feature.
Notes to the table above

1. This feature was not in the orginal version, but has since become
almost standard.
2. This feature is fairly new and so is often not found on many
versions of the shell, it is gradually making its way into
standard distribution.
3. The Vi emulation of this shell is thought by many to be
incomplete.
4. This feature is not standard but unoffical patches exist to
perform this.
5. A version called 'pdksh' is freely available, but does not have
the full functionality of the AT&T version.
6. This can be done via the shells programmable completion mechanism.
7. Only by specifing a file via the ENV environment variable.
Sudo (superuser do) allows a system administrator to work using his own account and switch to
root or other user identity available on the system only for commands that need it. It also provide
a better logging and "ticketing" system. Sudo is useful on all version and flavors of Unix, with
possible exception of Solaris 10 and later because native OS mechanisms (RBAC) are superior.  
For a brief history of Sudo see history section

Main advantages

 The simplification of delegation of root.  In large enterprises one of the services Unix group
provides is to set a temp root password on one or several boxes so that application administrator
can perform some operations which require root access. Communication of temp root password
creates a vulnerability due to which an intelligent person can access root account on some other
systems that currently are converted for temp root.  For example this is the case if temp password
is created using some formula that contains as a part server name (for example prefix) and a
random string (as a suffix). In case of sudo you can use automatic remote enrollment of user into
particular group (wheel  or sysadmin, depending on flavor of Unix used) and do not even need to
communicate anything to the user other that he got the ability to become root from his account.

Ssh or Tivoli TCM permit mass changing group membership for the wheel group. In this case
you need to execute just one command to enroll the person who need temporary root access to aq
group of servers. What is important you can automatically provide deenrollement after the
requested period is over: just schedule at command that remove the person from the group at the
end of agreed period.

This way there is no possibility of forgetting to remove the privileges of the user  for whom the
specified term of root access is expired: Deletion is done automatically when the period expires
by the at command scheduled beforehand.  Otherwise this operation requires human action of
resetting password back, the operation that more often then not is performed later then in due
time.
 
 Audit trail for all attempts to switch to root both successful and unsuccessful.  That
tremendously help in troubleshooting, especially  as is typical for large enterprise environment
when there are multiple cooks (aka sysadmins ;-)  in the kitchen and the left hand does not know
what right hand is doing.
 
 Simplification of switching to root for administrators.  Sudo can be configured in such a way
that members of wheel group do not need additional authentication to became the root. That saves
time and troubles  of remembering multiple root passwords if you need to administer multiple
systems (or a creation of a security vulnerability if such password are constructed using a pre-
defined formula; intelligent users can soon get this "secret" and use it to their own advantage to
cut red tape).
 Ubuntu goes one step further here in the default sudo configuration. By default, the root account
password is locked in Ubuntu. This means that you cannot login as root directly. However,
since the root account physically exists it is still possible to sudo to root and to run programs with
root-level privileges.
 
 Prevention of costly mistakes. The main value of sudo for sysadmins is that it helps to prevent
some really horrible things that often happen when you work as root and do something in a hurry
as root. With sudo you can try to execute command as a regular user but it permissions are
inadequate and command fails repeat it as root using shortcut sudo !!. And that's probably the best
part of its usefulness. It  permits to execute single command as root without authentication (if
configured this way) by prefixing with sudo.  In most cases that all administrator needs so long
(and dangerous) sessions of using root account to trivial operations can be eliminated. If
passwords for regular accounts are one time (e.g. reasonably secure as is the case if SecurID or
similar one-time password system is used)  this is a very attractive solution. Please remember that
a lot of systems were destroyed by administrators using some badly though out command as root
without proper testing.  See Admin Horror Stories
 
 Possibility to grant "read-only" access to certain files. From this point of view this capability
of granting command access is most useful for providing a "read-only" and "re-write"  assess to
some root owned files by granting usage of explicit cat (or cp in case of re-write access)
commands for selected files, or other simple commands with explicit arguments (not vi or other
commands with the possibility escaping to command shell). But even this is dangerous and this
privilege can be abused. If cp command granted, the wrapper can be provided which to modify
the file the user needs first to save it in his own directory and then executes move to target
destination command.   With ACLs this capability is now less useful but still worth mentioning.
 
 Possibility to grant only selected commands executed with extended privileges to certain
users. Suse is definitely not RBAC, but some functionality similar to RBAC can be imitated by
granting access to several command (for example the ability to start and stop particular
application server).  The command should be unable to escape to shell and be simple enough to
exclude this possibility (it is a bad idea to grant the ability to execute find caommand as root for
obvious reasons) Compete specification of arguments usually is required to prevent attempts to
use the command as a "universal root opener".   While definitely useful for restarting daemons
and some similar actions this is not a RBAC and you should keep such assignments simple.
Excessive zeal here really hurts.  The alternative is using one command accounts like shutdown
and restart, where the logging with password executes the required command. This might be
more convenient for operators.
 
 Logging of each command, providing a clear audit trail of "who did what."  When
used in tandem with remote syslogd, sudo can log all commands to a central loghost (as
well as on the local host).  Please note the for AIX you need to enable syslog first.
 
 "Time ticketing" system. When a user invokes sudo and enters their password, they are granted
a ticket for 5 minutes (this timeout is configurable at compile-time). Each subsequent sudo
command updates the ticket for another 5 minutes. This gives some level of protection in
situation where there is a danger of  accidentally leaving a root shell where others can physically
get to your keyboard.  While certainly annoying, this is a useful facility for critical boxes.

Sudo is distributed under a BSD-style license and currently maintained by Todd Miller. The
current homepage is http://www.courtesan.com/sudo/ 
Conceptually sudo is a really simple package that unfortunately grow too complex with time. It
consist of  just three utilities (sudo, visudo and sudoreplay) and a single configuration file:

1. sudo(8) command: setuid root wrapper that users invoke to execute commands as root (or other
privileged account).
 
2. /etc/sudoers. This is a configuration file which can also be located in /etc/sudo or /usr/local/etc
directory. It is sudo's roles definition file specifying who may run what commands as which user.
It is fully documented in sudoers(5). See also  a good into in Gentoo Sudo(ers) Guide — Gentoo
Linux Documentation
 
3. visudo(8) command allows administrators to edit the sudoers file without risking locking
themselves out of the system.
 
4. Sudoreplay plays back or lists the session logs created by sudo. When replaying, sudoreplay can
play the session back in real-time, or the playback speed may be adjusted (faster or slower) based
on the command line options.

List sessions run by user millert:

sudoreplay -l user millert

List sessions run by user bob with a command containing the string vi:

sudoreplay -l user bob command vi

List sessions run by user jeff that match a regular expression:

sudoreplay -l user jeff command '/bin/[a-z]*sh'

List sessions run by jeff or bob on the console:

sudoreplay -l ( user jeff or user bob ) tty console

SUDO IN SOLARIS 10

Solaris 10 Role Based Access Control

(RBAC)
The problem with the traditional model is not just that root (superuser) is so powerful, but that a
regular user accounts are not powerful enough to fix their own problems.

There were some limited attempts to address this problem in Unix in the past (wheel group and
immutable file attributes in BSD, sudo, extended attributes (ACL), etc), but Role Based Access
Control (RBAC) as implemented in Solaris 10 is probably the most constructive way to address
this complex problem in its entirety.  
RBAC is not a completely new thing. Previous versions of RBAC with more limited capabilities
existed for more then ten years in previous versions of Solaris. It was introduced in Trusted
Solaris and was later incorporated into the Solaris 8. In was improved and several additional
predefined roles were introduced in Solaris 9.  Still they generally fall short of expectations and
only Solaris 10 implementation has the qualities necessary for enterprize adoption of this feature.

Among predefined roles that is several that are immediately useful and usable:

1. All Provides a role access to commands without security attributes: all commands that do not
need root permission to run in a regular Solaris system (Solaris without RBAC implementation) :
 
2. Primary Administrator Administrator role that is equivalent to the root user.
 
3. System Administrator. Secondary administrators who can administer users (add, remove user
accounts, etc). Has privileges solaris.admin.usermgr.read solaris.admin.usermgr.write that
provides read/write access to users’ configuration files. Cannot change passwords.
 
4. Operator. Has few security related capabilities but still capable of mounting volumes. Also has
solaris.admin.usermgr.read privilege that provides read access to users’ configuration files.
 
5. Basic Solaris User.  Enables users to perform tasks that are not related to security.
 
6. Printer Management. Dedicated to printer administration.

But the original implementation has had severe limitation in defining new roles which blocked
wide adoption of this feature: in practice most system commands that were needed for roles
should be run as root. Still even the old implementation that exited till Solaris 9 has sudo-style
capability of one time assumption of the command with specific (additional) privileges is
accomplished by pfexec command.  If the user is assigned "Primary Administrator" profile
then pfexec command became almost exact replica of the typical sudo usage. 

Also if role has no password then switch of context does not require additional authentication
(only authorized users can assume roles). That can be convenient for some application roles.

There were several problems with early RBAC implementations:

 Limited flexibility in constructing new roles

 Hidden dangers of running selected commands with root privileges (that danger that is typified by
sudo).
 Fuzzy interaction of RBAC facility with the extended attributes facility (ACL).
 The four flat flat files that Solaris 8 implementation introduced suggested some questionable
quality of engineering.

All-in-all Solaris RBAC until version 10 has limited appeal to most organization and unless there
was a stron push from the top was considered by most administrators too complex to be
implemented properly. It also has some deficiencies even in comparison with sudo.  The only
"cheap and sure"  application in the old RBAC implementation was conversion of root account to
role and conversion of  operators to operator roles. Conversion of application-related accounts
like oracle into roles was also  possible, but more problematic.

That changed with Solaris 10 when RBAC model was extended with Solaris privileges model. It
extended the ability to create new custom roles by assigning very granular privileges to the
roles.   Previously such tricks needed heavy usage of ACLs, and as any ACL-based solution
were  both expensive and heavy maintenance solutions

There are three distinct properties of roles:

1. A role is not accessible for normal logins (root is a classic example of an account that should
not be accessible by normal login; most application accounts fall into the same category)
2. Users can gain access to it only explicitly changing his identity via su command, the activity
that is logged and can be checked for compliance.
3. Role account uses a special shell (pfksh or pfsh).  Please note that bash is not on the list :-)

Each user can assume as many roles as is required for him to perform his responsibilities (one at
a time) and switch to a particular role for performing a subset of operations that are provided for
this role. Theoretically an administrator can map user responsibilities into a set of roles and then
grant users the ability to "assume" only those roles that match their job responsibilities. And no
user should beg for root access any longer :-)  But the devil is in details: even with Solaris 10
power this easier said then done. Role engineering is a pretty tough subject in itself even if
technical capabilities are here and it requires time and money to implement properly. 

Still it looks like Solaris 10 was the first Unix that managed to breaks old Unix dichotomy of
"root and everybody else". In this sense Solaris 10 is the first XXI century Unix.   The privilege
model that was incorporated in RBAC made it more flexible and useful, surpassing sudo in most
respects. One time execution of a command, for example vi, with additional privileges still
remains the problem as the command can have a backdoor to shell.  

Like its predecessor sudo, Solaris RBAC provides the ability selectively package superuser
privileges  for assignment to user accounts by assigning them packages of the appropriate
privileges. For example, the need for root account can be diminished by dividing those
capabilities into several packages and assigning them separately to individuals sharing
administrative responsibilities (still root remains a very powerful account as it owns most
important files).

It might be useful to distinguish between following notions:

 Authorization - A right that is used to grant access to a restricted function

 Profile - A mechanism used for grouping authorizations and commands for subsequent
assignment to role or to a user. You can assign one or several profile to role.
 Role - A special type of user account that you cannot login directly, but can only su to it.  It
intended for application accounts and sometimes is useful as a container for performing a set of
administrative tasks
  Role shell Special shell alias (for example pfksh, instead of ksh) that gives the shell capability to
consult RBAC database before execution of the command.  Please note that bash can't be used as
role shell.

RBAC relies on a database that consist of four flat files (naming suggests that Microsoft agents
penetrated Sun on large scale ;-), as the proper way to group related configuration files in Unix is
to use common prefix, like rbas_user, rbac_prof, rbac_exec, rbac_auth, but Unix is flexible and you can
create such links and forget about this problem):

 /etc/user_attr (main RBAC file)

 /etc/security/prof_attr (right profile attributes/authorizations)
 /etc/security/exec_attr (profile execution attributes)
 /etc/security/auth_attr (authorization attributes).

As usual syntax is pretty wild and is a testimony that in Sun left hand does not know what right
is doing. Essentially this is another mini-language in a family of approximately a hundred mini-
languages that Sun boldly introduced for configuration files while naively expecting that
administrators say with Solaris no matter what perverse syntax they are using in "yet another
configuration file" (TM) :-).   HEre are some details on those configuration files:

1. /etc/user_attr (main RBAC file, essentially the extension of /etc/passwd)

/etc/user_attr 
lists the accounts that are roles, associates regular users with roles. Consists
of type of the account (type=) and authorizations list (auth= ) and profiles (profiles=,
which is an indirect way to assign authorizations). If type is notmal then the account is a
regular traditional Unix account for example:

root::::type=normal;auth=solaris.*,solaris.grant

If type is "role" then this is a new type of account -- role account.  For example:

datesetter::::type=role;profile=Date Management

By default all Solaris users are granted  Basic Solaris User profile. The default profile stored in
/etc/security/policy.conf is applicable to all accounts that do not have an explicit assignment.

Effective profile for normal users can also be changed, for example for the user Joe Doers the
profile can be changed to Log Management:

doerj::::type=normal;profile=Date Management

2. /etc/security/prof_attr (right profile attributes/authorizations) Associates names of right

profiles (or simply profiles, although the tem is confusing)  with the set of authorizations. Only
authorizations listed in /etc/security/auth_attr are allowed.

This is a little bit problematic implementation as right profile is essentially a

parameterless macro that is substituted into a predefined set of authorizations. But
surprisingly the only form of this macro definition is plain vanilla list. There is no
wildcard capabilities of regular expression capabilities in specifying them. Also there is
no way to deny certain lower level authorization while granting a higher level
authorizations. For example, I cannot specify expressions like (solaris.admin.usermgr.* -
solaris.admin.usermgr.write). There is also no possibility to grant global access to a
specific operation reading like solaris.*.*.read  In general I see no any attempt to
incorporate the access control logistics typical for TCP wrappers, firewalls and similar
programs. That makes creation of a profile less flexible then it should be, but hopefully
this is not that frequent operation anyway, and you can write Perl scripts that generate
any combination of authorizations you want quote easily, so the damage is minor.

Like I mentioned in my lecture before there are several predefined right profiles (all of
the them can be modified by sysadmin):

o All right profile that provides a role access to commands without security attributes. In
a non-RBAC system, these commands would be all commands that do not need root
permission to run.
o Primary Administrator right profile that is designed specifically for the Primary
Administrator role. In a non-RBAC system, this role would be equivalent to the root user.
o System Administrator. right profile that is designed specifically for a junior level
System Administrator role. The System Administrator rights profile uses discrete
supplementary profiles to create a powerful role.
o Operator right profile Designed specifically for the Operator role. The Operator rights
profile uses a few discrete supplementary profiles to create a basic role.
o Basic Solaris User right profile that enables users to perform tasks that are not related to
security.
o Printer Management. right profile dedicated to the single area of printer administration.

Each profile consists of  one or more authorizations, for example:

Basic Solaris User:::Automatically assigned rights:

auths=

solaris.profmgr.read,solaris.jobs.users, solaris.mail.mailq, solaris.admin.usermgr.read,

solaris.admin.logsvc.read, solaris.admin.fsmgr.read, solaris.admin.serialmgr.read,
solaris.admin.diskmgr.read, solaris.admin.procmgr.user, solaris.compsys.read,
solaris.admin.printer.read, solaris.admin.prodreg.read, solaris.admin.dcmgr.read,
solaris.snmp.read,solaris.project.read, solaris.admin.patchmgr.read, solaris.network.hosts.read,
solaris.admin.volmgr.read;

profiles=All; help=RtDefault.html

Notes:

it uses mnemonic name for a privilege set, not dotted representation. There are many
Sun-supplied profiles (30 in Solaris 9), for example:

o Primary Administrator (profile that permits performing all administrative tasks)

 o Basic Solaris User (Default profile assigned to the new accounts )
o Operator (Can perform simple administrative tasks).
 
3. /etc/security/exec_attr(profile execution attributes) - this is sudo style file that defines the
commands assigned to a profile and under which EUID and EGID. The fields in the
/etc/security/exec_attr database are separated by colons:

name:policy:type:res1:res2:id:attr

o name The name of the profile. Profile names are case sensitive.
o policy -- The security policy (priviledge) associated with this entry. In Solaris 9 the suser
(superuser policy model) is the only valid policy entry.
o type The type of entity whose attributes are specified. The only valid type is cmd
(command).
o res1,res2 Reserved for future use.  Reserved for future use.
o id A string identifying the entity. You can use the asterisk (*) wildcard. Commands
should have the full path or a path with a wildcard. To specify arguments, write a script
with the arguments, and point the id to the script.
o attr An optional list of key-value pairs that describes the security attributes to apply to
the entity when executed. You can specify zero or more keys. The list of valid key words
depends on the policy being enforced. There are four valid keys: euid, uid, egid, and gid.
o euid and uid – Contain a single user name or a numeric user ID. Commands designated
with euid run with the effective UID indicated, which is similar to setting the setuid bit
on an executable file. Commands designated with uid run with both the real and effective
UIDs set to the UID you specify.
o egid and gid – Contain a single group name or numeric group ID. Commands designated
with egid run with the effective GID indicated, which is similar to setting the setgid bit
on an executable file. Commands designated with gid run with both the real and effective
GIDs set to the GID you specify.

For example

Date Management:suser:cmd:::/usr/bin/date:euid=0

Adds to the profile "Date Management" the ability to execute command " /usr/bin/date
 

4. /etc/security/auth_attr (authorization attributes)- This is a system generated static file that

predefines a hierachical sets of privileges available on a particular system (92 in Solaris 9, 126 in
Solaris 10). Privileges (authorizations) are structured like DNS with dots separating each
constituent:
 
o Authorizations for the Solaris OE use solaris as a prefix.
 
o The suffix indicates what is being authorized, typically the functional area and operation.
For example grant or delete or modify.
 
o When there is no suffix (that is, the authname consists of a prefix, a functional area, and
ends with a period), the authname serves as a heading for use by applications in their
 GUI rather than as an authorization. The authname solaris.printmgr. is an example of a
heading.
 
o When authname ends with the word grant, the authname serves as a grant authorization
and lets the user delegate related authorizations (that is, authorizations with the same
prefix and functional area) to other users. The authname solaris.printmgr.grant is an
example of a grant authorization. It gives the user the right to delegate such
authorizations as solaris.printmgr.admin and solaris.printmgr.nobanner to other
users.
 

Only system programmers can add entries to this database. It also identifies the help file that
explains a particular privilege set.

For example:

solaris.admin.usermgr.:::User Accounts::help=AuthUsermgrHeader.html
solaris.admin.usermgr.write:::Manage
Users::help=AuthUsermgrWrite.html
solaris.admin.usermgr.read:::View Users and
Roles::help=AuthUsermgrRead.html
solaris.admin.usermgr.pswd:::Change
Password::help=AuthUserMgrPswd.html

5. In addition to those four file, /etc/security/policy.conf  lets you grant specific rights profiles and
authorizations to all users.  Essentially provides system default authorizations for all users.
Entries consist of key-value pairs, for example:

6. AUTHS_GRANTED=solaris.device.cdrw

PROFS_GRANTED=Basic Solaris User

The solaris.device.cdrw authorization provides access to the cdrw command.

# grep ’solaris.device.cdrw’
/etc/security/auth_attr solaris.device.cdrw:::CD-R/RW Recording
Authorizations::help=DevCDRW.html

The Basic Solaris User profile grants users access to all listed authorizations.

Paradoxically RBAC can be more useful for application accounts then to "human" accounts. 
That means that for a large organization an optimal plan for conversion to RBAC is first to
convert system and applications accounts to roles.

 Among system roles are root and operator. Both roles can (and probably should ) be converted to
role even in previous versions of Solaris.
 By application accounts we mean the account used for structuring permissions and launching
processes for a particular enterprise software like Oracle, Webshere, Apache, Sendmail, bind, etc)
because privileges requirements for those accounts are static.
The main command line tools include:

 roleadd – Adds a role account on the system.

 rolemod – Modifies a role’s login information.
 useradd – Adds a user account on the system.

Additional commands that you can use with RBAC operations.

 auths Displays authorizations for a user.

 pam_roles Identifies the role account management module for the Password Authentication
Module (PAM). Checks for authorization to assume a role.
 pfexec Executes commands with the attributes specified in the exec_attr database.
 roles Displays roles granted to a user.
 roleadd Adds a role account to the system.
 roledel Deletes a role’s account from the system.
 rolemod Modifies a role’s account information in the system.

Command that has role relat4ed options:

 useradd Adds a user account to the system. Use the -R option to assign a role to a user’s account.
 userdel Deletes a user’s login from the system.
 usermod Modifies a user’s account information in the system.

Related commands:

 makedbm Makes a dbm file.

 nscd Identifies the name service cache daemon, which is useful for caching the user_attr, prof_attr,
and exec_attr databases

Useful Commands
Command Description

<esc><esc> or <esc>\ autocompletion

file * Show file types, such as "ascii".

find <path> -name <name> Finds a file in the OS at the starting path on down
-print

grep

groups <username> Shows groups for a user

kmtune Displays kernel tunable parameters

listusers -g <group> Lists the users in a group

ln -s <file or directory> Creates a symbolic link to a file or a directory
<symbolic link>

pr -o10 -l64 -F -h <HEADER> prints using margins of 10, page length of 64, header on each page
<file> | lp

rcp <source> remote copy

remote_machine:<path>

remsh <host> opens a remote command shell on the host machine

rlogin <host> remote login to host machine

sar

set_parms Change host name (/etc/set_parms)

tail -f <file> Looks at end of file and keeps it open to watch changes

top realtime display of processes

uname -a information about the system

print $(uname -i)16op|dc prints hex system id

who -u, who -Rm who is using the system

Process to recover from a lost/forgotten root password

1. Power off the server

2. Power on the server
3. Interupt the boot process during the 10 second interval (display will indicate this on the screen)
4. bo pri
5. Answer yes to interact with the ipl
6. At the prompt "hpux -is" for single user mode
7. cd /sbin
8. passwd root  and  follow prompts to put in new password
9. shutdown -r 0  to reboot to multiuser mode
Solaris tar command to backup data on tape
device
by nixcraft on March 22, 2005 · 17 comments

Tar name come from Tape ARchiver. It is both a file format and the name of the program used
to handle such file. Tar archive files have names ending in ".tar". If an archive is compressed, the
compression program adds its own suffix as usual, resulting in filename endings like ".tar.Z",
".tar.gz", and ".tar.bz2". Tar doesn't require any particular filename suffix in order to recognize a
file as an archive. Tar was originally created for backups on magnetic tape, but it can be used to
create tar files anywhere on a filesystem. Archives that have been created with tar are commonly
referred to as tarballs.

Create a new set of backup

To create a Tar file, use tar command as follows:
# tar cvf /dev/rmt/X file1 file2 dir1 dir2 file2 …
Where

 c – Create a new files on tape/archive

 v – verbose i.e. show list of files while backing up
 f – tape device name or file

For example, backup /export/home/vivek/sprj directory to tape device /dev/rmt/0, enter

# tar cvf /dev/rmt/0 /export/home/vivek/sprj/
Remember c option should only use to create new set of backup.

Appending or backing up more files to same tape using tar

tar provides r option for appending files to tape. For example to backup /data2/tprj/alpha1 files to
same tape i.e. appending files to a first tape device:
# tar rvf /dev/rmt/0 /data2/tprj/alpha1/*
Where

 r – append files to the end of an archive/tape

List files on a tape using tar command
To display file listing of a first tape use tar as follows:
# tar tvf /dev/rmt/0
To listing the Contents of a Stored Directory (for example wwwroot directory):
# tar tvf /dev/rmt/0 wwwroot
Where

 t – list the contents of an archive/tape

Retrieve / restore tape backup taken with tar

1) Use tar command as follows to retrieve tape drive backup to current directory:
(a) Change directory where you would like to restore files:
# cd /path/to/restore
# pwd
(b) Now, do a restore from tape:
# tar xvf /dev/rmt/0

To specify target directory use –C option

Restore everything to /data2 directory:

# tar xvf /dev/rmt/0 –C /data2
To retrieve directory or file use tar as follows:
# tar xvf /dev/rmt/0 tprj
Note that Solaris tar command is little different from GNU tar, if you wish to use gnu tar with
Solaris use command gtar. Gnu tar accepts same command line options plus bunch of additional
options :)

TAPE ERROR

I've backed up several files to tape using tar, and wish to list those that have backed up.

% tar cvf /dev/rmt/2un /s_1/oradata/pgpub/config.ora

a /s_1/oradata/pgpub/config.ora 2 tape blocks

But when I go to list the files:

% tar tvf /dev/rmt/2un

tar: tape read error

What am I doing wrong? How can I verify that it was backed up?

Did you rewind the tape before giving the tar tvf command?
whoops, That was it! My mistake. Thanks a ton!
am not sure what the u means, but the n stands for NOrewind, so rewind your tape (mt
-f /dev/rmt/2 rewind) and test it again. have a look at the mt man page
Question: How do I find out all the available file attributes. i.e I would like to know more about
a file or directory than what the ls -l command displays.

Answer: Everything in Unix is treated as files. This includes devices, directories and sockets —
all of these are files. Stat command displays file or filesystem status as explained in this article.

File Stat – Display Information About File

For example, to find out more information about 101hacks.txt file, execute the stat command as
shown below.

$ stat 101hacks.txt
File: `/home/sathiyamoorthy/101hacks.txt'
Size: 854 Blocks: 8 IO Block: 4096 regular file
Device: 801h/2049d Inode: 1058122 Links: 1
Access: (0600/-rw-------) Uid: ( 1000/ sathiya) Gid: ( 1000/ sathiya)
Access: 2009-06-28 19:29:57.000000000 +0530
Modify: 2009-06-28 19:29:57.000000000 +0530
Change: 2009-06-28 19:29:57.000000000 +0530

Details of Linux Stat Command Output

 File: `/home/sathiyamoorthy/101hacks.txt’ – Absolute path name of the file.

 Size: 854 – File size in bytes.
 Blocks: 8 – Total number of blocks used by this file.
 IO Block: 4096 – IO block size for this file.
 regular file – Indicates the file type. This indicates that this is a regular file. Following are
available file types.
o regular file. ( ex: all normal files ).
o directory. ( ex: directories ).
o socket. ( ex: sockets ).
o symbolic link. ( ex: symbolic links. )
o block special file ( ex: hard disk ).
o character special file. ( ex: terminal device file ).
 Device: 801h/2049d  – Device number in hex and device number in decimal
 Inode: 1058122 – Inode number is a unique number for each file which is used for the internal
maintenance by the file system.
 Links: 1 – Number of links to the file
 Access: (0600/-rw——-): Access specifier displayed in both octal and character format. Let us
see explanation about both the format.
 Uid: ( 1000/ sathiya) – File owner’s user id and user name are displayed.
 Gid: ( 1000/ sathiya) – File owner’s group id and group name are displayed.
 Access: 2009-06-28 19:29:57.000000000 +0530 – Last access time of the file.
 Modify: 2009-06-28 19:29:57.000000000 +0530 – Last modification time of the file.
 Change: 2009-06-28 19:29:57.000000000 +0530 – Last change time of the inode data of that
file.
Dir Stat – Display Information About Directory

You can use the same command to display the information about a directory as shown below.

$ stat /home/ramesh
File: `/home/ramesh'
Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: 803h/2051d Inode: 5521409 Links: 7
Access: (0755/drwxr-xr-x) Uid: ( 401/ramesh) Gid: ( 401/ramesh)
Access: 2009-01-01 12:17:42.000000000 -0800
Modify: 2009-01-01 12:07:33.000000000 -0800
Change: 2009-01-09 12:07:33.000000000 -0800

Details of File Permission:

File Permission In Octal Format

This information about the file is displayed in the Access field when you execute stat command.
Following are the values for read, write and execute permission in Unix.

UNIX File Attributes & Access Permissions

Remember when we first looked at the listing from ls -la (Figure 12), and we skipped over most
of the symbols in the first column? Well, "They're baaaack!" But that's OK, because we're ready
for them now. Let's take a closer look at a single entry from that listing, and see if we can't
demystify that first column.

-rw-r----- 1 picard STAFF 1397 May 28 12:50 mj.ultra

OK, let's isolate the part in question: the first column. We'll diagram it so you understand what
each position means.

Figure 13. Diagram of UNIX File Attribute

After the first character, which identifies the entry type (Remember? d for a directory, and - for a
regular file.), you will find exactly nine (9) other characters. In order to make sense of these, you
have to break them up in your head, into three groups of three symbols each (3 times 3 = 9).

The first three symbols represent the access permissions/attributes which apply to the user who
owns the file (usually you).
The second group of three symbols represents the access permissions/attributes which apply to
the group associated with the file. In UNIX, every userid is a member of one or more groups, and
separate permissions/attributes can be set for the group, as opposed to the owner. In the example
we used in Figure 12 (Chapter 22), the group associated with this file is "STAFF."

The third and last group of three symbols represents the access permissions and attributes which
apply to all other userids (other than the owning user and members of the owning group).

Now, let's look at the individual sets of attributes. Each of the three sets (User, Group, and
Others) is identical, with respect to the meaning of the three characters of which it is composed.
That is to say, each of the three positions in each set means the same thing from set to set.

Figure 14. UNIX File-Access Attributes

Changing File attributes with the chmod command

We'll just mention briefly the use of the chmod (change permission mode) command to change
these attributes and permissions, as this is getting into advanced territory, beyond the scope of
this introductory manual. However, it is important that you are at least aware of these flags and
their meanings so you can investigate this area further when and if the need arises.

The basic form of the chmod command is:

chmod who add-or-remove what_permissions filename

Note

there should not be any spaces between the "who", "add-or-remove", and "what_permissions"
portions of the command, in a real chmod command. The spaces were included in the above
diagram to make it more readable. See the following examples for samples of proper syntax.)

We'll break that diagram down a little further, and then give some examples.

Command "Breakdown": chmod. 

chmod

This is the name of the command.

who

Any combination of u (for "user"), g (for "group"), or o (for "others"), or a (for "all"--that
is, user, group, and others).

add-or-remove

Use + to add the attribute (set the flag), or - to remove the attribute (clear the flag).

what_permissions

Any combination of r (for Read), w (for Write), or x (for Execute).

filename

A file or directory name (or wildcard pattern) to which you wish to apply the listed
permission changes.

Examples of Using chmod. 

chmod a+r *

Makes all files in the current directory readable by anyone.

chmod u-w special.documentfilename

"Write-protects" special.document so that you, the user/owner, can't change it (without first issuing
another chmod command to make the file writable).

chmod g+r group.stuff

 Value Meaning
 4 Read Permission
 2 Write Permission
 1 Execute Permission

File Permission In Character Format

This information about the file is displayed in the Access field when you execute stat command.

 File Type: First bit of the field mentions the type of the file.
 User Permission: 2nd, 3rd and 4th character specifies the read, write and execute permission of
the user.
 Group Permission: 5th, 6th and 7th character specifies the read, write and execute permission
of the group.
  Others Permission: 8th, 9th and 10th character specifies the read, write and execute permission
of the others.

Display Information About File System

You can also use stat command to display the file system information as shown below.

$ stat -f /
File: "/"
ID: 0 Namelen: 255 Type: ext2/ext3
Blocks: Total: 2579457 Free: 1991450 Available: 1860421 Size: 4096
Inodes: Total: 1310720 Free: 1215875

What are Inodes?

Origin of term
The exact reason for designating these as "i" nodes is unknown. When asked, Unix pioneer
Dennis Ritchie replied:[1]

In truth, I don't know either. It was just a term that we started to use. "Index" is my best guess, because
of the slightly unusual file system structure that stored the access information of files as a flat array on
the disk, with all the hierarchical directory information living aside from this. Thus the i-number is an
index in this array, the i-node is the selected element of the array. (The "i-" notation was used in the 1st
edition manual; its hyphen was gradually dropped.)

[edit] Details
An important part of a file system is the data structures that contain information about the files.
Each file is associated with an inode (identified by an inode number, often referred to as an i-
number or inode).

Inodes basically store information about files and folders, such as (user and group) ownership,
access mode (read, write, execute permissions) and file type. On many types of file systems the
number of inodes available is fixed at file system creation, limiting the maximum number of files
the file system can hold. A typical space allocation for inodes in a file system is 1% of total size.

The inode number indexes a table of inodes in a known location on the device; from the inode
number, the kernel can access the contents of the inode, including the location of the file
allowing access to the file.

A file's inode number can be found using the ls -i command. The ls -l command displays
some of the inode contents for each file.

Some Unix-style file systems such as ReiserFS omit an inode table, but must store equivalent
data in order to provide equivalent capabilities. The data may be called stat data, in reference to
the stat system call that provides the data to programs.
File names and directory implications:

 Inodes do not contain file names, only file metadata.

 Unix directories are lists of "link" structures, each of which contains one filename and one inode
number.
 The kernel must search a directory looking for a particular filename and then convert the
filename to the correct corresponding inode number.

The kernel's in-memory representation of this data is called struct inode in Linux. Systems derived
from BSD use the term vnode, with the v of vnode referring to the kernel's virtual file system
layer.

[edit] POSIX inode description

The POSIX standard mandates filesystem behavior that is strongly influenced by traditional
UNIX filesystems. Regular files must have the following attributes:

 The size of the file in bytes.

 Device ID (this identifies the device containing the file).
 The User ID of the file's owner.
 The Group ID of the file.
 The file mode which determines the file type and how the file's owner, its group, and others can
access the file.
 Additional system and user flags to further protect the file (limit its use and modification).
 Timestamps telling when the inode itself was last changed ( ctime, changing time), the file
content last modified (mtime, modification time), and last accessed (atime, access time).
 A link count telling how many hard links point to the inode.
 Pointers to the disk blocks that store the file's contents (see inode pointer structure).

The stat system call retrieves a file's inode number and some of the information in the inode.

[edit] Implications

 Files can have multiple names. If multiple names hard link to the same inode then the names are
equivalent. I.e., the first to be created has no special status. This is unlike symbolic links, which
depend on the original name, not the inode (number).
 An inode may have no links. Unlinked files are removed from disk and its resources are freed for
reallocation but deletion must wait until all processes that have opened it finish accessing it.
This includes executable files which are implicitly held open by the processes executing them.
 It is typically not possible to map from an open file to the filename that was used to open it. The
operating system immediately converts the filename to an inode number then discards the
filename. This means that the getcwd() and getwd() library functions search the parent
directory to find a file with an inode matching the working directory, then search that directory's
parent, and so on until reaching the root directory. SVR4 and Linux systems maintain extra
information to make this possible.
 Historically, it was possible to hard link directories. This made the directory structure into an
arbitrary directed graph as opposed to a directed acyclic graph (DAG). It was even possible for a
 directory to be its own parent. Modern systems generally prohibit this confusing state, except
that the parent of root is still defined as root.
 A file's inode number stays the same when it is moved to another directory on the same device,
or when the disk is defragmented which may change its physical location. This also implies that
completely conforming inode behavior is impossible to implement with many non-Unix file
systems, such as FAT and its descendants, which don't have a way of storing this lasting
"sameness" when both a file's directory entry and its data are moved around.
 Installation of new libraries is simple with inode filesystems. A running process can access a
library file while another process replaces that file, creating a new inode, and an all new
mapping will exist for the new file so that subsequent attempts to access the library get the new
version. This facility eliminates the need to reboot to replace currently mapped libraries. For this
reason, when updating programs, best practice is to delete the old executable first and create a
new inode for the updated version, so that any processes executing the old version may proceed
undisturbed.

[edit] Practical considerations

Many computer programs used by system administrators in UNIX operating systems often
designate files with inode numbers. Examples include popular disk integrity checking utilities
such as the fsck or pfiles. Thus, the need naturally arises to translate inode numbers to file
pathnames and vice versa. This can be accomplished using the file finding utility find with the
-inum option, or the ls command with the proper option (-i on POSIX compliant platforms).

It is possible to use up a device's set of inodes. When this happens, new files cannot be created
on the device, even though there may be free space available. For example, a mail server may
have many small files that don't fill up the disk, but use many inodes to point to the numerous
files.

Filesystems (such as JFS, ext4, or XFS) escape this limitation via support extents and/or dynamic
inode allocation, which can 'grow' the filesystem and/or increase the number of inodes.

Inode is a unique number given to a file in Unix OS.

Every
file in Unix has a inode number. unix treats
directories/folders as a file so they are also having a
inode value.
When a file system is created, data structures that
contain
information about files are created. Each file has an
inode
and is identified by an inode number (often "i-
number" or
even shorter, "ino") in the file system where it resides.
Inodes store information on files such as user and
group
ownership, access mode (read, write, execute
permissions)
and type of file. There is a fixed number of inodes,
which
indicates the maximum number of files each filesystem
can hold.

A file's inode number can be found using the ls -i

command,
while the ls -l command will retrieve inode
information.
This is description of inode information which it
contain:
 * The length of the file in bytes.
* Device ID (this identifies the device containing the
file).
* The User ID of the file's owner.
* The Group ID of the file.
* The file mode, which determines what users can
read,
write, and execute the file.
* Timestamps telling when the inode itself was last
modified (ctime, change time), the file content last
modified (mtime, modification time), and last accessed
(atime, access time).
* A reference count telling how many hard links
point to
the inode.
* Pointers to the disk blocks that store the file's
content

Unix File System Nodes (inodes)

Unix directories and files don't really have names.  They are numbered, using node numbers
called inodes, vnodes, or even gnodes (depending on the version of Unix).  You won't find the
name of a particular file or directory in or near the file or directory itself.  All the name-to-
number mappings of files and directories are stored in the parent directories.  For each file or
directory, a link count keeps track of how many parent directories contain a name-number
mapping for each node.  When a link count goes to zero, no directory points to the node and
Unix is free to reclaim the disk space.

Unix permits all files to have many name-to-number mappings.  So, a file may appear to have
several different "names" (Unix calls them "links"); that is, several names that all map to the
same node number (and thus to the same file).   Or, the file may have the same "name"; but, that
name may appear in different directories. 

Anyone can create a link to any file to which they have access. They don't need to be able to read
or write the file itself to make the link; they only need write permission on the directory in which
the name-to-number map (the name, or "link") is being created.

Directories are not allowed to have many name-to-number mappings.  Each directory name-to-
number map is allowed to appear in exactly one parent directory and no more.   This restriction
means that every directory has only one "name".  It prevents loops and cycles in the file system
tree.  (Many things are simpler if the tree has no cycles.)

Since a parent directory may have many sub-directories, and since the name ".." (dot dot) in
every one of those sub-directories is a map to the node number of the parent directory, the link
count of the parent directory is increased by one for every sub-directory the parent contains. 
Every directory also contains the name "." (dot), a map to the directory itself, so the smallest
link count of any Unix directory is 2: one for the map in the parent directory that gives the
directory its "name", and one for the dot map in the directory itself.

Example

Suppose the root directory has node number #2.  Here is a small part of a Unix file system tree,
showing hypothetical node numbers:

     

Node #2 Node #555 Node #123

. (dot) 2 . (dot) 555 . (dot) 123

.. (dot dot) 2 .. (dot dot) 2 .. (dot dot) 2

home 123 rm 546 ian 111

bin 555 ls 984 stud0002 755

 usr 654 cp 333 stud0001 883

ln 333 stud0003 221

mv 333

Note how one directory (#555) has three name-to-number maps for the same node.   All three names (cp, ln, mv) refer
to the same node number, in this case a file containing an executable program.  (This program looks at its name and
behaves differently depending on which name you use to call it.)

 
   

Node #111 Node #333 Node #335

. (dot) 111
Disk blocks Disk blocks
.. (dot dot) 123
for the for the
.profile 334
cp / ln / mv .login
.login 335
file file
.logout 433

(link count: 3) (link count: 1)

Example

Here are two shell programs that are linked into different directories under different names.  The
only way you can tell which names point to the same program files is by looking at the inode
numbers using the "-i" option to ls:

# ls -i /sbin/sh /usr/bin/sh
136724 /sbin/sh 279208 /usr/bin/sh
# ncheck -i 279208,136724
/dev/dsk/c0t3d0s0:
279208 /usr/lib/rsh
136724 /sbin/jsh
136724 /sbin/sh
279208 /usr/bin/jsh
279208 /usr/bin/sh

The ncheck command is usable only by the Super User. It finds all pathnames that lead to a
particular inode.
Damage

When a Unix file system suffers damage, one or more nodes may become unreadable.  If the
damaged or lost nodes are file nodes, the file content pointed to by those nodes will be missing
or incomplete.  If any of the nodes are directory nodes, containing the names of files and sub-
directories, the files and sub-directories that were once mapped by those nodes will lose their
"names".

The Unix file-system checking program ("fsck") usually notices the existence of files and sub-
directories that no longer have names, and it gives them false names and links them into a special
directory  named "lost+found" when the system reboots itself.  The system admin must go into
the directory and figure out what the files are, what their names are, and where they belong.

Many File Systems

A Unix file system is equivalent to a single disk partition. Each Unix file system has its own set
of node numbers. Since the overall hierarchical tree on a Unix system may transparently include
pieces from several file systems, some items in the hierarchical tree will appear to have the same
node numbers, but will actually be different files residing on different file systems.

A directory's name-to-number mapping applies only within a single Unix file system. It isn't
possible for a directory to map to a node number in a different file system (i.e. in a different disk
partition). A special "mount" command is used to splice together different file systems into one
hierarchical tree.

 Answer:

All UNIX files have its description stored in a structure called inode. The inode contains
info about the file-size, its location, time of last access, time of last modification,
permission and so on. Directories are also represented as files and have an associated
inode. In addition to descriptions about the file, the inode contains pointers to the data
blocks of the file. If the file is large, inode has indirect pointer to a block of pointers to
additional data blocks (this further aggregates for larger files). A block is typically 8k.
Inodeconsistsofthefollowingfields:
Fileowneridentifier
Filetype

Fileaccesspermissions
Fileaccesstimes
Numberoflinks
Filesize
Location of the file data

UNIX INODE STRUCTURE

Mode (file type and permissions)
Link count
Owner's UID number
Owner's GID number
File size in bytes
Time file was last accessed
Time file was last modified
Time inode was last changed
12 direct block pointers
(32/64 bits each)
to reference up to 96KB
1 single indirect block pointer
(32/64 bits) to reference up to 16MB
1 double indirect block pointer
(32/64 bits) to reference up to 32GB
1 triple indirect block pointer
(32/64 bits) to reference up to 70TB
Count of data blocks actually held
1 data block
(8KB)
per pointer
2048 direct pointers
1 data block
(8KB)
per pointer
2048 indirect pointers
2048 direct pointers
1 data block
(8KB)
per pointer
Other direct pointers
1 data block
(8KB)
per pointer
2048 indirect pointers 2048 indirect pointers 2048 direct pointers
Other indirect pointers Other direct pointers
© 2000 Integrated Services
info@iserv.xs4all.nl
Author: Drs.M.Waldorp-Bonk
inode.sda 20000709

Inode status (flags)

Optional: extra fields/reserved fields
Legenda:
- each (unix) file system has its own inode table; on disk each
cylinder group will hold a relevant part of that table
- each inode is referenced by a "device + inode number" pair
- each file is assigned an inode number which is unique within
that file system; each directory structure will consist of a list of
"filename + inode number" pairs; inodes won't hold filenames
- reserved inode numbers: 0, 1, 2
0: deleted files/directories
1: (fs dependent) file system creation time/bad blocks count/.....
2: refers to the root directory of the file system
- the "mode" field will always be the first field in the inode;
the order of the other fields is file system dependent
- timestamps: in seconds since 00:00:00 GMT 01-01-1970
- access time: updated after each read/write of file
- modification time: updated after each write to file
- inode change time: updated after each modification of one of
the fields in the inode (chmod, chown, chgrp, ln, ...)
- triple indirect pointer: use is fs and max.file size dependent
- status/flags like "compress file" or "do not update access time"
or "do not extend file" are file system dependent
- extra fields may hold: an inode generation number (for NFS)
and/or ACL info (sometimes this field contains a "continuation
inode number": a pointer to a special inode that holds ACL info)
and/or a file type identification (for device files: major and minor
number; for directories: inode number of parent directory);
all extra/reserved fields are file system dependent!

The UNIX filesystem controls the way that information in files and directories is stored on disk
and other forms of secondary storage. It controls which users can access what items and how.
The filesystem is therefore one of the most basic tools for enforcing UNIX security on your
system.

Information stored in the UNIX filesystem is arranged as a tree structure of directories and files.
The tree is constructed from directories and subdirectories within a single directory, which is
called the root . [1] Each directory, in turn, can contain other directories or entries such as files,
pointers (symbolic links) to other parts of the filesystem, logical names that represent devices
(such as /dev/tty ), and many other types.[2]

[1] This is where the "root" user (superuser) name originates: the owner of the root of the
filesystem.

[2] For example, the UNIX "process" filesystem in System V contains entries that represent
processes that are currently executing.

This chapter explains, from the user's point of view, how the filesystem represents and protects
information.

5.1 Files

From the simplest perspective, everything visible to the user in a UNIX system can be
represented as a "file" in the filesystem  - including processes and network connections. Almost
all of these items are represented as "files" each having at least one name, an owner, access
rights, and other attributes. This information is actually stored in the filesystem in an inode
(index node), the basic filesystem entry. An inode stores everything about a filesystem entry
except its name; the names are stored in directories and are associated with pointers to inodes.

5.1.1 Directories

One special kind of entry in the filesystem is the directory . A directory is nothing more than a
simple list of names and inode numbers. A name can consist of any string of any characters with
the exception of a "/" character and the "null" character (usually a zero byte).[3] There is a limit
to the length of these strings, but it is usually quite long: 1024 or longer on many modern
versions of UNIX ; older AT&T versions limit names to 14 characters or less.

[3] Some versions of UNIX may further restrict the characters that can be used in filenames and
directory names.

These strings are the names of files, directories, and the other objects stored in the filesystem.
Each name can contain control characters, line feeds, and other characters. This can have some
interesting implications for security, and we'll discuss those later in this and other chapters.

Associated with each name is a numeric pointer that is actually an index on disk for an inode. An
inode contains information about an individual entry in the filesystem; these contents are
described in the next section.

Nothing else is contained in the directory other than names and inode numbers. No protection
information is stored there, nor owner names, nor data. The directory is a very simple relational
database that maps names to inode numbers. No restriction on how many names can point to the
same inode exists, either. A directory may have 2, 5, or 50 names that each have the same inode
number. In like manner, several directories may have names that associate to the same inode.
These are known as links [4] to the file. There is no way of telling which link was the first
created, nor is there any reason to know: all the names are equal in what they access. This is
often a confusing idea for beginning users as they try to understand the "real name" for a file.

[4] These are hard links or direct links . Some systems support a different form of pointer,
known as a symbolic link , that behaves in a different way.

This also means that you don't actually delete a file with commands such as rm. Instead, you
unlink the name - you sever the connection between the filename in a directory and the inode
number. If another link still exists, the file will continue to exist on disk. After the last link is
removed, and the file is closed, the kernel will reclaim the storage because there is no longer a
method for a user to access it.

Every normal directory has two names always present. One entry is for " . " (dot), and this is
associated with the inode for the directory itself; it is self-referential. The second entry is for " .. "
(dot-dot), which points to the "parent" of this directory - the directory next closest to the root in
the tree-structured filesystem. The exception is the root directory itself, named "/". In the root
directory, ".." is also a link to "/".

5.1.2 Inodes

For each object in the filesystem, UNIX stores administrative information in a structure known
as an inode. Inodes reside on disk and do not have names. Instead, they have indices (numbers)
indicating their positions in the array of inodes.

Each inode generally contains:

  The location of the item's contents on the disk, if any

 The item's type (e.g., file, directory, symbolic link)

 The item's size, in bytes, if applicable

 The time the file's inode was last modified (the ctime )

 The time the file's contents were last modified (the mtime )

 The time the file was last accessed (the atime ) for read ( ) , exec ( ), etc

 A reference count: the number of names the file has

 The file's owner (a UID )

 The file's group (a GID )

 The file's mode bits (also called file permissions or permission bits )

The last three pieces of information, stored for each item, and coupled with UID/GID
information about executing processes, are the fundamental data that UNIX uses for practically
all operating system security.

Other information can also be stored in the inode, depending on the particular version of UNIX
involved. Some systems may also have other nodes such as vnodes, cnodes, and so on. These are
simply extensions to the inode concept to support foreign files, RAID [5] disks, or other special
kinds of filesystems. We'll confine our discussion to inodes, as that abstraction contains most of
the information we need.

[5] RAID means Redundant Array of Inexpensive Disks. It is a technique for combining many
low-cost hard disks into a single unit that offers improved performance and reliability.

Figure 5.1 shows how information is stored in an inode.

Figure 5.1: Files and inodes

 Figure 5.1: Files and inodes

5.1.3 Current Directory and Paths

Every item in the filesystem with a name can be specified with a pathname . The word pathname
is appropriate because a pathname represents the path to the entry from the root of the filesystem.
By following this path, the system can find the inode of the referenced entry.

Pathnames can be absolute or relative. Absolute pathnames always start at the root, and thus
always begin with a "/", representing the root directory. Thus, a pathname such as
/homes/mortimer/bin/crashme represents a pathname to an item starting at the root directory.

A relative pathname always starts interpretation from the current directory of the process
referencing the item. This concept implies that every process has associated with it a current
directory . Each process inherits its current directory from a parent process after a fork (see
Appendix C, UNIX Processes ). The current directory is initialized at login from the sixth field
of the user record in the /etc/passwd file: the home directory . The current directory is then
updated every time the process performs a change-directory operation (chdir or cd). Relative
pathnames also imply that the current directory is at the front of the given pathname. Thus, after
executing the command cd /usr, the relative pathname lib/makekey would actually be referencing
the pathname /usr/lib/makekey . Note that any pathname that doesn't start with a "/" must be
relative.

5.1.4 Using the ls Command

You can use the ls command to list all of the files in a directory. For instance, to list all the files
in your current directory, type:

%
ls -a
 instructions letter notes invoice more-stuff stats %

You can get a more detailed listing by using the ls -lF command:

%
ls -lF
total 161 -rw-r--r-- 1 sian user 505 Feb 9 13:19 instructions
-rw-r--r-- 1 sian user 3159 Feb 9 13:14 invoice -rw-r--r-- 1 sian
user 6318 Feb 9 13:14 letter -rw------- 1 sian user 15897
Feb 9 13:20 more-stuff -rw-r----- 1 sian biochem 4320 Feb 9 13:20
notes -rwxr-xr-x 1 sian user 122880 Feb 9 13:26 stats* %

The first line of output generated by the ls command ("total 161" in the example above)
indicates the number of kilobytes taken up by the files in the directory.[6] Each of the other lines
of output contains the fields, from left to right, as described in Table 5.1 .

[6] Some older versions of UNIX reported this in 512-byte blocks rather than in kilobytes.

Table 5.1: ls Output

Field Contents Meaning

- The file's type; for regular files, this field is always a dash
rw-r--r-- The file's permissions
1 The number of "hard" links to the file; the number of "names" for the file
sian The name of the file's owner
user The name of the file's group
505 The file's size, in bytes
Feb 9 13:19 The file's modification time
instructions The file's name

The ls -F option makes it easier for you to understand the listing by printing a special character
after the filename to indicate what it is, as shown in Table 5.2 .

Table 5.2: ls -F Tag Meanings

Symbol Meaning
(blank) Regular file or named pipe (FIFO[7])
* Executable program or command file
/ Directory
= Socket
@ Symbolic link

[7] A FIFO is a First-In, First-Out buffer, which is a special kind of named pipe.

Thus, in the directory shown earlier, the file stats is an executable program file; the rest of the
files are regular text files.
The -g option to the ls command alters the output, depending on the version of UNIX being
used.

If you are using the Berkeley-derived version of ls,[8] you must use the ls -g option to display
the file's group in addition to the file's owner:

[8] On Solaris systems, this program is named /usr/ucb/ls .

%
ls -lFg
total 161 -rw-r--r-- 1 sian user 505 Feb 9 13:19 instructions
-rw-r--r-- 1 sian user 3159 Feb 9 13:14 invoice -rw-r--r-- 1 sian
user 6318 Feb 9 13:14 letter -rw------- 1 sian user 15897 Feb
9 13:20 more-stuff -rw-r----- 1 sian biochem 4320 Feb 9 13:20 notes
-rwxr-xr-x 1 sian user 122880 Feb 9 13:26 stats* %

If you are using an AT&T -derived version of ls,[9] using the -g option causes the ls command to
only display the file's group:

[9] On Solaris systems, this program is named /bin/ls .

%
ls -lFg
total 161 -rw-r--r-- 1 user 505 Feb 9 13:19 instructions -rw-r--r--
1 user 3159 Feb 9 13:14 invoice -rw-r--r-- 1 user 6318 Feb 9
13:14 letter -rw------- 1 user 15897 Feb 9 13:20 more-stuff -rw-r-----
1 biochem 4320 Feb 9 13:20 notes -rwxr-xr-x 1 user 122880 Feb 9
13:26 stats* %

5.1.5 File Times

The times shown with the ls -l command are the modification times of the files ( mtime). You
can obtain the time of last access (the atime) by providing the -u option (for example, by typing
ls -lu ). Both of these time values can be changed with a call to a system library routine.[10]
Therefore, as the system administrator, you should be in the habit of checking the inode change
time ( ctime) by providing the -c option; for example, ls -lc . You can't reset the ctime of a file
under normal circumstances. It is updated by the operating system whenever any change is made
to the inode for the file.

[10] utimes ( )

Because the inode changes when the file is modified, ctime reflects the time of last writing,
protection change, or change of owner. An attacker may change the mtime or atime of a file, but
the ctime will usually be correct.

Note that we said "usually." A clever attacker who gains superuser status can change the system
clock and then touch the inode to force a misleading ctime on a file. Furthermore, an attacker can
change the ctime by writing to the raw disk device and bypassing the operating system checks
altogether. And if you are using Linux with the ext2 filesystem, an attacker can modify the inode
contents directly using the debugfs command.

For this reason, if the superuser account on your system has been compromised, you should not
assume that any of the three times stored with any file or directory are correct.

NOTE: Some programs will change the ctime on a file without actually changing the file itself.
This can be misleading when you are looking for suspicious activity. The file command is one
such offender. The discrepancy occurs because file opens the file for reading to determine its
type, thus changing the atime on the file. By default, most versions of file then reset the atime to
its original value, but in so doing change the ctime. Some security scanning programs use the file
program within them (or employ similar functionality), and this may result in wide-scale changes
in ctime unless they are run on a read-only version of the filesystem.

5.1.6 Understanding File Permissions

The file permissions on each line of the ls listing tell you what the file is and what kind of file
access (that is, ability to read, write, or execute) is granted to various users on your system.

Here are two examples of file permissions:

-rw------- drwxr-xr-x

The first character of the file's mode field indicates the type of file described in Table 5.3 .

Table 5.3: File Types

Contents Meaning
- Plain file
d Directory
c Character device (tty or printer)
b Block device (usually disk or CD-ROM)
l Symbolic link ( BSD or V.4)
s Socket ( BSD or V.4)
= or p FIFO (System V, Linux)

The next nine characters taken in groups of three indicate who on your computer can do what
with the file. There are three kinds of permissions:

Permission to read

w
 Permission to write

Permission to execute

Similarly, there are three classes of permissions:

owner

The file's owner

group

Users who are in the file's group

other

Everybody else on the system (except the superuser)

In the ls -l command, privileges are illustrated graphically (see Figure 5.2 ).

Figure 5.2: Basic permissions

5.1.7 File Permissions in Detail

The terms read , write , and execute have very specific meanings for files, as shown in Table
5.4 .

Table 5.4: File Permissions

Character Permission Meaning

r READ Read access means exactly that: you can open a file with the open()
system call and you can read its contents with read.
w WRITE Write access means that you can overwrite the file with a new one or
modify its contents. It also means that you can use write() to make the
file longer or truncate() or ftruncate() to make the file shorter.
 Table 5.4: File Permissions

Character Permission Meaning

x EXECUTE Execute access makes sense only for programs. If a file has its execute
bits set, you can run it by typing its pathname (or by running it with one
of the family of exec() system calls). How the program gets executed
depends on the first two bytes of the file.
The first two bytes of an executable file are assumed to be a magic
number indicating the nature of the file. Some numbers mean that the file
is a certain kind of machine code file. The special two-byte sequence "#!"
means that it is an executable script of some kind. Anything with an
unknown value is assumed to be a shell script and is executed
accordingly.

File permissions apply to devices, named sockets, and FIFOS exactly as they do for regular files.
If you have write access, you can write information to the file or other object; if you have read
access, you can read from it; and if you don't have either access, you're out of luck.

File permissions do not apply to symbolic links. Whether or not you can read the file pointed to
by a symbolic link depends on the file's permissions, not the link's. In fact, symbolic links are
almost always created with a file permission of "rwxrwxrwx" (or mode 0777, as explained later
in this chapter) and are ignored by the operating system.[11]

[11] Apparently, some vendors have found a use for the mode bits inside a symbolic link's inode.
HP-UX 10.0 uses the sticky bit of symbolic links to indicate "transition links" - portability links
to ease the transition from previous releases to the new SVR4 filesystem layout.

Note the following facts about file permissions:

 You can have execute access without having read access. In such a case, you can run a
program without reading it. This ability is useful in case you wish to hide the function of
a program. Another use is to allow people to execute a program without letting them
make a copy of the program (see the note later in this section).

 If you have read access but not execute access, you can then make a copy of the file and
run it for yourself. The copy, however, will be different in two important ways: it will
have a different absolute pathname; and it will be owned by you, rather than by the
original program's owner.

 On some versions of UNIX (including Linux), an executable command script must have
both its read and execute bits set to allow people to run it.

Most people think that file permissions are pretty basic stuff. Nevertheless, many UNIX systems
have had security breaches because their file permissions are not properly set.
NOTE: Sun's Network Filesystem ( NFS ) servers allow a client to read any file that has either
the read or the execute permission set. They do so because there is no difference, from the NFS
server's point of view, between a request to read the contents of a file by a user who is using the
read() system call and a request to execute the file by a user who is using the exec() system
call. In both cases, the contents of the file need to be transferred from the NFS server to the NFS
client. (For a detailed description, see Chapter 20, NFS .)

Get a Grip on the Grep! – 15 Practical Grep

Command Examples (this is for linux be
generic for other flavours)
First create the following demo_file that will be used in the examples below to demonstrate grep
command.

$ cat demo_file
THIS LINE IS THE 1ST UPPER CASE LINE IN THIS FILE.
this line is the 1st lower case line in this file.
This Line Has All Its First Character Of The Word With Upper Case.

Two lines above this line is empty.

And this is the last line.

1. Search for the given string in a single file

The basic usage of grep command is to search for a specific string in the specified file as shown
below.

Syntax:
grep "literal_string" filename
$ grep "this" demo_file
this line is the 1st lower case line in this file.
Two lines above this line is empty.

2. Checking for the given string in multiple files.

Syntax:
grep "string" FILE_PATTERN

This is also a basic usage of grep command. For this example, let us copy the demo_file to
demo_file1. The grep output will also include the file name in front of the line that matched the
specific pattern as shown below. When the Linux shell sees the meta character, it does the
expansion and gives all the files as input to grep.

$ cp demo_file demo_file1

$ grep "this" demo_*

demo_file:this line is the 1st lower case line in this file.
demo_file:Two lines above this line is empty.
demo_file:And this is the last line.
demo_file1:this line is the 1st lower case line in this file.
demo_file1:Two lines above this line is empty.
demo_file1:And this is the last line.

3. Case insensitive search using grep -i

Syntax:
grep -i "string" FILE

This is also a basic usage of the grep. This searches for the given string/pattern case
insensitively. So it matches all the words such as “the”, “THE” and “The” case insensitively as
shown below.

$ grep -i "the" demo_file

THIS LINE IS THE 1ST UPPER CASE LINE IN THIS FILE.
this line is the 1st lower case line in this file.
This Line Has All Its First Character Of The Word With Upper Case.
And this is the last line.

4. Match regular expression in files

Syntax:
grep "REGEX" filename

This is a very powerful feature, if you can use use regular expression effectively. In the
following example, it searches for all the pattern that starts with “lines” and ends with “empty”
with anything in-between. i.e To search “lines[anything in-between]empty” in the demo_file.

Determining a File's Type

To determine a file's type, specify the -l option to the ls. When this option is specified, ls lists the file type for the
specified files. For example, the command

$ ls -l /home/ranga/.profile
produces the following output:

-rwxr-xr-x 1 ranga users 2368 Jul 11 15:57 .profile*

Here, you see that the very first character is a hyphen (-). This indicates that the file is a regular file. For special
files, the first character will be one of the letters given in Table 5.1.

To obtain file type information about a directory, you must specify the -d option along with the -l option:
$ ls -ld /home/ranga
This produces the following output:

drwxr-xr-x 27 ranga users 2048 Jul 23 23:49

/home/ranga/

Table 5.1  Special Characters for Different File Types

Character File Type

- Regular file

l Symbolic link

c Character special

b Block special

p Named pipe

s Socket

d Directory file

I'll provide the actual descriptions of each of these file types in the following sections.

Regular Files

Regular files are the most common type of files you will encounter. These files store any kind of data.
This data can be stored as plain text, an application- specific format, or a special binary format that the system can
execute.

UNIX does not have to understand the data contained in a regular file. A regular file can store any form of raw data
because UNIX does not interpret the data that is in the file.

Note - Often simply determining that a file is a regular file tells you very little about the file itself.
Usually you need to know whether a particular file is a binary program, a shell script, or a library. In
these instances, the file program is very useful.

It is invoked as follows:

file filename
 Here, filename is the name of the file you want to examine. As an example, on my system, the
command

$ file /sbin/sh
produces the following output:

/sbin/sh: ELF 32-bit MSB executable SPARC Version 1,

statically linked, stripped
Here you see that the file, /sbin/sh, is an executable program. Try it out on a few files to get an idea of
the kind of information that it can give you.

Symbolic Links

A symbolic link is a special file that points to another file on the system. When you access one of these
files, it has a pathname stored inside it. Use this pathname to advance to the file or directory on the system
represented by the pathname stored in the symbolic link.

For readers who are familiar with Windows or Mac OS, a symbolic link is similar to a shortcut or an alias.

You can use symbolic links to make a file appear as though it is located in many different places or has many
different names in the file system. Symbolic links can point to any type of file or directory.

The ls -l output for a symbolic link looks like this:

lrwxrwxrwx 1 root root 9 Oct 23 13:58 /bin/

-> ./usr/bin/
The output indicates that the directory /bin is really a link to the directory ./usr/bin.

The relative path in the output is not relative to your current working directory: it is relative to the directory where
the link resides. In this case, the link /bin resides in the / directory, thus ./usr/bin indicates that /bin is a link to the
directory /usr/bin.

Creating Symbolic Links

Create symbolic links using the ln command with the -s option. The syntax is as follows:

ln -s source destination
Here, source is either the absolute or relative path to the original version of the file, and destination is the name you
want the link to have.

For example, the following command

$ ln -s /home/httpd/html/users/ranga /home/ranga/public_html
creates a link in my home directory to my Web files. If you encounter an error while creating a link, ln will display
an error message. Otherwise, it displays no output.
In this example, you used absolute paths. In practice, relative paths are preferred for the source and the destination.
For example, the actual commands I used to create a link to my Web files are the following:

$ cd
$ ln -s ../httpd/html/users/ranga ./public_html
You can see the relative path by using ls -l:

$ ls -l ./public_html
lrwxrwxrwx 1 ranga users 26 Nov 9 1997 public_html
-> ../httpd/html/users/ranga
This output indicates that the file is a link and also shows the file or directory that the link points to.

Common Errors
The two most common errors encountered when creating symbolic links happen when

 The destination already exists.

 The destination is a directory.

If the specified destination is a file, it does not create the requested link. For example, if the file .exrc exists in my
home directory, the command

$ ln -s /etc/exrc .exrc
produces the following error message:

ln: cannot create .exrc: File exists

If the specified destination is a directory, ln creates a link in that directory with the same name as the source. For
example, if the directory pub exists in the current directory, the following command

$ ln -s /home/ftp/pub/ranga pub
creates the link pub/ranga rather than complaining that the destination is a directory. I mention this behavior of ln as
a common error because forgetting about that fact is a common shell script bug.

Device Files
You can access UNIX devices through reading and writing to device files. These device files are access points to the
device within the file systems.

Usually, device files are located under the /dev directory. The two main types of device files are

 Character special files

 Block special files

Character Special Files

Character special files provide a mechanism for communicating with a device one character at a time.
Usually character devices represent a "raw" device. The output of ls on a character special file looks like the
following:
crw------- 1 ranga users 4, 0 Feb 7 13:47 /dev/tty0

The first letter in the output is c, therefore you know that this particular file is a character special file, but
you also see two extra numbers before the date. The first number is called the major number and the second number
is called the minor number. UNIX uses these two numbers to identify the device driver that this file communicates
with.

Block Special Files

Block special files also provide a mechanism for communicating with device drivers via the file system.
These files are called block devices because they transfer large blocks of data at a time. This type of file typically
represents hard drives and removable media.

Look at the ls -l output for a typical block device. For example, /dev/sda:

brw-rw---- 1 root disk 8, 0 Feb 7 13:47 /dev/sda

Here the first character is b, indicating that this file is a block special file. Just like the character special files, these
files also have a major and a minor number.

Named Pipes

One of the greatest features of UNIX is that you can redirect the output of one program to the input of
another program with very little work. For example, the command who | grep ranga takes the output of the who
command and makes it the input to the grep command. This is called piping the output of one command into
another. You will examine input and output redirection in great detail in Chapter 13, "Input/Output."

On the command line, temporary anonymous pipes are used, but sometimes more control is needed than the
command line provides. For such instances, UNIX provides a way to create a named pipe, so that two or more
process can communicate with each other via a file that acts like a pipe. Because these files allow process to
communicate with one another, they are one of the most popular forms of interprocess communication (IPC for
short) available under UNIX.

Sockets

Socket files are another form of interprocess communication, but sockets can pass data and information
between two processes that are not running on the same machine. Socket files are created when communication to a
process on another machine located on a network is required. Internet tools in use today, such as Web browsers, use
sockets to make a connection to the Web server.

Solaris 10 Filesystems

Softw are

<< PreviousNext >>

What Are Your Choices?

May 26, 2008,
Volume 123, Issue 4
 Solaris 10 provides admins with more [storage] choices than any other operating system.

-- John Finigan
 

One of the most enthusiastic responses around to the various file system choices in the Solaris 10 Operating System
(Solaris OS) can be found in the OSNews article by John Finigan, who says, "Solaris 10 provides admins with more
[storage] choices than any other operating system. Right out of the box, it offers two filesystems, two volume
managers, an iscsi target and initiator, and, naturally, an NFS server. Add a couple of Sun packages and you have
volume replication, a cluster filesystem, and a hierarchical storage manager...in one well-tested package. Trust your
data to the still-in-development features found in OpenSolaris, and you can have a fibre channel target and an in-
kernel CIFS server, among other things." Quite a package!

Finigan discusses the four common on-disk filesystems for Solaris 10 OS, which are: UFS, ZFS, SAM and QFS, and
VxFS. Of each in turn he writes:

UFS

UFS is old technology but stable, fast filesystem whose code Sun has continuously tuned and improved over the last
decade to the extent, probably, of having wrung the last bit of performance out of this particular type of filesystem.
Solaris 10 OS can only boot from a UFS root filesystem, Finigan notes but, nevertheless, UFS remains a good
choice for databases though it will likely be replaced by ZFS.

ZFS

No magic bullet, Finigan writes, this "first third-generation UNIX FS," which is both hybrid filesystem and volume
manager. Integrating these two functionalities makes ZFS flexible but also the source of layering violation, which he
sees as a minor impediment. A big plus with ZFS in Finigan's view is not having to worry about resizing a
filesystem. He also likes the robustness of ZFS's error checking, which is validated on every read and write. Bad
disks, bad controllers and bad fibre paths are protected against. Also finding favor with Finigan is the copy-on-write
feature of ZFS, which makes for nearly free snapshot and clone functionality, as well as the solution's administrative
features and self-healing capability. Eventually, he would like to see per-user-quota support but he still sees ZFS as
the best filesystem option.

SAM and QFS

Though SAM and QFS are designed to work together (the former is a hierarchical storage manager, the latter a
cluster filesystem), each may be used separately. Without the need for an extra LVM to do striping or concatenation,
a QFS filesystem can span multiple disks, allowing data to be striped or round-robined. With QFS, metadata can be
separated from data making it possible for a few disks to serve random metadata and the rest to serve a sequential
data workload. As QFS cannot manage its own RAID if it is striping, users would need to employ a hardware
controller, a traditional volume manager or a raw ZFS volume.

VxFS

Finigan characterizes VxFS as a fast, extent-based, journaled, clusterable filesystem. Using VxFS, he writes, "If you
ever wanted to unmount a volume from your AIX box and mount it on Linux or Solaris, now you can." VxFS and its
tightly integrated companion, VxVM, enable users to resize filesystems and their underlying volumes with little
trouble. One reservation Finigan registers about these twins is that they are closed source solutions.

Other options in the Sun Open Storage line -- though not mentioned by Finigan -- that deserve attention include the
following:

Squashfs

Squashfs is a compressed read-only file system optimized for memory-constrained systems. Project members have
plans to write a complete driver that will enable the kernel to boot from a Squashfs file system, thus enabling Live-
CD/DVDs to use it. This would also allow Solaris to access embedded devices using Squashfs, such as some
Motorola phones.

JFS: Journaled File System

JFS provides users with a log-based, byte-level file system developed for transaction-oriented, high performance
systems. JFS is scalable and robust and, with its quick restart capability, can restore a file system to a consistent state
within seconds or minutes.

Ext3 File System Support

Recent achievements in this project have enabled the driver to mount and clearly unmount ext2 or ext3 filesystem;
browse directories; read, write, remove or rename files, links and directories; check and set file's attributes or owner;
create link, symlink, file and directory. In the works are improvements in performance and stability; enabling
POSIX compliance; added support for both extended attributes and for SPARC-based machines.

Fuse on OpenSolaris

FUSE stands for 'File system in User Space'. It provides a simple interface to allow implementation of a fully
functional file system in user-space. FUSE originates from the Linux community and is included in the Linux kernel
(2.6.14+). The team is working on a port of FUSE to OpenSolaris using the FUSE implementation for FreeBSD as a
reference with the implementation for OpenSolaris re-using much code from fuse4bsd. Documentation about the
FreeBSD version of FUSE can be found at the fuse4bsd website.

ZFS on Disk Encryption Support

This project is working to provide on-disk encryption/decryption support for ZFS datasets. The project will cover
the addition of encryption and decryption to the ZFS IO pipeline and the key management for ZFS datasets. Results
will deliver in multiple phases to support different key management strategies, including one that provides support
for secure deletion based on encrypted datasets. [...read more...]
A file system (often also written as filesystem) is a method of storing and organizing computer
files and their data. Essentially, it organizes these files into a database for the storage,
organization, manipulation, and retrieval by the computer's operating system.

File systems are used on data storage devices such as hard disks or CD-ROMs to maintain the
physical location of the files. Beyond this, they might provide access to data on a file server by
acting as clients for a network protocol (e.g., NFS, SMB, or 9P clients), or they may be virtual
and exist only as an access method for virtual data (e.g., procfs). It is distinguished from a
directory service and registry.

Aspects of file systems

Most file systems make use of an underlying data storage device that offers access to an array of
fixed-size physical sectors, generally a power of 2 in size (512 bytes or 1, 2, or 4 KiB are most
common). The file system is responsible for organizing these sectors into files and directories,
and keeping track of which sectors belong to which file and which are not being used. Most file
systems address data in fixed-sized units called "clusters" or "blocks" which contain a certain
number of disk sectors (usually 1-64). This is the smallest amount of disk space that can be
allocated to hold a file. However, file systems need not make use of a storage device at all. A file
system can be used to organize and represent access to any data, whether it is stored or
dynamically generated (e.g., procfs).

[edit] File names

A file name is a name assigned to a file in order to secure storage location in the computer
memory. Whether the file system has an underlying storage device or not, file systems typically
have directories which associate file names with files, usually by connecting the file name to an
index in a file allocation table of some sort, such as the FAT in a DOS file system, or an inode in
a Unix-like file system. Directory structures may be flat, or allow hierarchies where directories
may contain subdirectories. In some file systems, file names are structured, with special syntax
for filename extensions and version numbers. In others, file names are simple strings, and per-
file metadata is stored elsewhere.

[edit] Metadata

Other bookkeeping information is typically associated with each file within a file system. The
length of the data contained in a file may be stored as the number of blocks allocated for the file
or as an exact byte count. The time that the file was last modified may be stored as the file's
timestamp. Some file systems also store the file creation time, the time it was last accessed, and
the time that the file's meta-data was changed. (Note that many early PC operating systems did
not keep track of file times.) Other information can include the file's device type (e.g., block,
character, socket, subdirectory, etc.), its owner user-ID and group-ID, and its access permission
settings (e.g., whether the file is read-only, executable, etc.).

Arbitrary attributes can be associated on advanced file systems, such as NTFS, XFS, ext2/ext3,
some versions of UFS, and HFS+, using extended file attributes. This feature is implemented in
the kernels of Linux, FreeBSD and Mac OS X operating systems, and allows metadata to be
associated with the file at the file system level. This, for example, could be the author of a
document, the character encoding of a plain-text document, or a checksum.

[edit] Hierarchical file systems

The hierarchical file system (not to be confused with Apple's HFS) was an early research interest
of Dennis Ritchie of Unix fame; previous implementations were restricted to only a few levels,
notably the IBM implementations, even of their early databases like IMS. After the success of
Unix, Ritchie extended the file system concept to every object in his later operating system
developments, such as Plan 9 and Inferno.

[edit] Facilities

Traditional file systems offer facilities to create, move and delete both files and directories. They
lack facilities to create additional links to a directory (hard links in Unix), rename parent links
(".." in Unix-like OS), and create bidirectional links to files.

Traditional file systems also offer facilities to truncate, append to, create, move, delete and in-
place modify files. They do not offer facilities to prepend to or truncate from the beginning of a
file, let alone arbitrary insertion into or deletion from a file. The operations provided are highly
asymmetric and lack the generality to be useful in unexpected contexts. For example,
interprocess pipes in Unix have to be implemented outside of the file system because the pipes
concept does not offer truncation from the beginning of files.

[edit] Secure access

See also: Secure computing

Secure access to basic file system operations can be based on a scheme of access control lists or
capabilities. Research has shown access control lists to be difficult to secure properly, which is
why research operating systems tend to use capabilities.[citation needed] Commercial file systems still
use access control lists.

[edit] Types of file systems

File system types can be classified into disk file systems, network file systems and special
purpose file systems.

[edit] Disk file systems

A disk file system is a file system designed for the storage of files on a data storage device, most
commonly a disk drive, which might be directly or indirectly connected to the computer.
Examples of disk file systems include FAT (FAT12, FAT16, FAT32, exFAT), NTFS, HFS and
HFS+, HPFS, UFS, ext2, ext3, ext4, btrfs, ISO 9660, ODS-5, Veritas File System, VMFS ZFS,
ReiserFS, Linux SWAP and UDF. Some disk file systems are journaling file systems or
versioning file systems.

ISO 9660 and Universal Disk Format are the two most common formats that target Compact
Discs and DVDs. Mount Rainier is a newer extension to UDF supported by Linux 2.6 series and
Windows Vista that facilitates rewriting to DVDs in the same fashion as has been possible with
floppy disks.

[edit] Flash file systems

Main article: Flash file system

A flash file system is a file system designed for storing files on flash memory devices. These are
becoming more prevalent as the number of mobile devices is increasing, and the capacity of flash
memories increase.

While a disk file system can be used on a flash device, this is suboptimal for several reasons:

 Erasing blocks: Flash memory blocks have to be explicitly erased before they can be rewritten.
The time taken to erase blocks can be significant, thus it is beneficial to erase unused blocks
while the device is idle.
 Random access: Disk file systems are optimized to avoid disk seeks whenever possible, due to
the high cost of seeking. Flash memory devices impose no seek latency.
 Wear levelling: Flash memory devices tend to wear out when a single block is repeatedly
overwritten; flash file systems are designed to spread out writes evenly.

Log-structured file systems have many of the desirable properties for a flash file system. Such
file systems include JFFS2 and YAFFS.

[edit] Tape file systems

A tape file system is a file system and tape format designed to store files on tape in a self-
describing form. Magnetic tapes are sequential storage media, posing challenges to the creation
and efficient management of a general-purpose file system. IBM has recently announced a new
file system for tape called the Linear Tape File System. The IBM implementation of this file
system has been released as the open-source IBM Long Term File System product.

[edit] Database file systems

A recent concept for file management is the idea of a database-based file system. Instead of, or in
addition to, hierarchical structured management, files are identified by their characteristics, like
type of file, topic, author, or similar metadata.

[edit] Transactional file systems

Some programs need to update multiple files "all at once." For example, a software installation
may write program binaries, libraries, and configuration files. If the software installation fails,
the program may be unusable. If the installation is upgrading a key system utility, such as the
command shell, the entire system may be left in an unusable state.

Transaction processing introduces the isolation guarantee, which states that operations within a
transaction are hidden from other threads on the system until the transaction commits, and that
interfering operations on the system will be properly serialized with the transaction. Transactions
also provide the atomicity guarantee, that operations inside of a transaction are either all
committed, or the transaction can be aborted and the system discards all of its partial results. This
means that if there is a crash or power failure, after recovery, the stored state will be consistent.
Either the software will be completely installed or the failed installation will be completely rolled
back, but an unusable partial install will not be left on the system.

Windows, beginning with Vista, added transaction support to NTFS, abbreviated TxF. TxF is the
only commercial implementation of a transactional file system, as transactional file systems are
difficult to implement correctly in practice. There are a number of research prototypes of
transactional file systems for UNIX systems, including the Valor file system[1], Amino[2], LFS [3],
and a transactional ext3 file system on the TxOS kernel[4], as well as transactional file systems
targeting embedded systems, such as TFFS [5].

Ensuring consistency across multiple file system operations is difficult, if not impossible,
without file system transactions. File locking can be used as a concurrency control mechanism
for individual files, but it typically does not protect the directory structure or file metadata. For
instance, file locking cannot prevent TOCTTOU race conditions on symbolic links. File locking
also cannot automatically roll back a failed operation, such as a software upgrade; this requires
atomicity.

Journaling file systems are one technique used to introduce transaction-level consistency to file
system structures. Journal transactions are not exposed to programs as part of the OS API; they
are only used internally to ensure consistency at the granularity of a single system call.

[edit] Network file systems

Main article: Distributed file system

A network file system is a file system that acts as a client for a remote file access protocol,
providing access to files on a server. Examples of network file systems include clients for the
NFS, AFS, SMB protocols, and file-system-like clients for FTP and WebDAV.

[edit] Shared disk file systems

Main article: Shared disk file system

A shared disk file system is one in which a number of machines (usually servers) all have access
to the same external disk subsystem (usually a SAN). The file system arbitrates access to that
subsystem, preventing write collisions. Examples include GFS from Red Hat, GPFS from IBM,
and SFS from DataPlow.

[edit] Special purpose file systems

Main article: Special file system

A special purpose file system is basically any file system that is not a disk file system or network
file system. This includes systems where the files are arranged dynamically by software,
intended for such purposes as communication between computer processes or temporary file
space.
Special purpose file systems are most commonly used by file-centric operating systems such as
Unix. Examples include the procfs (/proc) file system used by some Unix variants, which grants
access to information about processes and other operating system features.

Deep space science exploration craft, like Voyager I and II used digital tape-based special file
systems. Most modern space exploration craft like Cassini-Huygens used Real-time operating
system file systems or RTOS influenced file systems. The Mars Rovers are one such example of
an RTOS file system, important in this case because they are implemented in flash memory.

[edit] File systems and operating systems

Most operating systems provide a file system, as a file system is an integral part of any modern
operating system. Early microcomputer operating systems' only real task was file management
— a fact reflected in their names (see DOS). Some early operating systems had a separate
component for handling file systems which was called a disk operating system. On some
microcomputers, the disk operating system was loaded separately from the rest of the operating
system. On early operating systems, there was usually support for only one, native, unnamed file
system; for example, CP/M supports only its own file system, which might be called "CP/M file
system" if needed, but which didn't bear any official name at all.

Because of this, there needs to be an interface provided by the operating system software
between the user and the file system. This interface can be textual (such as provided by a
command line interface, such as the Unix shell, or OpenVMS DCL) or graphical (such as
provided by a graphical user interface, such as file browsers). If graphical, the metaphor of the
folder, containing documents, other files, and nested folders is often used (see also: directory and
folder).

[edit] Flat file systems

In a flat file system, there are no subdirectories—everything is stored at the same (root) level on
the media, be it a hard disk, floppy disk, etc. While simple, this system rapidly becomes
inefficient as the number of files grows, and makes it difficult for users to organize data into
related groups.

Like many small systems before it, the original Apple Macintosh featured a flat file system,
called Macintosh File System. Its version of Mac OS was unusual in that the file management
software (Macintosh Finder) created the illusion of a partially hierarchical filing system on top of
EMFS. This structure meant that every file on a disk had to have a unique name, even if it
appeared to be in a separate folder. MFS was quickly replaced with Hierarchical File System,
which supported real directories.

A recent addition to the flat file system family is Amazon's S3, a remote storage service, which is
intentionally simplistic to allow users the ability to customize how their data is stored. The only
constructs are buckets (imagine a disk drive of unlimited size) and objects (similar, but not
identical to the standard concept of a file). Advanced file management is allowed by being able
to use nearly any character (including '/') in the object's name, and the ability to select subsets of
the bucket's content based on identical prefixes.
[edit] File systems under Unix-like operating systems

Unix-like operating systems create a virtual file system, which makes all the files on all the
devices appear to exist in a single hierarchy. This means, in those systems, there is one root
directory, and every file existing on the system is located under it somewhere. Unix-like systems
can use a RAM disk or network shared resource as its root directory.

Unix-like systems assign a device name to each device, but this is not how the files on that
device are accessed. Instead, to gain access to files on another device, the operating system must
first be informed where in the directory tree those files should appear. This process is called
mounting a file system. For example, to access the files on a CD-ROM, one must tell the
operating system "Take the file system from this CD-ROM and make it appear under such-and-
such directory". The directory given to the operating system is called the mount point – it might,
for example, be /media. The /media directory exists on many Unix systems (as specified in the
Filesystem Hierarchy Standard) and is intended specifically for use as a mount point for
removable media such as CDs, DVDs, USB drives or floppy disks. It may be empty, or it may
contain subdirectories for mounting individual devices. Generally, only the administrator (i.e.
root user) may authorize the mounting of file systems.

Unix-like operating systems often include software and tools that assist in the mounting process
and provide it new functionality. Some of these strategies have been coined "auto-mounting" as a
reflection of their purpose.

1. In many situations, file systems other than the root need to be available as soon as the
operating system has booted. All Unix-like systems therefore provide a facility for mounting file
systems at boot time. System administrators define these file systems in the configuration file
fstab or vfstab in Solaris Operating Environment, which also indicates options and mount points.
2. In some situations, there is no need to mount certain file systems at boot time, although their
use may be desired thereafter. There are some utilities for Unix-like systems that allow the
mounting of predefined file systems upon demand.
3. Removable media have become very common with microcomputer platforms. They allow
programs and data to be transferred between machines without a physical connection.
Common examples include USB flash drives, CD-ROMs, and DVDs. Utilities have therefore been
developed to detect the presence and availability of a medium and then mount that medium
without any user intervention.
4. Progressive Unix-like systems have also introduced a concept called supermounting; see, for
example, the Linux supermount-ng project. For example, a floppy disk that has been
supermounted can be physically removed from the system. Under normal circumstances, the
disk should have been synchronized and then unmounted before its removal. Provided
synchronization has occurred, a different disk can be inserted into the drive. The system
automatically notices that the disk has changed and updates the mount point contents to reflect
the new medium. Similar functionality is found on Windows machines.
5. A similar innovation preferred by some users is the use of autofs, a system that, like
supermounting, eliminates the need for manual mounting commands. The difference from
supermount, other than compatibility in an apparent greater range of applications such as
access to file systems on network servers, is that devices are mounted transparently when
requests to their file systems are made, as would be appropriate for file systems on network
 servers, rather than relying on events such as the insertion of media, as would be appropriate
for removable media.

[edit] File systems under Linux

Linux supports many different file systems, but common choices for the system disk include the
ext* family (such as ext2, ext3 and ext4), XFS, JFS, ReiserFS and btrfs.

[edit] File systems under Solaris

The Sun Microsystems Solaris operating system in earlier releases defaulted to (non-journaled or
non-logging) UFS for bootable and supplementary file systems. Solaris defaulted to, supported,
and extended UFS.

Support for other file systems and significant enhancements were added over time, including
Veritas Software Corp. (Journaling) VxFS, Sun Microsystems (Clustering) QFS, Sun
Microsystems (Journaling) UFS, and Sun Microsystems (open source, poolable, 128 bit
compressible, and error-correcting) ZFS.

Kernel extensions were added to Solaris to allow for bootable Veritas VxFS operation. Logging
or Journaling was added to UFS in Sun's Solaris 7. Releases of Solaris 10, Solaris Express,
OpenSolaris, and other open source variants of the Solaris operating system later supported
bootable ZFS.

Logical Volume Management allows for spanning a file system across multiple devices for the
purpose of adding redundancy, capacity, and/or throughput. Legacy environments in Solaris may
use Solaris Volume Manager (formerly known as Solstice DiskSuite.) Multiple operating
systems (including Solaris) may use Veritas Volume Manager. Modern Solaris based operating
systems eclipse the need for Volume Management through leveraging virtual storage pools in
ZFS.

Purpose of storage
Many different forms of storage, based on various natural phenomena, have been invented. So
far, no practical universal storage medium exists, and all forms of storage have some drawbacks.
Therefore a computer system usually contains several kinds of storage, each with an individual
purpose.

A digital computer represents data using the binary numeral system. Text, numbers, pictures,
audio, and nearly any other form of information can be converted into a string of bits, or binary
digits, each of which has a value of 1 or 0. The most common unit of storage is the byte, equal to
8 bits. A piece of information can be handled by any computer whose storage space is large
enough to accommodate the binary representation of the piece of information, or simply data.
For example, using eight million bits, or about one megabyte, a typical computer could store a
short novel.
Traditionally the most important part of every computer is the central processing unit (CPU, or
simply a processor), because it actually operates on data, performs any calculations, and controls
all the other components.

Without a significant amount of memory, a computer would merely be able to perform fixed
operations and immediately output the result. It would have to be reconfigured to change its
behavior. This is acceptable for devices such as desk calculators or simple digital signal
processors. Von Neumann machines differ in that they have a memory in which they store their
operating instructions and data. Such computers are more versatile in that they do not need to
have their hardware reconfigured for each new program, but can simply be reprogrammed with
new in-memory instructions; they also tend to be simpler to design, in that a relatively simple
processor may keep state between successive computations to build up complex procedural
results. Most modern computers are von Neumann machines.

In practice, almost all computers use a variety of memory types, organized in a storage hierarchy
around the CPU, as a trade-off between performance and cost. Generally, the lower a storage is
in the hierarchy, the lesser its bandwidth and the greater its access latency is from the CPU. This
traditional division of storage to primary, secondary, tertiary and off-line storage is also guided
by cost per bit.
[edit] Hierarchy of storage

Various forms of storage, divided according to their distance from the central processing unit. The
fundamental components of a general-purpose computer are arithmetic and logic unit, control circuitry,
storage space, and input/output devices. Technology and capacity as in common home computers
around 2005.

[edit] Primary storage

Direct links to this section: Primary storage, Main memory, Internal Memory.

Primary storage (or main memory or internal memory), often referred to simply as memory,
is the only one directly accessible to the CPU. The CPU continuously reads instructions stored
there and executes them as required. Any data actively operated on is also stored there in
uniform manner.

Historically, early computers used delay lines, Williams tubes, or rotating magnetic drums as
primary storage. By 1954, those unreliable methods were mostly replaced by magnetic core
memory. Core memory remained dominant until the 1970s, when advances in integrated circuit
technology allowed semiconductor memory to become economically competitive.

This led to modern random-access memory (RAM). It is small-sized, light, but quite expensive at
the same time. (The particular types of RAM used for primary storage are also volatile, i.e. they
lose the information when not powered).

As shown in the diagram, traditionally there are two more sub-layers of the primary storage,
besides main large-capacity RAM:

 Processor registers are located inside the processor. Each register typically holds a word of data
(often 32 or 64 bits). CPU instructions instruct the arithmetic and logic unit to perform various
calculations or other operations on this data (or with the help of it). Registers are the fastest of
all forms of computer data storage.
 Processor cache is an intermediate stage between ultra-fast registers and much slower main
memory. It's introduced solely to increase performance of the computer. Most actively used
information in the main memory is just duplicated in the cache memory, which is faster, but of
much lesser capacity. On the other hand it is much slower, but much larger than processor
registers. Multi-level hierarchical cache setup is also commonly used—primary cache being
smallest, fastest and located inside the processor; secondary cache being somewhat larger and
slower.

Main memory is directly or indirectly connected to the central processing unit via a memory bus.
It is actually two buses (not on the diagram): an address bus and a data bus. The CPU firstly
sends a number through an address bus, a number called memory address, that indicates the
desired location of data. Then it reads or writes the data itself using the data bus. Additionally, a
memory management unit (MMU) is a small device between CPU and RAM recalculating the
actual memory address, for example to provide an abstraction of virtual memory or other tasks.

As the RAM types used for primary storage are volatile (cleared at start up), a computer
containing only such storage would not have a source to read instructions from, in order to start
the computer. Hence, non-volatile primary storage containing a small startup program (BIOS) is
used to bootstrap the computer, that is, to read a larger program from non-volatile secondary
storage to RAM and start to execute it. A non-volatile technology used for this purpose is called
ROM, for read-only memory (the terminology may be somewhat confusing as most ROM types
are also capable of random access).

Many types of "ROM" are not literally read only, as updates are possible; however it is slow and
memory must be erased in large portions before it can be re-written. Some embedded systems
run programs directly from ROM (or similar), because such programs are rarely changed.
Standard computers do not store non-rudimentary programs in ROM, rather use large capacities
of secondary storage, which is non-volatile as well, and not as costly.

Recently, primary storage and secondary storage in some uses refer to what was historically
called, respectively, secondary storage and tertiary storage.[2]
[edit] Secondary storage

A hard disk drive with protective cover removed.

Secondary storage (also known as external memory or auxiliary storage), differs from primary
storage in that it is not directly accessible by the CPU. The computer usually uses its input/output
channels to access secondary storage and transfers the desired data using intermediate area in
primary storage. Secondary storage does not lose the data when the device is powered down—it
is non-volatile. Per unit, it is typically also two orders of magnitude less expensive than primary
storage. Consequently, modern computer systems typically have two orders of magnitude more
secondary storage than primary storage and data is kept for a longer time there.

In modern computers, hard disk drives are usually used as secondary storage. The time taken to
access a given byte of information stored on a hard disk is typically a few thousandths of a
second, or milliseconds. By contrast, the time taken to access a given byte of information stored
in random access memory is measured in billionths of a second, or nanoseconds. This illustrates
the significant access-time difference which distinguishes solid-state memory from rotating
magnetic storage devices: hard disks are typically about a million times slower than memory.
Rotating optical storage devices, such as CD and DVD drives, have even longer access times.
With disk drives, once the disk read/write head reaches the proper placement and the data of
interest rotates under it, subsequent data on the track are very fast to access. As a result, in order
to hide the initial seek time and rotational latency, data are transferred to and from disks in large
contiguous blocks.

When data reside on disk, block access to hide latency offers a ray of hope in designing efficient
external memory algorithms. Sequential or block access on disks is orders of magnitude faster
than random access, and many sophisticated paradigms have been developed to design efficient
algorithms based upon sequential and block access . Another way to reduce the I/O bottleneck is
to use multiple disks in parallel in order to increase the bandwidth between primary and
secondary memory.[3]

Some other examples of secondary storage technologies are: flash memory (e.g. USB flash
drives or keys), floppy disks, magnetic tape, paper tape, punched cards, standalone RAM disks,
and Iomega Zip drives.

The secondary storage is often formatted according to a file system format, which provides the
abstraction necessary to organize data into files and directories, providing also additional
information (called metadata) describing the owner of a certain file, the access time, the access
permissions, and other information.

Most computer operating systems use the concept of virtual memory, allowing utilization of
more primary storage capacity than is physically available in the system. As the primary memory
fills up, the system moves the least-used chunks (pages) to secondary storage devices (to a swap
file or page file), retrieving them later when they are needed. As more of these retrievals from
slower secondary storage are necessary, the more the overall system performance is degraded.

[edit] Tertiary storage

Large tape library. Tape cartridges placed on shelves in the front, robotic arm moving in the back. Visible
height of the library is about 180 cm.

Tertiary storage or tertiary memory,[4] provides a third level of storage. Typically it involves a
robotic mechanism which will mount (insert) and dismount removable mass storage media into a
storage device according to the system's demands; this data is often copied to secondary storage
before use. It is primarily used for archival of rarely accessed information since it is much slower
than secondary storage (e.g. 5–60 seconds vs. 1-10 milliseconds). This is primarily useful for
extraordinarily large data stores, accessed without human operators. Typical examples include
tape libraries and optical jukeboxes.

When a computer needs to read information from the tertiary storage, it will first consult a
catalog database to determine which tape or disc contains the information. Next, the computer
will instruct a robotic arm to fetch the medium and place it in a drive. When the computer has
finished reading the information, the robotic arm will return the medium to its place in the
library.
[edit] Off-line storage

Off-line storage is a computer data storage on a medium or a device that is not under the control
of a processing unit.[5] The medium is recorded, usually in a secondary or tertiary storage device,
and then physically removed or disconnected. It must be inserted or connected by a human
operator before a computer can access it again. Unlike tertiary storage, it cannot be accessed
without human interaction.

Off-line storage is used to transfer information, since the detached medium can be easily
physically transported. Additionally, in case a disaster, for example a fire, destroys the original
data, a medium in a remote location will probably be unaffected, enabling disaster recovery. Off-
line storage increases general information security, since it is physically inaccessible from a
computer, and data confidentiality or integrity cannot be affected by computer-based attack
techniques. Also, if the information stored for archival purposes is accessed seldom or never, off-
line storage is less expensive than tertiary storage.

In modern personal computers, most secondary and tertiary storage media are also used for off-
line storage. Optical discs and flash memory devices are most popular, and to much lesser extent
removable hard disk drives. In enterprise uses, magnetic tape is predominant. Older examples are
floppy disks, Zip disks, or punched cards.

[edit] Characteristics of storage

A 1GB DDR RAM memory module (detail)

Storage technologies at all levels of the storage hierarchy can be differentiated by evaluating
certain core characteristics as well as measuring characteristics specific to a particular
implementation. These core characteristics are volatility, mutability, accessibility, and
addressibility. For any particular implementation of any storage technology, the characteristics
worth measuring are capacity and performance.

[edit] Volatility
Non-volatile memory 

Will retain the stored information even if it is not constantly supplied with electric power. It is
suitable for long-term storage of information. Nowadays used for most of secondary, tertiary,
and off-line storage. In 1950s and 1960s, it was also used for primary storage, in the form of
magnetic core memory.

Volatile memory 

Requires constant power to maintain the stored information. The fastest memory technologies
of today are volatile ones (not a universal rule). Since primary storage is required to be very fast,
it predominantly uses volatile memory.

[edit] Differentiation
Dynamic random access memory 

A form of volatile memory which also requires the stored information to be periodically re-read
and re-written, or refreshed, otherwise it would vanish.

Static memory 

A form of volatile memory similar to DRAM with the exception that it never needs to be
refreshed as long as power is applied. (It loses its content if power is removed).

[edit] Mutability
Read/write storage or mutable storage 

Allows information to be overwritten at any time. A computer without some amount of

read/write storage for primary storage purposes would be useless for many tasks. Modern
computers typically use read/write storage also for secondary storage.

Read only storage 

Retains the information stored at the time of manufacture, and write once storage (Write Once
Read Many) allows the information to be written only once at some point after manufacture.
These are called immutable storage. Immutable storage is used for tertiary and off-line storage.
Examples include CD-ROM and CD-R.

Slow write, fast read storage 

Read/write storage which allows information to be overwritten multiple times, but with the
write operation being much slower than the read operation. Examples include CD-RW and flash
memory.
[edit] Accessibility
Random access 

Any location in storage can be accessed at any moment in approximately the same amount of
time. Such characteristic is well suited for primary and secondary storage.

Sequential access 

The accessing of pieces of information will be in a serial order, one after the other; therefore the
time to access a particular piece of information depends upon which piece of information was
last accessed. Such characteristic is typical of off-line storage.

[edit] Addressability
Location-addressable 

Each individually accessible unit of information in storage is selected with its numerical memory
address. In modern computers, location-addressable storage usually limits to primary storage,
accessed internally by computer programs, since location-addressability is very efficient, but
burdensome for humans.

File addressable 

Information is divided into files of variable length, and a particular file is selected with human-
readable directory and file names. The underlying device is still location-addressable, but the
operating system of a computer provides the file system abstraction to make the operation
more understandable. In modern computers, secondary, tertiary and off-line storage use file
systems.

Content-addressable 

Each individually accessible unit of information is selected based on the basis of (part of) the
contents stored there. Content-addressable storage can be implemented using software
(computer program) or hardware (computer device), with hardware being faster but more
expensive option. Hardware content addressable memory is often used in a computer's CPU
cache.

[edit] Capacity
Raw capacity 

The total amount of stored information that a storage device or medium can hold. It is
expressed as a quantity of bits or bytes (e.g. 10.4 megabytes).

Memory storage density 

The compactness of stored information. It is the storage capacity of a medium divided with a
unit of length, area or volume (e.g. 1.2 megabytes per square inch).
[edit] Performance
Latency 

The time it takes to access a particular location in storage. The relevant unit of measurement is
typically nanosecond for primary storage, millisecond for secondary storage, and second for
tertiary storage. It may make sense to separate read latency and write latency, and in case of
sequential access storage, minimum, maximum and average latency.

Throughput 

The rate at which information can be read from or written to the storage. In computer data
storage, throughput is usually expressed in terms of megabytes per second or MB/s, though bit
rate may also be used. As with latency, read rate and write rate may need to be differentiated.
Also accessing media sequentially, as opposed to randomly, typically yields maximum
throughput.

[edit] Energy use

 Storage devices that reduce fan usage, automatically shut-down during inactivity, and low
power hard drives can reduce energy consumption 90 percent. [6]
 2.5 inch hard disk drives often consume less power than larger ones. [7][8] Low capacity solid-state
drives have no moving parts and consume less power than hard disks. [9][10][11] Also, memory may
use more power than hard disks.[11]

[edit] Fundamental storage technologies

As of 2008, the most commonly used data storage technologies are semiconductor, magnetic,
and optical, while paper still sees some limited usage. Some other fundamental storage
technologies have also been used in the past or are proposed for development.

[edit] Semiconductor

Semiconductor memory uses semiconductor-based integrated circuits to store information. A

semiconductor memory chip may contain millions of tiny transistors or capacitors. Both volatile
and non-volatile forms of semiconductor memory exist. In modern computers, primary storage
almost exclusively consists of dynamic volatile semiconductor memory or dynamic random
access memory. Since the turn of the century, a type of non-volatile semiconductor memory
known as flash memory has steadily gained share as off-line storage for home computers. Non-
volatile semiconductor memory is also used for secondary storage in various advanced electronic
devices and specialized computers.

[edit] Magnetic
show]

Magnetic storage media

v • d • e
Magnetic storage uses different patterns of magnetization on a magnetically coated surface to
store information. Magnetic storage is non-volatile. The information is accessed using one or
more read/write heads which may contain one or more recording transducers. A read/write head
only covers a part of the surface so that the head or medium or both must be moved relative to
another in order to access data. In modern computers, magnetic storage will take these forms:

 Magnetic disk
o Floppy disk, used for off-line storage
o Hard disk drive, used for secondary storage
 Magnetic tape data storage, used for tertiary and off-line storage

In early computers, magnetic storage was also used for primary storage in a form of magnetic
drum, or core memory, core rope memory, thin-film memory, twistor memory or bubble
memory. Also unlike today, magnetic tape was often used for secondary storage.

[edit] Optical
[show]

v • d • e

Optical storage media

Optical storage, the typical optical disc, stores information in deformities on the surface of a
circular disc and reads this information by illuminating the surface with a laser diode and
observing the reflection. Optical disc storage is non-volatile. The deformities may be permanent
(read only media ), formed once (write once media) or reversible (recordable or read/write
media). The following forms are currently in common use:[12]

 CD, CD-ROM, DVD, BD-ROM: Read only storage, used for mass distribution of digital information
(music, video, computer programs)
 CD-R, DVD-R, DVD+R, BD-R: Write once storage, used for tertiary and off-line storage
 CD-RW, DVD-RW, DVD+RW, DVD-RAM, BD-RE: Slow write, fast read storage, used for tertiary
and off-line storage
 Ultra Density Optical or UDO is similar in capacity to BD-R or BD-RE and is slow write, fast read
storage used for tertiary and off-line storage.

Magneto-optical disc storage is optical disc storage where the magnetic state on a
ferromagnetic surface stores information. The information is read optically and written by
combining magnetic and optical methods. Magneto-optical disc storage is non-volatile,
sequential access, slow write, fast read storage used for tertiary and off-line storage.

3D optical data storage has also been proposed.

WHat is the difference between Veritas filesystem and veritas volume manager?
I had trouble with this in the beginning also.

Volume Manager has to do with managing the disks, raid, striping, plexes, etc....

Veritas Filesystem is just that. Anything that you need to do to manage the data on the
disks. Such as mkfs, mount, online resizing and defragmenting. It also gives you control
over I/O, caching, and allocation.

Remember:
VM is managing the disks.

Veritas Filesystem is managing the data on the disks.

The Veritas filesystem is the filesystem used for specific partitions. It is controlled by the
Veritas Volume Manager which is a gui software interface to create the different volumes
being used (by Oracle, Sybase, and other applications). Normally UNIX OS does not need to
be placed under Veritas but could (although failures in drives then creates more of a
headache for the OS recovery.)

Logical Volume Managers

Logical volume management is a relatively new approach to UNIX disks and filesystems. Instead
of dealing directly with physical disk partitions, the logical volume manager (abbreviated LVM)
splits disk space up into logical partitions. The difference between an ordinary disk partition and
a logical partition is analogous to the difference between a physical filesystem and the UNIX
logical filesystem. A logical partition may span multiple physical disks, but is accessed
transparently as if it were a single disk.

There are numerous advantages to using a logical volume manager:

 They offer greater flexibility for disk partitioning.

 The size of logical volumes can be modified according to need, while the operating system is
running
 Logical volumes can span multiple disks.
 Disk mirroring is often supported, for greater data reliability

The simplest way to understand logical volume management is to look at the structure of the
system from the bottom up:

Disks, Physical Volumes, and Volume Groups

The lowest level is that of the physical disk itself. A physical disk is formatted into a physical volume for
use by the LVM. Each physical volume is split up into discrete chunks, called physical partitions or
physical extents. Note that this is not the same thing as an ordinary disk partition; physical
partitions/extents are allocatable units of space, on the order of 4MB.

Physical volumes are combined into a volume group. A volume group is thus a collection of
disks, treated as one large storage area. The physical volume is analogous to the physical disk
under the ordinary UNIX partitioning scheme -- it is a single storage area that can be split up into
several independent filesystems.

Logical Volumes
Logical volumes are the LVM's equivalent to the ordinary UNIX disk partition. A logical volume is made
into a filesystem, or may be used as a swap device, a boot device, and so on. A logical volume consists of
some number of physical partitions/extents, allocated from a single volume group. The allocation of
physical partitions within a volume group is generally arbitrary, though some LVM's may allow a logical
volume to be allocated from a specific physical volume. Logical volumes may be any size that is a
multiple of the size of a physical partition within a given volume group.

VI Editor Commands
Vi has two modes insertion mode and command mode.
The editor begins in command mode, where cursor movement and copy/paste editing occur.
Most commands execute as soon as typed except for "colon" commands which execute when
you press the return key.

Switch to Text or Insert mode:

Open line
    O        
above cursor

Insert text at Insert text at Insert text Append text at

I i a A
beginning of line cursor after cursor line end

Open line
    o        
below cursor

Switch to Command mode:

Switch to <ESC
command mode >

Cursor Movement (command mode):

Scroll Backward 1
    <ctrl>b    
screen

    Scroll Up 1/2 <ctrl>u    

 screen

Go to beginning Go to end of
0 Go to line n nG $
of line line

Scroll Down 1/2 Go to line

    <ctrl>d :##
screen number ##

Scroll Forward 1
    <ctrl>f    
screen

    Go to last line G    

Scroll by
()        
sentence f/b

w Move left, down,

Scroll by word f/b hjkl Left 6 chars 6h
b up, right

Scroll by Directional Arrow

{} Go to line #6 6G
paragraph f/b Movement Keys

Deleting text (command mode):

Change
cw Replace one character r    
word

Delete Delete entire line (to

dw Delete text at cursor x dd
word buffer)

Delete (backspace) text at Delete 5 lines (to

    X 5dd
cursor buffer)

Delete current to end of

    D Delete lines 5-10 :5,10d
line

Editing (command mode):

Copy
Copy n lines 1-
Copy line yy nyy :1,2t 3
lines 2/paste
after 3
 Paste
above
P      
current
line

Paste Move
below lines 4- :4,5m
p    
current 5/paste 6
line after 6

Join
        previous J
line

Find next
Search Search
string
backward ?string forward /string n
occurrenc
for string for string
e

% (entire
file) s
(search
and
Ignore
replace) /
: case
old text     :set ic
%s/oldstring/newstring/cg during
with new/
search
c
(confirm)
g (global
- all)

Repeat Undo Undo all

last . previous u changes U
command command to line

Save and Quit (command mode):

Save changes to Save changes and Save file to

:w :wq :w file
buffer quit vi new file

Save lines to :10,15w

    Quit without saving :q!
new file file
:syntax on Turn on syntax highlighting
:syntax off Turn off syntax highlighting
:set number Turn on Line numbering (shorthand :set nu)
:set nonumber Turn off Line numbering (shorthand :set nonu)

:set ignorecase Ignore case sensitivity when searching

:set noignorecase Restore case sensitivity (default)

:set autoindent Turn on Auto-indentation

Use the command >> to indent and the << command to outdent
:set shiftwidth=4 Set indentation to four spaces
:set noautoindent Turn off Auto-indentation

What is vi?
The default editor that comes with the UNIX operating system is called vi (visual editor).
[Alternate editors for UNIX environments include pico and emacs, a product of GNU.]

The UNIX vi editor is a full screen editor and has two modes of operation:

1. Command mode commands which cause action to be taken on the file, and
2. Insert mode in which entered text is inserted into the file.

In the command mode, every character typed is a command that does something to the text file
being edited; a character typed in the command mode may even cause the vi editor to enter
the insert mode. In the insert mode, every character typed is added to the text in the file;
pressing the <Esc> (Escape) key turns off the Insert mode.
While there are a number of vi commands, just a handful of these is usually sufficient for
beginning vi users. To assist such users, this Web page contains a sampling of basic vi
commands. The most basic and useful commands are marked with an asterisk ( * or star) in the
tables below. With practice, these commands should become automatic.

NOTE: Both UNIX and vi are case-sensitive. Be sure not to use a capital letter in place of a
lowercase letter; the results will not be what you expect.

To Get Into and Out Of vi

To Start vi
To use vi on a file, type in vi filename. If the file named filename exists, then the first page
(or screen) of the file will be displayed; if the file does not exist, then an empty file and screen
are created into which you may enter text.

* vi filename edit filename starting at line 1

   vi -r filename recover filename that was being edited when system crashed

To Exit vi
Usually the new or modified file is saved when you leave vi. However, it is also possible to quit
vi without saving the file.

Note: The cursor moves to bottom of screen whenever a colon ( :) is typed. This type of
command is completed by hitting the <Return> (or <Enter>) key.

* :x<Return> quit vi, writing out modified file to file named in original invocation

  :wq<Return> quit vi, writing out modified file to file named in original invocation

  :q<Return> quit (or exit) vi

* :q!<Return> quit vi even though latest changes have not been saved for this vi call

Moving the Cursor

Unlike many of the PC and MacIntosh editors, the mouse does not move the cursor within the
vi editor screen (or window). You must use the the key commands listed below. On some UNIX
platforms, the arrow keys may be used as well; however, since vi was designed with the Qwerty
keyboard (containing no arrow keys) in mind, the arrow keys sometimes produce strange effects
in vi and should be avoided.

If you go back and forth between a PC environment and a UNIX environment, you may find that
this dissimilarity in methods for cursor movement is the most frustrating difference between the
two.

In the table below, the symbol ^ before a letter means that the <Ctrl> key should be held
down while the letter key is pressed.

j or <Return>
* move cursor down one line
  [or down-arrow]

* k [or up-arrow] move cursor up one line

h or <Backspace>
* move cursor left one character
  [or left-arrow]

l or <Space>
* move cursor right one character
  [or right-arrow]

* 0 (zero) move cursor to start of current line (the one with the cursor)
 *$ move cursor to end of current line

  w move cursor to beginning of next word

  b move cursor back to beginning of preceding word

  :0<Return> or 1G move cursor to first line in file

  :n<Return> or nG move cursor to line n

  :$<Return> or G move cursor to last line in file

Screen Manipulation
The following commands allow the vi editor screen (or window) to move up or down several
lines and to be refreshed.

  ^f move forward one screen

  ^b move backward one screen

  ^d move down (forward) one half screen

  ^u move up (back) one half screen

  ^l redraws the screen

  ^r redraws the screen, removing deleted lines

Adding, Changing, and Deleting Text

Unlike PC editors, you cannot replace or delete text by highlighting it with the mouse. Instead
use the commands in the following tables.

Perhaps the most important command is the one that allows you to back up and undo your last
action. Unfortunately, this command acts like a toggle, undoing and redoing your most recent
action. You cannot go back more than one step.

* u UNDO WHATEVER YOU JUST DID; a simple toggle

The main purpose of an editor is to create, add, or modify text for a file.
Inserting or Adding Text
The following commands allow you to insert and add text. Each of these commands puts the vi
editor into insert mode; thus, the <Esc> key must be pressed to terminate the entry of text and
to put the vi editor back into command mode.

* i insert text before cursor, until <Esc> hit

  I insert text at beginning of current line, until <Esc> hit

* a append text after cursor, until <Esc> hit

  A append text to end of current line, until <Esc> hit

* o open and put text in a new line below current line, until <Esc> hit

* O open and put text in a new line above current line, until <Esc> hit

Changing Text
The following commands allow you to modify text.

*r replace single character under cursor (no <Esc> needed)

  R replace characters, starting with current cursor position, until <Esc> hit

change the current word with new text,

  cw
starting with the character under cursor, until <Esc> hit

change N words beginning with character under cursor, until <Esc> hit;
  cNw
  e.g., c5w changes 5 words

  C change (replace) the characters in the current line, until <Esc> hit

  cc change (replace) the entire current line, stopping when <Esc> is hit

change (replace) the next N lines, starting with the current line,
  Ncc or cNc
stopping when <Esc> is hit

Deleting Text
The following commands allow you to delete text.

*x delete single character under cursor

  Nx delete N characters, starting with character under cursor

  dw delete the single word beginning with character under cursor

 delete N words beginning with character under cursor;
  dNw
  e.g., d5w deletes 5 words

  D delete the remainder of the line, starting with current cursor position

* dd delete entire current line

delete N lines, beginning with the current line;

  Ndd or dNd
  e.g., 5dd deletes 5 lines

Cutting and Pasting Text

The following commands allow you to copy and paste text.

  yy copy (yank, cut) the current line into the buffer

  Nyy or yNy copy (yank, cut) the next N lines, including the current line, into the buffer

 p put (paste) the line(s) in the buffer into the text after the current line

Other Commands

Searching Text
A common occurrence in text editing is to replace one word or phase by another. To locate
instances of particular sets of characters (or strings), use the following commands.

  /string search forward for occurrence of string in text

  ?string search backward for occurrence of string in text

 n move to next occurrence of search string

 N move to next occurrence of search string in opposite direction

Determining Line Numbers

Being able to determine the line number of the current line or the total number of lines in the
file being edited is sometimes useful.

  :.= returns line number of current line at bottom of screen

  := returns the total number of lines at bottom of screen

 provides the current line number, along with the total number of lines,
  ^g
in the file at the bottom of the screen

Saving and Reading Files

These commands permit you to input and output files other than the named file with which you
are currently working.

read file named filename and insert after current line

  :r filename<Return>
(the line with cursor)

  :w<Return> write current contents to file named in original vi call

  :w newfile<Return> write current contents to a new file named newfile

write the contents of the lines numbered 12 through 35 to a new file

  :12,35w smallfile<Return>
named smallfile

  :w! prevfile<Return> write current contents over a pre-existing file named prevfile

vi command summary
The following tables contain all the basic vi commands.

Starting vi

Command Description
vi file start at line 1 of file
vi +n file start at line n of file
vi + file start at last line of file
vi +/pattern file start at pattern in file
vi -r file recover file after a system crash

Saving files and quitting vi

Command Description
:e file edit file (save current file with :w first)
:w save (write out) the file being edited
:w file save as file
:w! file save as an existing file
:q quit vi
:wq save the file and quit vi
:x save the file if it has changed and quit vi
:q! quit vi without saving changes

Moving the cursor

Keys pressed Effect

h left one character
l or <Space> right one character
k up one line
j or <Enter> down one line
b left one word
w right one word
( start of sentence
) end of sentence
{ start of paragraph
} end of paragraph
1G top of file
nG line n
G end of file
<Ctrl>W first character of insertion
<Ctrl>U up ½ screen
<Ctrl>D down ½ screen
<Ctrl>B up one screen
<Ctrl>F down one screen

Inserting text

Keys pressed Text inserted

a after the cursor
A after last character on the line
i before the cursor
I before first character on the line
o open line below current line
O open line above current line

Changing and replacing text

Keys pressed Text changed or replaced
cw word
3cw three words
cc current line
5cc five lines
r current character only
R current character and those to its right
s current character
S current line
~ switch between lowercase and uppercase

Deleting text

Keys pressed Text deleted

x character under cursor
12x 12 characters
X character to left of cursor
dw word
3dw three words
d0 to beginning of line
d$ to end of line
dd current line
5dd five lines
d{ to beginning of paragraph
d} to end of paragraph
:1,. d to beginning of file
:.,$ d to end of file
:1,$ d whole file

Using markers and buffers

Command Description
mf set marker named ``f''
`f go to marker ``f''
´f go to start of line containing marker ``f''
"s12yy copy 12 lines into buffer ``s''
"ty} copy text from cursor to end of paragraph into buffer ``t''
"ly1G copy text from cursor to top of file into buffer ``l''
"kd`f cut text from cursor up to marker ``f'' into buffer ``k''
"kp paste buffer ``k'' into text

Searching for text

Search Finds
/and next occurrence of ``and'', for example, ``and'', ``stand'', ``grand''
?and previous occurrence of ``and''
/^The next line that starts with ``The'', for example, ``The'', ``Then'', ``There''
/^The\> next line that starts with the word ``The''
/end$ next line that ends with ``end''
/[bB]ox next occurrence of ``box'' or ``Box''
n repeat the most recent search, in the same direction
N repeat the most recent search, in the opposite direction

Searching for and replacing text

Command Description
:s/pear/peach/g replace all occurrences of ``pear'' with ``peach'' on current line
:/orange/s//lemon/g change all occurrences of ``orange'' into ``lemon'' on next line containing
``orange''
:.,$/\<file/directory/g replace all words starting with ``file'' by ``directory'' on every line from
current line onward, for example, ``filename'' becomes ``directoryname''
:g/one/s//1/g replace every occurrence of ``one'' with 1, for example, ``oneself'' becomes
``1self'', ``someone'' becomes ``some1''

Matching patterns of text

Expression Matches
. any single character
zero or more of the previous expression
. zero or more arbitrary characters
\< beginning of a word
\> end of a word
\ quote a special character
\ the character `` ''
^ beginning of a line
$ end of a line
[set] one character from a set of characters
[XYZ] one of the characters ``X'', ``Y'', or ``Z''
[[:upper:]][[:lower:]]* one uppercase character followed by any number of lowercase characters
[^set] one character not from a set of characters
[^XYZ[:digit:]] any character except ``X'', ``Y'', ``Z'', or a numeric digit

Options to the :set command

Option Effect
all list settings of all options
ignorecase ignore case in searches
list display <Tab> and end-of-line characters
mesg display messages sent to your terminal
nowrapscan prevent searches from wrapping round the end or beginning of a file
number display line numbers
report=5 warn if five or more lines are changed by command
term=ansi set terminal type to ``ansi''
terse shorten error messages
warn display ``[No write since last change]'' on shell escape if file has not been saved

Unix (officially trademarked as UNIX, sometimes also written as UNIX with small caps) is a
computer operating system originally developed in 1969 by a group of AT&T employees at Bell
Labs, including Ken Thompson, Dennis Ritchie, Brian Kernighan, Douglas McIlroy, and Joe
Ossanna. Today's Unix systems are split into various branches, developed over time by AT&T as
well as various commercial vendors and non-profit organizations.

The Open Group, an industry standards consortium, owns the “Unix” trademark. Only systems
fully compliant with and certified according to the Single UNIX Specification are qualified to
use the trademark; others might be called "Unix system-like" or "Unix-like" (though the Open
Group disapproves of this term). However, the term "Unix" is often used informally to denote
any operating system that closely resembles the trademarked system.

During the late 1970s and early 1980s, the influence of Unix in academic circles led to large-
scale adoption of Unix (particularly of the BSD variant, originating from the University of
California, Berkeley) by commercial startups, the most notable of which are Solaris, HP-UX and
AIX. Today, in addition to certified Unix systems such as those already mentioned, Unix-like
operating systems such as Linux and BSD descendants (FreeBSD, NetBSD, and OpenBSD) are
commonly encountered. The term "traditional Unix" may be used to describe a Unix or an
operating system that has the characteristics of either Version 7 Unix or UNIX System V.
Overview
Unix operating systems are widely used in servers, workstations, and mobile devices.[1] The Unix
environment and the client–server program model were essential elements in the development of
the Internet and the reshaping of computing as centered in networks rather than in individual
computers.

Both Unix and the C programming language were developed by AT&T and distributed to
government and academic institutions, which led to both being ported to a wider variety of
machine families than any other operating system. As a result, Unix became synonymous with
"open systems".

Unix was designed to be portable, multi-tasking and multi-user in a time-sharing configuration.

Unix systems are characterized by various concepts: the use of plain text for storing data; a
hierarchical file system; treating devices and certain types of inter-process communication (IPC)
as files; and the use of a large number of software tools, small programs that can be strung
together through a command line interpreter using pipes, as opposed to using a single monolithic
program that includes all of the same functionality. These concepts are collectively known as the
Unix philosophy.

Under Unix, the "operating system" consists of many of these utilities along with the master
control program, the kernel. The kernel provides services to start and stop programs, handles the
file system and other common "low level" tasks that most programs share, and, perhaps most
importantly, schedules access to hardware to avoid conflicts if two programs try to access the
same resource or device simultaneously. To mediate such access, the kernel was given special
rights on the system, leading to the division between user-space and kernel-space.

The microkernel concept was introduced in an effort to reverse the trend towards larger kernels
and return to a system in which most tasks were completed by smaller utilities. In an era when a
"normal" computer consisted of a hard disk for storage and a data terminal for input and output
(I/O), the Unix file model worked quite well as most I/O was "linear". However, modern systems
include networking and other new devices. As graphical user interfaces developed, the file model
proved inadequate to the task of handling asynchronous events such as those generated by a
mouse, and in the 1980s non-blocking I/O and the set of inter-process communication
mechanisms was augmented (sockets, shared memory, message queues, semaphores), and
functionalities such as network protocols were moved out of the kernel.

[edit] History
In the 1960s, Massachusetts Institute of Technology, AT&T Bell Labs, and General Electric
developed an experimental operating system called Multics for the GE-645 mainframe.[2] Multics
was highly innovative, but had many problems.

Bell Labs, frustrated by the size and complexity of Multics but not the aims, slowly pulled out of
the project. Their last researchers to leave Multics, Ken Thompson, Dennis Ritchie, M. D.
McIlroy, and J. F. Ossanna,[3] decided to redo the work on a much smaller scale. At the time,
Ritchie says "What we wanted to preserve was not just a good environment in which to do
programming, but a system around which a fellowship could form. We knew from experience
that the essence of communal computing, as supplied by remote-access, time-shared machines, is
not just to type programs into a terminal instead of a keypunch, but to encourage close
communication."[3]

While Ken Thompson still had access to the Multics environment, he wrote simulations for the
new file and paging system on it. He also programmed a game called Space Travel, but the game
needed a more efficient and less expensive machine to run on, and eventually a little-used PDP-7
at Bell Labs fit the bill.[4] On this PDP7, a team of Bell Labs researchers led by Thompson and
Ritchie, including Rudd Canaday, developed a hierarchical file system, the notions of computer
processes and device files, a command-line interpreter, and some small utility programs.[3]

[edit] 1970s

In the 1970s Brian Kernighan coined the project name Unics as a play on Multics, (Multiplexing
Information and Computer Services). Unics could eventually support multiple simultaneous
users, and it was renamed Unix.

Up until this point there had been no financial support from Bell Labs. When the Computer
Science Research Group wanted to use Unix on a much larger machine than the PDP-7,
Thompson and Ritchie managed to trade the promise of adding text processing capabilities to
Unix for a PDP-11/20 machine. This led to some financial support from Bell. For the first time in
1970, the Unix operating system was officially named and ran on the PDP-11/20. It added a text
formatting program called roff and a text editor. All three were written in PDP-11/20 assembly
language. Bell Labs used this initial "text processing system", made up of Unix, roff, and the
editor, for text processing of patent applications. Roff soon evolved into troff, the first electronic
publishing program with a full typesetting capability. The UNIX Programmer's Manual was
published on November 3, 1971.

In 1972, Unix was rewritten in the C programming language, contrary to the general notion at the
time "that something as complex as an operating system, which must deal with time-critical
events, had to be written exclusively in assembly language".[5] The migration from assembly
language to the higher-level language C resulted in much more portable software, requiring only
a relatively small amount of machine-dependent code to be replaced when porting Unix to other
computing platforms.

Under a 1958 consent decree in settlement of an antitrust case, AT&T (the parent organization of
Bell Labs) had been forbidden from entering the computer business. Unix could not, therefore,
be turned into a product; indeed, under the terms of the consent decree, Bell Labs was required to
license its nontelephone technology to anyone who asked. Ken Thompson quietly began
answering requests by shipping out tapes and disk packs – each, according to legend, with a note
signed “love, ken”.[6]

AT&T made Unix available to universities and commercial firms, as well as the United States
government under licenses. The licenses included all source code including the machine-
dependent parts of the kernel, which were written in PDP-11 assembly code. Copies of the
annotated Unix kernel sources circulated widely in the late 1970s in the form of a much-copied
book by John Lions of the University of New South Wales, the Lions' Commentary on UNIX 6th
Edition, with Source Code, which led to considerable use of Unix as an educational example.

Versions of the Unix system were determined by editions of its user manuals. For example,
"Fifth Edition UNIX" and "UNIX Version 5" have both been used to designate the same version.
Development expanded, with Versions 4, 5, and 6 being released by 1975. These versions added
the concept of pipes, which led to the development of a more modular code-base and quicker
development cycles. Version 5 and especially Version 6 led to a plethora of different Unix
versions both inside and outside Bell Labs, including PWB/UNIX and the first commercial Unix,
IS/1. As more of Unix was rewritten in C, portability also increased. A group at the University of
Wollongong ported Unix to the Interdata 7/32. Bell Labs developed several ports for research
purposes and internal use at AT&T. Target machines included an Intel 8086-based computer
(with custom-built MMU) and the UNIVAC 1100.[7]

In May 1975 ARPA documented the benefits of the Unix time-sharing system which "presents
several interesting capabilities" as an arpa network mini-host in RFC 681.

In 1978, UNIX/32V was released for DEC's then new VAX system. By this time, over 600
machines were running Unix in some form. Version 7 Unix, the last version of Research Unix to
be released widely, was released in 1979. Versions 8, 9 and 10 were developed through the
1980s but were only released to a few universities, though they did generate papers describing
the new work. This research led to the development of Plan 9 from Bell Labs, a new portable
distributed system.

[edit] 1980s

A Unix desktop running the X Window System graphical user interface. Shown are a number of client
applications common to the MIT X Consortium's distribution, including Tom's Window Manager, an X
Terminal, Xbiff, xload, and a graphical manual page browser.

AT&T licensed UNIX System III, based largely on Version 7, for commercial use, the first
version launching in 1982. This also included support for the VAX. AT&T continued to issue
licenses for older Unix versions. To end the confusion between all its differing internal versions,
AT&T combined them into UNIX System V Release 1. This introduced a few features such as
the vi editor and curses from the Berkeley Software Distribution of Unix developed at the
University of California, Berkeley. This also included support for the Western Electric 3B series
of machines.

In 1983, the U.S. Department of Justice settled its second antitrust case against AT&T and broke
up the Bell System. This relieved AT&T from the 1956 consent decree that had prevented them
from turning Unix into a product. AT&T promptly rushed to commercialize Unix System V, a
move that nearly killed Unix.[6]

Since the newer commercial UNIX licensing terms were not as favorable for academic use as the
older versions of Unix, the Berkeley researchers continued to develop BSD Unix as an
alternative to UNIX System III and V, originally on the PDP-11 architecture (the 2.xBSD
releases, ending with 2.11BSD) and later for the VAX-11 (the 4.x BSD releases). Many
contributions to Unix first appeared on BSD releases, notably the C shell with job control
(modelled on ITS). Perhaps the most important aspect of the BSD development effort was the
addition of TCP/IP network code to the mainstream Unix kernel. The BSD effort produced
several significant releases that contained network code: 4.1cBSD, 4.2BSD, 4.3BSD, 4.3BSD-
Tahoe ("Tahoe" being the nickname of the Computer Consoles Inc. Power 6/32 architecture that
was the first non-DEC release of the BSD kernel), Net/1, 4.3BSD-Reno (to match the "Tahoe"
naming, and that the release was something of a gamble), Net/2, 4.4BSD, and 4.4BSD-lite. The
network code found in these releases is the ancestor of much TCP/IP network code in use today,
including code that was later released in AT&T System V UNIX and early versions of Microsoft
Windows. The accompanying Berkeley sockets API is a de facto standard for networking APIs
and has been copied on many platforms.

Other companies began to offer commercial versions of the UNIX System for their own mini-
computers and workstations. Most of these new Unix flavors were developed from the System V
base under a license from AT&T; however, others were based on BSD instead. One of the
leading developers of BSD, Bill Joy, went on to co-found Sun Microsystems in 1982 and created
SunOS for their workstation computers. In 1980, Microsoft announced its first Unix for 16-bit
microcomputers called Xenix, which the Santa Cruz Operation (SCO) ported to the Intel 8086
processor in 1983, and eventually branched Xenix into SCO UNIX in 1989.

For a few years during this period (before PC compatible computers with MS-DOS became
dominant), industry observers expected that UNIX, with its portability and rich capabilities, was
likely to become the industry standard operating system for microcomputers.[8] In 1984 several
companies established the X/Open consortium with the goal of creating an open system
specification based on UNIX. Despite early progress, the standardization effort collapsed into the
"Unix wars", with various companies forming rival standardization groups. The most successful
Unix-related standard turned out to be the IEEE's POSIX specification, designed as a
compromise API readily implemented on both BSD and System V platforms, published in 1988
and soon mandated by the United States government for many of its own systems.

AT&T added various features into UNIX System V, such as file locking, system administration,
STREAMS, new forms of IPC, the Remote File System and TLI. AT&T cooperated with Sun
Microsystems and between 1987 and 1989 merged features from Xenix, BSD, SunOS, and
System V into System V Release 4 (SVR4), independently of X/Open. This new release
consolidated all the previous features into one package, and heralded the end of competing
versions. It also increased licensing fees.

During this time a number of vendors including Digital Equipment, Sun, Addamax and others
began building trusted versions of UNIX for high security applications, mostly designed for
military and law enforcement applications.

[edit] 1990s

In 1990, the Open Software Foundation released OSF/1, their standard Unix implementation,
based on Mach and BSD. The Foundation was started in 1988 and was funded by several Unix-
related companies that wished to counteract the collaboration of AT&T and Sun on SVR4.
Subsequently, AT&T and another group of licensees formed the group "UNIX International" in
order to counteract OSF. This escalation of conflict between competing vendors gave rise again
to the phrase "Unix wars".

In 1991, a group of BSD developers (Donn Seeley, Mike Karels, Bill Jolitz, and Trent Hein) left
the University of California to found Berkeley Software Design, Inc (BSDI). BSDI produced a
fully functional commercial version of BSD Unix for the inexpensive and ubiquitous Intel
platform, which started a wave of interest in the use of inexpensive hardware for production
computing. Shortly after it was founded, Bill Jolitz left BSDI to pursue distribution of 386BSD,
the free software ancestor of FreeBSD, OpenBSD, and NetBSD.

By 1993 most commercial vendors had changed their variants of Unix to be based on System V
with many BSD features added on top. The creation of the COSE initiative that year by the major
players in Unix marked the end of the most notorious phase of the Unix wars, and was followed
by the merger of UI and OSF in 1994. The new combined entity, which retained the OSF name,
stopped work on OSF/1 that year. By that time the only vendor using it was Digital, which
continued its own development, rebranding their product Digital UNIX in early 1995.

Shortly after UNIX System V Release 4 was produced, AT&T sold all its rights to UNIX to
Novell. (Dennis Ritchie likened this to the Biblical story of Esau selling his birthright for the
proverbial "mess of pottage".[9]) Novell developed its own version, UnixWare, merging its
NetWare with UNIX System V Release 4. Novell tried to use this to battle against Windows NT,
but their core markets suffered considerably.

In 1993, Novell decided to transfer the UNIX trademark and certification rights to the X/Open
Consortium.[10] In 1996, X/Open merged with OSF, creating the Open Group. Various standards
by the Open Group now define what is and what is not a "UNIX" operating system, notably the
post-1998 Single UNIX Specification.
In 1995, the business of administering and supporting the existing UNIX licenses, plus rights to
further develop the System V code base, were sold by Novell to the Santa Cruz Operation.[11]
Whether Novell also sold the copyrights is currently the subject of litigation (see below).

In 1997, Apple Computer sought out a new foundation for its Macintosh operating system and
chose NEXTSTEP, an operating system developed by NeXT. The core operating system, which
was based on BSD and the Mach kernel, was renamed Darwin after Apple acquired it. The
deployment of Darwin in Mac OS X makes it, according to a statement made by an Apple
employee at a USENIX conference, the most widely used Unix-based system in the desktop
computer market.

[edit] 2000s

In 2000, SCO sold its entire UNIX business and assets to Caldera Systems, which later on
changed its name to The SCO Group.

The dot-com bubble (2001–2003) led to significant consolidation of versions of Unix. Of the
many commercial variants of Unix that were born in the 1980s, only Solaris, HP-UX, and AIX
were still doing relatively well in the market, though SGI's IRIX persisted for quite some time.
Of these, Solaris had the largest market share in 2005.[12]

In 2003, the SCO Group started legal action against various users and vendors of Linux. SCO
had alleged that Linux contained copyrighted Unix code now owned by The SCO Group. Other
allegations included trade-secret violations by IBM, or contract violations by former Santa Cruz
customers who had since converted to Linux. However, Novell disputed the SCO Group's claim
to hold copyright on the UNIX source base. According to Novell, SCO (and hence the SCO
Group) are effectively franchise operators for Novell, which also retained the core copyrights,
veto rights over future licensing activities of SCO, and 95% of the licensing revenue. The SCO
Group disagreed with this, and the dispute resulted in the SCO v. Novell lawsuit. On August 10,
2007, a major portion of the case was decided in Novell's favor (that Novell had the copyright to
UNIX, and that the SCO Group had improperly kept money that was due to Novell). The court
also ruled that "SCO is obligated to recognize Novell's waiver of SCO's claims against IBM and
Sequent". After the ruling, Novell announced they have no interest in suing people over Unix
and stated, "We don't believe there is Unix in Linux".[13][14][15] SCO successfully got the 10th
Circuit Court of Appeals to partially overturn this decision on August 24, 2009 which sent the
lawsuit back to the courts for a jury trial.[16][17][18]

On March 30, 2010, following a jury trial, Novell, and not The SCO Group, is "unanimously
[found]" to be the owner of the UNIX and UnixWare copyrights.[19] The SCO Group, through
bankruptcy trustee Edward Cahn, has decided to continue the lawsuit against IBM for causing a
decline in SCO revenues.[20]

See also: SCO-Linux controversies

In 2005, Sun Microsystems released the bulk of its Solaris system code (based on UNIX System
V Release 4) into an open source project called OpenSolaris. New Sun OS technologies, notably
the ZFS file system, were first released as open source code via the OpenSolaris project. Soon
afterwards, OpenSolaris spawned several non-Sun distributions. In 2010, after Oracle acquired
Sun, OpenSolaris was officially discontinued, but the development of derivatives continued.

[edit] Standards
Beginning in the late 1980s, an open operating system standardization effort now known as
POSIX provided a common baseline for all operating systems; IEEE based POSIX around the
common structure of the major competing variants of the Unix system, publishing the first
POSIX standard in 1988. In the early 1990s a separate but very similar effort was started by an
industry consortium, the Common Open Software Environment (COSE) initiative, which
eventually became the Single UNIX Specification administered by The Open Group. Starting in
1998 the Open Group and IEEE started the Austin Group, to provide a common definition of
POSIX and the Single UNIX Specification.

In an effort towards compatibility, in 1999 several Unix system vendors agreed on SVR4's
Executable and Linkable Format (ELF) as the standard for binary and object code files. The
common format allows substantial binary compatibility among Unix systems operating on the
same CPU architecture.

The Filesystem Hierarchy Standard was created to provide a reference directory layout for Unix-
like operating systems, particularly Linux.

[edit] Components
See also: List of Unix programs

The Unix system is composed of several components that are normally packed together. By
including – in addition to the kernel of an operating system – the development environment,
libraries, documents, and the portable, modifiable source-code for all of these components, Unix
was a self-contained software system. This was one of the key reasons it emerged as an
important teaching and learning tool and has had such a broad influence.

The inclusion of these components did not make the system large – the original V7 UNIX
distribution, consisting of copies of all of the compiled binaries plus all of the source code and
documentation occupied less than 10MB, and arrived on a single 9-track magnetic tape. The
printed documentation, typeset from the on-line sources, was contained in two volumes.

The names and filesystem locations of the Unix components have changed substantially across
the history of the system. Nonetheless, the V7 implementation is considered by many to have the
canonical early structure:

 Kernel – source code in /usr/sys, composed of several sub-components:

o conf – configuration and machine-dependent parts, including boot code
o dev – device drivers for control of hardware (and some pseudo-hardware)
o sys – operating system "kernel", handling memory management, process scheduling,
system calls, etc.
 o h – header files, defining key structures within the system and important system-specific
invariables
 Development Environment – Early versions of Unix contained a development environment
sufficient to recreate the entire system from source code:
o cc – C language compiler (first appeared in V3 Unix)
o as – machine-language assembler for the machine
o ld – linker, for combining object files
o lib – object-code libraries (installed in /lib or /usr/lib) libc, the system library with C run-
time support, was the primary library, but there have always been additional libraries
for such things as mathematical functions (libm) or database access. V7 Unix introduced
the first version of the modern "Standard I/O" library stdio as part of the system library.
Later implementations increased the number of libraries significantly.
o make – build manager (introduced in PWB/UNIX), for effectively automating the build
process
o include – header files for software development, defining standard interfaces and
system invariants
o Other languages – V7 Unix contained a Fortran-77 compiler, a programmable arbitrary-
precision calculator (bc, dc), and the awk scripting language, and later versions and
implementations contain many other language compilers and toolsets. Early BSD
releases included Pascal tools, and many modern Unix systems also include the GNU
Compiler Collection as well as or instead of a proprietary compiler system.
o Other tools – including an object-code archive manager (ar), symbol-table lister (nm),
compiler-development tools (e.g. lex & yacc), and debugging tools.
 Commands – Unix makes little distinction between commands (user-level programs) for system
operation and maintenance (e.g. cron), commands of general utility (e.g. grep), and more
general-purpose applications such as the text formatting and typesetting package. Nonetheless,
some major categories are:
o sh – The "shell" programmable command line interpreter, the primary user interface on
Unix before window systems appeared, and even afterward (within a "command
window").
o Utilities – the core tool kit of the Unix command set, including cp, ls, grep, find and many
others. Subcategories include:
 System utilities – administrative tools such as mkfs, fsck, and many others.
 User utilities – environment management tools such as passwd, kill, and others.
o Document formatting – Unix systems were used from the outset for document
preparation and typesetting systems, and included many related programs such as nroff,
troff, tbl, eqn, refer, and pic. Some modern Unix systems also include packages such as
TeX and Ghostscript.
o Graphics – The plot subsystem provided facilities for producing simple vector plots in a
device-independent format, with device-specific interpreters to display such files.
Modern Unix systems also generally include X11 as a standard windowing system and
GUI, and many support OpenGL.
o Communications – Early Unix systems contained no inter-system communication, but
did include the inter-user communication programs mail and write. V7 introduced the
early inter-system communication system UUCP, and systems beginning with BSD
release 4.1c included TCP/IP utilities.
  Documentation – Unix was the first operating system to include all of its documentation online
in machine-readable form. The documentation included:
o man – manual pages for each command, library component, system call, header file, etc.
o doc – longer documents detailing major subsystems, such as the C language and troff

[edit] Impact
See also: Unix-like

The Unix system had significant impact on other operating systems. It won its success by:

 Direct interaction.
 Moving away from the total control of businesses like IBM and DEC.
 AT&T being willing to give the software away for free.
 Running on cheap hardware.
 Being easy to adopt and move to different machines.

It was written in high level language rather than assembly language (which had been thought
necessary for systems implementation on early computers). Although this followed the lead of
Multics and Burroughs, it was Unix that popularized the idea.

Unix had a drastically simplified file model compared to many contemporary operating systems,
treating all kinds of files as simple byte arrays. The file system hierarchy contained machine
services and devices (such as printers, terminals, or disk drives), providing a uniform interface,
but at the expense of occasionally requiring additional mechanisms such as ioctl and mode flags
to access features of the hardware that did not fit the simple "stream of bytes" model. The Plan 9
operating system pushed this model even further and eliminated the need for additional
mechanisms.

Unix also popularized the hierarchical file system with arbitrarily nested subdirectories,
originally introduced by Multics. Other common operating systems of the era had ways to divide
a storage device into multiple directories or sections, but they had a fixed number of levels, often
only one level. Several major proprietary operating systems eventually added recursive
subdirectory capabilities also patterned after Multics. DEC's RSX-11M's "group, user" hierarchy
evolved into VMS directories, CP/M's volumes evolved into MS-DOS 2.0+ subdirectories, and
HP's MPE group.account hierarchy and IBM's SSP and OS/400 library systems were folded into
broader POSIX file systems.

Making the command interpreter an ordinary user-level program, with additional commands
provided as separate programs, was another Multics innovation popularized by Unix. The Unix
shell used the same language for interactive commands as for scripting (shell scripts – there was
no separate job control language like IBM's JCL). Since the shell and OS commands were "just
another program", the user could choose (or even write) his own shell. New commands could be
added without changing the shell itself. Unix's innovative command-line syntax for creating
chains of producer-consumer processes (pipelines) made a powerful programming paradigm
(coroutines) widely available. Many later command-line interpreters have been inspired by the
Unix shell.
A fundamental simplifying assumption of Unix was its focus on ASCII text for nearly all file
formats. There were no "binary" editors in the original version of Unix – the entire system was
configured using textual shell command scripts. The common denominator in the I/O system was
the byte – unlike "record-based" file systems. The focus on text for representing nearly
everything made Unix pipes especially useful, and encouraged the development of simple,
general tools that could be easily combined to perform more complicated ad hoc tasks. The focus
on text and bytes made the system far more scalable and portable than other systems. Over time,
text-based applications have also proven popular in application areas, such as printing languages
(PostScript, ODF), and at the application layer of the Internet protocols, e.g., FTP, SMTP,
HTTP, SOAP and SIP.

Unix popularized a syntax for regular expressions that found widespread use. The Unix
programming interface became the basis for a widely implemented operating system interface
standard (POSIX, see above).

The C programming language soon spread beyond Unix, and is now ubiquitous in systems and
applications programming.

Early Unix developers were important in bringing the concepts of modularity and reusability into
software engineering practice, spawning a "software tools" movement.

Unix provided the TCP/IP networking protocol on relatively inexpensive computers, which
contributed to the Internet explosion of worldwide real-time connectivity, and which formed the
basis for implementations on many other platforms. This also exposed numerous security holes
in the networking implementations.

The Unix policy of extensive on-line documentation and (for many years) ready access to all
system source code raised programmer expectations, and contributed to the 1983 launch of the
free software movement.

Over time, the leading developers of Unix (and programs that ran on it) established a set of
cultural norms for developing software, norms which became as important and influential as the
technology of Unix itself; this has been termed the Unix philosophy.

[edit] Free Unix-like operating systems

In 1983, Richard Stallman announced the GNU project, an ambitious effort to create a free
software Unix-like system; "free" in that everyone who received a copy would be free to use,
study, modify, and redistribute it. The GNU project's own kernel development project, GNU
Hurd, had not produced a working kernel, but in 1991 Linus Torvalds released the Linux kernel
as free software under the GNU General Public License. In addition to their use in the
GNU/Linux operating system, many GNU packages – such as the GNU Compiler Collection
(and the rest of the GNU toolchain), the GNU C library and the GNU core utilities – have gone
on to play central roles in other free Unix systems as well.

Linux distributions, comprising Linux and large collections of compatible software have become
popular both with individual users and in business. Popular distributions include Red Hat
Enterprise Linux, Fedora, SUSE Linux Enterprise, openSUSE, Debian GNU/Linux, Ubuntu,
Mandriva Linux, Slackware Linux and Gentoo.

A free derivative of BSD Unix, 386BSD, was also released in 1992 and led to the NetBSD and
FreeBSD projects. With the 1994 settlement of a lawsuit that UNIX Systems Laboratories
brought against the University of California and Berkeley Software Design Inc. (USL v. BSDi),
it was clarified that Berkeley had the right to distribute BSD Unix – for free, if it so desired.
Since then, BSD Unix has been developed in several different directions, including OpenBSD
and DragonFly BSD.

Linux and BSD are now rapidly occupying much of the market traditionally occupied by
proprietary Unix operating systems, as well as expanding into new markets such as the consumer
desktop and mobile and embedded devices. Due to the modularity of the Unix design, sharing
bits and pieces is relatively common; consequently, most or all Unix and Unix-like systems
include at least some BSD code, and modern systems also usually include some GNU utilities in
their distributions.

OpenSolaris is a relatively recent addition to the list of operating systems based on free software
licenses marked as such by FSF and OSI. It includes a number of derivatives that combines
CDDL-licensed kernel and system tools and also GNU userland and is currently the only open
source System V derivative available.

[edit] 2038
Main article: Year 2038 problem

Unix stores system time values as the number of seconds from midnight January 1, 1970 (the
"Unix Epoch") in variables of type time_t, historically defined as "signed long". On January 19,
2038 on 32 bit Unix systems, the current time will roll over from a zero followed by 31 ones
(0x7FFFFFFF) to a one followed by 31 zeros (0x80000000), which will reset time to the year 1901
or 1970, depending on implementation, because that toggles the sign bit.

Since times before 1970 are rarely represented in Unix time, one possible solution that is
compatible with existing binary formats would be to redefine time_t as "unsigned 32-bit integer".
However, such a kludge merely postpones the problem to February 7, 2106, and could introduce
bugs in software that computes time differences.

Some Unix versions have already addressed this. For example, in Solaris and Linux in 64-bit
mode, time_t is 64 bits long, meaning that the OS itself and 64-bit applications will correctly
handle dates for some 292 billion years. Existing 32-bit applications using a 32-bit time_t
continue to work on 64-bit Solaris systems but are still prone to the 2038 problem. Some vendors
have introduced an alternative 64-bit type and corresponding API, without addressing uses of the
standard time_t.

[edit] ARPANET

In May 1975 ARPA documented in RFC 681 detailed very specifically why Unix is the
operating system of choice for use as an ARPANET "mini-host". The evaluation process was
also documented. Unix required a license that was very expensive with US$20,000 for non-
university users and US$150 for an educational license. It was noted that for an "ARPA network
wide license" Bell "were open to suggestions in that area".

Specific features found beneficial were:

 Local processing facilities.

 Compilers.
 Editor.
 Document preparation system.
 Efficient file system and access control.
 Mountable and de-mountable volumes.
 Unified treatment of peripherals as special files.
 The network control program (NCP), is integrated within the Unix file system.
 Network connections treated as special files which can be accessed through standard Unix I/O
calls.
 The system closes all files on program exit.
 "desirable to minimize the amount of code added to the basic Unix kernel".

Development hardware used:

"The network software for Unix was developed on a PDP-11/50, with memory management,
two RK05 disk packs, two nine track magtape drives, four dectape drives, 32k words of core, and
three terminals. Presently this has been expanded to encompass a DH11 terminal multiplexor,
an RP03 moving head disk, a twin platter RF11 fixed head disk, floating point, and 48k of core.
User files are stored on the RP03. the RF11 is used as a swap disk and for temporary file storage;
one RK05 platter contains the system files, and the second contains login and accounting
information. In the near future, the system will be expanded to 128k words of core memory
with 10 dial in and 10 hard wired terminal lines"

"The base operating system occupies 24.5k words of memory. this system includes a large
number of device drivers, and enjoys a generous amount of space for I/O buffers and system
tables. A minimal system would require 40k words of hardware memory. It should be noted that
Unix also requires the memory management"

As a comparison the "network control program" (NCP) in kernel code uses 3.5k and the
swappable userspace is approx 8.5k.

 "After telnet has proved itself reliable, the open system call will be expanded to include further
parameterization .. ability to listen on a local socket"
 "After those extensions, net mail, then network FTP and finally network RJE will be
implemented. all will run as user programs so the kernel system size will not increase."
 "Gary Grossman who participated in the design and wrote the NCP daemon"
 "Steve Bunch who was the third member of our design group and wrote the kernel message
software."
[edit] Branding
See also: List of Unix systems

In October 1993, Novell, the company that owned the rights to the Unix System V source at the
time, transferred the trademarks of Unix to the X/Open Company (now The Open Group),[10] and
in 1995 sold the related business operations to Santa Cruz Operation.[11] Whether Novell also
sold the copyrights to the actual software was the subject of a 2006 federal lawsuit, SCO v.
Novell, which Novell won; the case is being appealed.[citation needed] Unix vendor SCO Group Inc.
accused Novell of slander of title.

The present owner of the trademark UNIX is The Open Group, an industry standards consortium.
Only systems fully compliant with and certified to the Single UNIX Specification qualify as
"UNIX" (others are called "Unix system-like" or "Unix-like").

By decree of The Open Group, the term "UNIX" refers more to a class of operating systems than
to a specific implementation of an operating system; those operating systems which meet The
Open Group's Single UNIX Specification should be able to bear the UNIX 98 or UNIX 03
trademarks today, after the operating system's vendor pays a fee to The Open Group. Systems
licensed to use the UNIX trademark include AIX, HP-UX, IRIX, Solaris, Tru64 (formerly
"Digital UNIX"), A/UX, Mac OS X,[21][22] and a part of z/OS.

Sometimes a representation like "Un*x", "*NIX", or "*N?X" is used to indicate all operating
systems similar to Unix. This comes from the use of the "*" and "?" characters as "wildcard"
characters in many utilities. This notation is also used to describe other Unix-like systems, e.g.
Linux, BSD, etc., that have not met the requirements for UNIX branding from the Open Group.

The Open Group requests that "UNIX" is always used as an adjective followed by a generic term
such as "system" to help avoid the creation of a genericized trademark.

"Unix" was the original formatting, but the usage of "UNIX" remains widespread because,
according to Dennis Ritchie, when presenting the original Unix paper to the third Operating
Systems Symposium of the American Association for Computing Machinery, “we had a new
typesetter and troff had just been invented and we were intoxicated by being able to produce
small caps.”[23] Many of the operating system's predecessors and contemporaries used all-
uppercase lettering, so many people wrote the name in upper case due to force of habit.

Several plural forms of Unix are used to refer to multiple brands of Unix and Unix-like systems.
Most common is the conventional "Unixes", but "Unices" (treating Unix as a Latin noun of the
third declension) is also popular. The Anglo-Saxon plural form "Unixen" is not common,
although occasionally seen. Trademark names can be registered by different entities in different
countries and trademark laws in some countries allow the same trademark name to be controlled
by two different entities if each entity uses the trademark in easily distinguishable categories.
The result is that Unix has been used as a brand name for various products including book
shelves, ink pens, bottled glue, diapers, hair driers and food containers.[24]
 [edit] See also

What is an Operating System

An operating system (OS) is a program that allows you to interact with the computer -- all of the
software and hardware on your computer. How?

Basically, there are two ways.

 With a command-line operating system (e.g., DOS), you type a text command and the
computer responds according to that command.
 With a graphical user interface (GUI) operating system (e.g., Windows), you interact with the
computer through a graphical interface with pictures and buttons by using the mouse and
keyboard.

With Unix you have in general the option of using either command-lines (more control and flexibility)
or GUIs (easier).

Unix and Windows: Two Major Classes of Operating Systems

And they have a competitive history and future. Unix has been in use for more than three decades.
Originally it rose from the ashes of a failed attempt in the early 1960s to develop a reliable
timesharing operating system. A few survivors from Bell Labs did not give up and developed a system
that provided a work environment described as "of unusual simplicity, power, and elegance".

Since the 1980's Unix's main competitor Windows has gained popularity due to the increasing power
of micro-computers with Intel-compatible processors. Windows, at the time, was the only major OS
designed for this type of processors. In recent years, however, a new version of Unix called Linux, also
specifically developed for micro-computers, has emerged. It can be obtained for free and is therefore
a lucrative choice for individuals and businesses.

On the server front, Unix has been closing in on Microsoft’s market share. In 1999, Linux scooted past
Novell's Netware to become the No. 2 server operating system behind Windows NT. In 2001 the
market share for the Linux operating system was 25 percent; other Unix flavors 12 percent. On the
client front, Microsoft is currently dominating the operating system market with over 90% market
share.

Because of Microsoft’s aggressive marketing practices, millions of users who have no idea what an
operating system is have been using Windows operating systems given to them when they purchased
their PCs. Many others are not aware that there are operating systems other than Windows. But you
are here reading an article about operating systems, which probably means that you are trying to
make conscious OS decisions for home use or for your organizations. In that case, you should at least
give Linux/Unix your consideration, especially if the following is relevant in your environment.

Advantages of Unix
- Unix is more flexible and can be installed on many different types of machines, including main-frame
computers, supercomputers and micro-computers.

- Unix is more stable and does not go down as often as Windows does, therefore requires less
administration and maintenance.

- Unix has greater built-in security and permissions features than Windows.

- Unix possesses much greater processing power than Windows.

- Unix is the leader in serving the Web. About 90% of the Internet relies on Unix operating systems
running Apache, the world's most widely used Web server.

- Software upgrades from Microsoft often require the user to purchase new or more hardware or
prerequisite software. That is not the case with Unix.

- The mostly free or inexpensive open-source operating systems, such as Linux and BSD, with their
flexibility and control, are very attractive to (aspiring) computer wizards. Many of the smartest
programmers are developing state-of-the-art software free of charge for the fast growing "open-
source movement”.

- Unix also inspires novel approaches to software design, such as solving problems by interconnecting
simpler tools instead of creating large monolithic application programs.

Remember, no one single type of operating system can offer universal answers to all your computing
needs. It is about having choices and making educated decisions.