PSACCT in Linux : User & Process History Tracking

One of my friends wanted to log the activity of an user in a Linux Server including other details like PTS / TTY the user logged in from, the logon times and the commands the user ran throughout the session. To begin with, I thought I should be logging the 'History' user-wise and with timestamps. But that didn't help much because we did not have a single interface from where we could monitor various users at the same time. Another problem was that,  the users' login times and terminals will not be simultaneously displayed. We will have to use separate commands like 'last' to get hold of that data. The second line of thought was to redirect all the .bash_history data for all users to 'syslog' but even that turned out to be too complex.

Finally, looking for a better and easier solution, I stumbled upon a toolset package called PSACCT (Process Accounting) . PSACCT can be used to log user activity in detail. Process accounting allows to view every command executed by a user including CPU and memory consumption data. With process accounting a system Admin can find out which command executed at what time

 

The psacct package contains many tools for monitoring process activities. These are: ac, lastcomm, accton and sa.

 

·         The ac command displays statistics about how long users have been logged on.

·         The lastcomm command displays information about previous executed commands.

·         The accton command turns process accounting on or off.

·         The sa command summarizes information about previously executed commmands.

 

Installing 'psacct' or 'acct' in Linux

psacct or acct both are the same sofwares except that the psacct package is only available for rpm based distributions such as RHEL, CentOS and Fedora, whereas acct package is available for distributions like Ubuntu, Debian based Operating Systems.

 

To install psacct package under rpm based distributions issue the following yum command.

 

# yum install psacct

To install acct package using apt-get command under Ubuntu / Debian OS's -

$ sudo apt-get install acct

OR

# apt-get install acct

 

Starting psacct or acct service

By default psacct service is in disabled mode and you need to start it manually underRHEL/CentOS/Fedora systems. Use the following command to check the status of service.

# /etc/init.d/psacct status

Process accounting is disabled.

You see the status showing as disabled, so let's start it manually using the following both commands. These two commands will create a /var/account/pacct file and start services.

# chkconfig psacct on

# /etc/init.d/psacct start

Starting process accounting:                               [  OK  ]

After starting service, check the status again, you will get status as enabled as shown below.

# /etc/init.d/psacct status

Process accounting is enabled.

Under Ubuntu, Debian and Mint service is started automatically, you don't need to start it again.

 

List Last Executed Commands of User

The 'lastcomm' command is used to search and display previously executed user commands information. You can also search commands of individual usernames. For example, we see commands of user (binpipe).

Please read the lastcomm man page for other details and command switches.

# lastcomm binpipe

su                      binpipe  pts/0      0.00 secs Wed April 13 15:56

ls                      binpipe  pts/0      0.00 secs Wed April 13 15:56

ls                      binpipe  pts/0      0.00 secs Wed April 13 15:56

ls                      binpipe  pts/0      0.00 secs Wed April 13 15:56

bash               F    binpipe  pts/0      0.00 secs Wed April 13 15:56

id                      binpipe  pts/0      0.00 secs Wed April 13 15:56

grep                    binpipe  pts/0      0.00 secs Wed April 13 15:56

grep                    binpipe  pts/0      0.00 secs Wed April 13 15:56

bash               F    binpipe  pts/0      0.00 secs Wed April 13 15:56

dircolors               binpipe  pts/0      0.00 secs Wed April 13 15:56

bash               F    binpipe  pts/0      0.00 secs Wed April 13 15:56

tput                    binpipe  pts/0      0.00 secs Wed April 13 15:56

tty                     binpipe  pts/0      0.00 secs Wed April 13 15:56

bash               F    binpipe  pts/0      0.00 secs Wed April 13 15:56

id                      binpipe  pts/0      0.00 secs Wed April 13 15:56

bash               F    binpipe  pts/0      0.00 secs Wed April 13 15:56

id                      binpipe  pts/0      0.00 secs Wed April 13 15:56

Search Logs for Commands

With the help of the lastcomm command you will be able to view individual use of an each commands.

# lastcomm ls

ls                      binpipe  pts/0      0.00 secs Wed April 13 15:56

ls                      binpipe  pts/0      0.00 secs Wed April 13 15:56

ls                      binpipe  pts/0      0.00 secs Wed April 13 15:56

 

Display Statistics of Users Connect Time

ac command without specifying any argument will displays total statistics of connect time in hours based on the user logins/logouts from the current wtmp file.

# ac

total     1814.03

 

Display Statistics of Users Day-wise

Using command "ac -d" will prints out the total login time in hours by day-wise.

# ac -d

April 17  total        5.23

April 18  total       15.20

April 24  total        3.21

April 25  total        2.27

April 26  total        2.64

April 27  total        6.19

March  1  total        6.41

March  3  total        2.42

March  4  total        2.52

March  5  total        6.11

March  8  total       12.98

March  9  total       22.65

March 11  total       16.18

Display Time Totals for each User

Using command "ac -p" will print the total login time of each user in hours.

# ac -p

        root                              1645.18

        binpipe                            168.96

        total     1814.14

Display Individual User Time

To get the total login statistics time of user "binpipe" in hours, use the command as.

# ac binpipe

 total      168.96

Display Day-Wise Logn Time of User

The following command will prints the day-wise total login time of user "binpipe" in hours.

# ac -d binpipe

March 11  total        8.01

March 12  total       24.00

March 15  total       70.50

March 16  total       23.57

March 17  total       24.00

March 18  total       18.70

Nov 20  total        0.18

Print All Account Activity Information

The "sa" command is used to print the summary of commands that were executed by users.

# sa

       2       9.86re       0.00cp     2466k   sshd*

       8       1.05re       0.00cp     1064k   man

       2      10.08re       0.00cp     2562k   sshd

      12       0.00re       0.00cp     1298k   psacct

       2       0.00re       0.00cp     1575k   troff

      14       0.00re       0.00cp      503k   ac

      10       0.00re       0.00cp     1264k   psacct*

      10       0.00re       0.00cp      466k   consoletype

       9       0.00re       0.00cp      509k   sa

       8       0.02re       0.00cp      769k   udisks-helper-a

       6       0.00re       0.00cp     1057k   touch

       6       0.00re       0.00cp      592k   gzip

       6       0.00re       0.00cp      465k   accton

       4       1.05re       0.00cp     1264k   sh*

       4       0.00re       0.00cp     1264k   nroff*

       2       1.05re       0.00cp     1264k   sh

       2       1.05re       0.00cp     1120k   less

       2       0.00re       0.00cp     1346k   groff

       2       0.00re       0.00cp     1383k   grotty

       2       0.00re       0.00cp     1053k   mktemp

       2       0.00re       0.00cp     1030k   iconv

       2       0.00re       0.00cp     1023k   rm

       2       0.00re       0.00cp     1020k   cat

       2       0.00re       0.00cp     1018k   locale

       2       0.00re       0.00cp      802k   gtbl

Where

o    9.86re is a "real time" as per wall clock minutes

o    0.01cp is a sum of system/user time in cpu minutes

o    2466k is a cpu-time averaged core usage, i.e. 1k units

o    sshd command name

Print Individual User Information

To get the information of individual user, use the options -u.

# sa -u

root       0.00 cpu      465k mem accton

root       0.00 cpu     1057k mem touch

root       0.00 cpu     1298k mem psacct

root       0.00 cpu      466k mem consoletype

root       0.00 cpu     1264k mem psacct           *

root       0.00 cpu     1298k mem psacct

root       0.00 cpu      466k mem consoletype

root       0.00 cpu     1264k mem psacct           *

root       0.00 cpu     1298k mem psacct

root       0.00 cpu      466k mem consoletype

root       0.00 cpu     1264k mem psacct           *

root       0.00 cpu      465k mem accton

root       0.00 cpu     1057k mem touch

Print Number of Processes

This command prints the total number of processes and CPU minutes. If you see continue increase in these numbers, then its time to look into the system about what is happening.

# sa -m

sshd                                    2       9.86re       0.00cp     2466k

root                                  127      14.29re       0.00cp      909k

Print Sort by Percentage

The command "sa -c" displays the highest percentage of users.

# sa -c

 132  100.00%      24.16re  100.00%       0.01cp  100.00%      923k

       2    1.52%       9.86re   40.83%       0.00cp   53.33%     2466k   sshd*

       8    6.06%       1.05re    4.34%       0.00cp   20.00%     1064k   man

       2    1.52%      10.08re   41.73%       0.00cp   13.33%     2562k   sshd

      12    9.09%       0.00re    0.01%       0.00cp    6.67%     1298k   psacct

       2    1.52%       0.00re    0.00%       0.00cp    6.67%     1575k   troff

      18   13.64%       0.00re    0.00%       0.00cp    0.00%      509k   sa

      14   10.61%       0.00re    0.00%       0.00cp    0.00%      503k   ac

      10    7.58%       0.00re    0.00%       0.00cp    0.00%     1264k   psacct*

      10    7.58%       0.00re    0.00%       0.00cp    0.00%      466k   consoletype

       8    6.06%       0.02re    0.07%       0.00cp    0.00%      769k   udisks-helper-a

       6    4.55%       0.00re    0.00%       0.00cp    0.00%     1057k   touch

       6    4.55%       0.00re    0.00%       0.00cp    0.00%      592k   gzip

       6    4.55%       0.00re    0.00%       0.00cp    0.00%      465k   accton

       4    3.03%       1.05re    4.34%       0.00cp    0.00%     1264k   sh*

       4    3.03%       0.00re    0.00%       0.00cp    0.00%     1264k   nroff*

       2    1.52%       1.05re    4.34%       0.00cp    0.00%     1264k   sh

       2    1.52%       1.05re    4.34%       0.00cp    0.00%     1120k   less

       2    1.52%       0.00re    0.00%       0.00cp    0.00%     1346k   groff

       2    1.52%       0.00re    0.00%       0.00cp    0.00%     1383k   grotty

       2    1.52%       0.00re    0.00%       0.00cp    0.00%     1053k   mktemp

Reset Kayako Admin password from Mysql Database

Kayako is a leading helpdesk application, which has found global acceptance. Since Kayako follows a strict account lockout policy, I have seen many System Admins looking for a quick way to reset password under such circumstances when the helpdesk front end is unaccessible due to a forgotten or a lost password. Here, we discuss a quick way to reset the admin password from the database backend.

Use the following MySQL query to fetch a list of your staff users:
select title,fullname,username from swstaff sw join swstaffgroup sg on (sw.staffgroupid=sg.staffgroupid);

Identify the username of the staff user you will reset the password of, then tailor this second MySQL query:
update swstaff set staffpassword=sha1('qwerty012!@#') where username='admin';

This will reset the password of the 'admin' staff user to 'qwerty012!@#'.

Shell Script to Check a Port is Open or Closed ~ BINPIPE

Shell Script to Check a Port is Open or Closed ~ BINPIPE

Shell Script to Check a Port is Open or Closed

The below Shell Script will check if a port is open or closed. Save the below code to a file named checkport and give executable permission to that file.

Usage: checkport <hostname or IP> <Port No.>

#!/bin/bash  nc -z $1 $2 1>/dev/null 2>&1; result=$?;   if [ $result -eq 0 ]; then       echo 'The port is open for tcp connections'   else       echo 'The port is closed'   fi

Using telnet `telnet <host_or_ip> <port>' will do the job, but it doesn't leave room for automation, so this script is better alternative.

Configuring MSMTP to work as SMTP Mail Relay for Php Mail() in Linux Server

Most of the times, the php mail function dosen't work in Linux servers because Postfix or Sendmail relays are misconfigured. To workaround this situation you can build your own mail relay using MSMTP and use an SMTP server as the carrier I have tested this solution in a CentOS 5.8 box, however it should work in any Linux distribution as we compile the main package (msmtp) from source.

Please follow the following command sequence to get going. In case of commands not working for you please post a comment so that we can help.

yum install make gcc pkgconfig
wget http://voxel.dl.sourceforge.net/sourceforge/msmtp/msmtp-1.x.x.tar.bz2
tar xjvf msmtp-1.x.x.tar.bz2
cd msmtp-x.x
./configure
make
make install
make clean

vi /etc/msmtp   (put the contents below)

#####NO_TLS#####
defaults
tls off
tls_certcheck off
tls_starttls off
#tls_trust_file /etc/ssl/certs/ca-certificates.crt

account default
host smtp.com
port 25
auth on
user support@testdomain.com
password xxxxxxx
from no-reply@testdomain.com
logfile /var/log/msmtp.log
#####

OR


#####WITH TLS#####
defaults
tls on
tls_certcheck off
tls_starttls on
#tls_trust_file /etc/ssl/certs/ca-certificates.crt

account default
host smtp.testdomain.com
port 587
auth on
user binpipe@testdomain.com
password xxxxxxx
from no-reply@testdomain.com
logfile /var/log/msmtp.log
#####


ln -s /etc/msmtp /usr/local/etc/msmtprc

touch /var/log/msmtp.log
chmod 777 /var/log/msmtp.log

Test with this:

echo -e "Subject: Test Mail\r\n\r\nThis is a test mail" |msmtp --debug --from=test@example.com -t tester@testmail.com


To integrate with php, write this in /etc/php.ini

;sendmail_path = /usr/sbin/sendmail -t -i
sendmail_path = /usr/local/bin/msmtp -t -i


service httpd restart

Now, php mail function will work using smtp as the relay and mails will go through.

Autocomplete in shell scripts with COMPGEN

Today, I was required to write a shell script which will detect a directory which has a random name-ending (Eg. version-YQEG$1200 etc), enter it and run some operations inside that directory. When we work manually in commandline, this is easy because since we know that the directory begins with 'version' we type version and hit 'tab' key which autocompletes it. But how to do that in a shell script ?

That is when this command compgen came to my rescue.

So lets have a look at a command that I did not know hitherto: compgen

The compgen builtin command expands a list of arguments to generate completion matches, so it allows us to display the commands, functions, directories and aliases available for the current user. In short compgen is a command that displays commands.

So compgen is a command builtin in the bash shell so if you use this shell you have this command available in the terminal,
Let’s see how we can use it directly from the command line, as first thing let’s say that in general compgen has different options that you can use to specify which kind of thing you are looking for, so the basic usage is:

compgen [option] [word]


This generate possible completion matches for word according to the options, which may be any option accepted by the complete builtin with the exception of -p and -r, and write the matches to the standard output let’s see some examples:

Working with directories

The option for directories is -d so compgen -d will give as output the list of all the directory in the current path, or you can ask for a specific path such as:

xubuntu-home:~$ compgen -d /etc/s /etc/sudoers.d /etc/sysctl.d /etc/sensors.d /etc/skel /etc/samba /etc/sane.d /etc/ssh /etc/speech-dispatcher /etc/systemd /etc/sgml /etc/ssl /etc/sound /etc/smartmontools /etc/snmp /etc/security

In this case I’ve asked compgen to show all the directory in /etc/ that start with the letter s, using just /etc/ as parameter it will show all the directories in /etc/.

Working with commands

In the same way compgen can also show all the available commands for an user with the flag -c : compgen -c if you run this command probably you’ll get a list of some thousands of commands so it’s better to use it with a | less or to show only some commands use an additional parameter that represents a part of the word you are looking for:

xubuntu-home:~$ compgen -c ss sshd ssgrep ssh-vulnkey ssindex ssconvert ssh-import-id ssh ssh-copy-id ssh-argv0 ssh-keygen ssh-askpass ssh-add ssh-keyscan ssh-agent ss

In this example I can see all the commands that i can use that start with ss

Other Functions

This is a short list of options that you can use with compgen:

Show all the bash built-ins commands.

compgen -b

Show all the files and directories in current path (or specify a path like in the directory)

compgen -f

Show all the bash keywords

xubuntu-home:~# compgen -k if then else elif fi case esac for select while until do done in function time { } ! [[ ]] coproc

Show all the bash functions
compgen -A function

Show all the alias available for the current user

xubuntu-home:~$ compgen -a vps egrep fgrep grep ls

Show all the usernames defined in the server:

xubuntu-home:~# compgen -u s sys sync syslog speech-dispatcher saned sshd sslh

Show all the groups defined in the server:

xubuntu-home:~# compgen -g p proxy plugdev pulse pulse-access postfix postdrop

Free Up RAM in Redhat & CentOS Linux

Memory normally is occupied by 3 main objects, caches debtries and inodes. However you can free memory at times with the following commands.

1. Sync essentials to disk first by running the following command:

Code:

sync  

2. Cleanup the RAM:

Writing to this will cause the kernel to drop clean caches, dentries and inodes from memory, causing that memory to become free.

To free pagecache:

Code:

echo 1 > /proc/sys/vm/drop_caches

To free dentries and inodes:

Code:

echo 2 > /proc/sys/vm/drop_caches

To free pagecache, dentries and inodes:

Code:

echo 3 > /proc/sys/vm/drop_caches

As this is a non-destructive operation and dirty objects are not freeable, the user should run `sync' first.

Create graphs using SAR command and KSAR in Linux

You can use KSar tool in combination with sar command to generate and analyze server load and other parameters.



Here's how:

1. Install Sysstat rpm in your server.
2. It will provide the sar package.
3. Let sar record data for a day.
4. Run the following command:
# LC_ALL=C sar -A > /tmp/log.txt
5. Install K-Sar in a windows or Linux Box. (The package is available in sourceforge)
6. Import the log file generated in step 4 above.
7. Generate the graph.

Please note that since Ksar is a java based tool, so it can run in both Linux and Windows machines having JRE installed. Just the sar log has to be fed into it to generate the graph as described above.

Monitor with SAR command in Linux

How to Check Queue Activity (sar -q)

1. Use the sar -q command to report the following information:

   • The Average queue length while the queue is occupied.

   • The percentage of time that the queue is occupied.

   $ sar -q 00:00:00 runq-sz %runocc swpq-sz %swpocc

   The following list describes the output from the -q option.

   runq-sz

   The number of kernel threads in memory that are waiting for a CPU to run. Typically, this value should be less than 2. Consistently higher values mean that the system might be CPU-bound.

   %runocc

   The percentage of time that the dispatch queues are occupied.

   swpq-sz

   Swap queue of processes for the sar command.

   %swpocc

   Swap queue of processes for the sar command.

   How to Check Unused Memory (sar -r)

2. Use the sar -r command to report the number of memory pages and swap-file disk blocks that are currently unused.

   $ sar -r 00:00:00 freemem freeswap 01:00:00 2135 401922

   The following list describes the output from the -r option:

   freemem

   The average number of memory pages that are available to user processes over the intervals sampled by the command. Page size is machine-dependent.

   freeswap

   The number of 512-byte disk blocks that are available for page swapping.

   How to Check CPU Utilization (sar -u)

3. Use the sar -u command to display CPU utilization statistics.

   $ sar -u 00:00:00 %usr %sys %wio %idle 01:00:00 0 0 0 100

   The sar command without any options is equivalent to the sar -u command. At any given moment, the processor is either busy or idle. When busy, the processor is in either user mode or system mode. When idle, the processor is either waiting for I/O completion or “sitting still” with no work to do.

   The following list describes output from the -u option:

   %usr

   Lists the percentage of time that the processor is in user mode.

   %sys

   Lists the percentage of time that the processor is in system mode.

   %wio

   Lists the percentage of time that the processor is idle and waiting for I/O completion.

   %idle

   Lists the percentage of time that the processor is idle and not waiting for I/O.

   A high %wio value generally means that a disk slowdown has occurred

   How to Check System Table Status (sar -v)

4. Use the sar -v command to report the status of the process table, inode table, file table, and shared memory record table.

   $ sar -v 00:00:00 proc-sz ov inod-sz ov file-sz ov lock-sz 01:00:00 43/922 0 2984/4236 0 322/322 0 0/0

How to Check Swapping Activity (sar -w)

1. Use the sar -w command to report swapping and switching activity.

   $ sar -w 00:00:00 swpin/s bswin/s swpot/s bswot/s pswch/s 01:00:00 0.00 0.0 0.00 0.0 22

   The following list describes target values and observations related to the sar -w command output.

   swpin/s

   The number of LWP transfers into memory per second.

   bswin/s

   The number of blocks transferred for swap-ins per second. /* (float)PGTOBLK(xx->cvmi.pgswapin) / sec_diff */.

   swpot/s

   The average number of processes that are swapped out of memory per second. If the number is greater than 1, you might need to increase memory.

   bswot/s

   The number of blocks that are transferred for swap-outs per second.

   pswch/s

   The number of kernel thread switches, per second.

   ---

   Note –

   All process swap-ins include process initialization.

How to Check Terminal Activity (sar -y)

1. Use the sar -y command to monitor terminal device activities.

   $ sar -y 00:00:00 rawch/s canch/s outch/s rcvin/s xmtin/s mdmin/s 01:00:00 0 0 0 0 0 0

   If you have a lot of terminal I/O, you can use this report to determine if any bad lines exist. The activities recorded are defined in the following list.

   rawch/s

   Input characters (raw queue) per second.

   canch/s

   Input characters that are processed by canon (canonical queue) per second.

   outch/s

   Output characters (output queue) per second.

   rcvin/s

   Receiver hardware interrupts per second.

   xmtin/s

   Transmitter hardware interrupts per second.

   mdmin/s

   Modem interrupts per second.

   The number of modem interrupts per second (mdmin/s) should be close to zero. The receive and transmit interrupts per second (xmtin/s and rcvin/s) should be less than or equal to the number of incoming or outgoing characters, respectively. If not, check for bad lines.

   How to Check Overall System Performance (sar -A)

2. Use the sar -A command to display statistics from all options to provide a view of overall system performance.

   This command provides a more global perspective. If data from more than a single time segment is shown, the report includes averages.

   How to Set Up Automatic Data Collection

3. Become superuser or assume an equivalent role.

   Roles contain authorizations and privileged commands.

4. Run the svcadm enable system/sar:default command.

   This version of the sadc command writes a special record that marks the time when the counters are reset to zero (boot time).

5. Edit the /var/spool/cron/crontabs/sys crontab file.

   ---

   Note –

   Do not edit a crontab file directly. Instead, use the crontab -e command to make changes to an existing crontab file.

   ---

   # crontab -e sys

6. Uncomment the following lines:

   0 * * * 0-6 /usr/lib/sa/sa1 20,40 8-17 * * 1-5 /usr/lib/sa/sa1 5 18 * * 1-5 /usr/lib/sa/sa2 -s 8:00 -e 18:01 -i 1200 -A

Generate CSR in Linux Server

To create a signed ssl certificate from a trusted certificate authority
you will need to generate a csr, certificate signing request, which will
then be sent to your CA where they can generate the ssl cert for you to
install on the server. This is only necessary for production servers
with a domain or sub domain pointing to it. For non-production use it is
fine to just use a self signed cert that will not cost anything.

yum -y install mod_ssl openssl
cd /etc/pki/tls/certs
openssl genrsa -out new_ssl_cert.key 2048
openssl req -new -key new_ssl_cert.key -out new_ssl_cert.csr

Install CurlFTPFS on CentOS or Redhat Linux

CurlFTPFS is a very good tool to mount remote FTP directory to the local filesystem. The following commands are used to install the package:

yum install glib2-devel  yum install fuse-devel  yum install libcurl-devel  wget http://sourceforge.net/projects/curlftpfs/files/latest/download  tar xvzf curlftpfs-0.9.2.tar.gz  cd curlftpfs-0.9.2  ./configure  make  make install      

If you have any dependency issues, then please use the following rpm package to install  CurlFTPFS:

http://pkgs.repoforge.org/fuse-curlftpfs/

Once done, the directory can be mounted with the command syntax from here:

http://curlftpfs.sourceforge.net/