Search Posts on Binpipe Blog

Installing HAProxy Loadbalancer in CentOS, Redhat or Ubuntu



HAProxy is fast evolving as a dependable Loadbalancing solution. Here I have outlined the steps to install HAProxy to build your own software load balancer.

Installing HAProxy
For most distributions you can install haproxy using your distribution's package manager.  For example, to install on Debian or Ubuntu, run:

sudo aptitude install haproxy

CentOS 5
We will need to set up access to the EPEL software repository to download haproxy on CentOS 5.  Run the commands:

[root@LB01 ~]# rpm -Uvh http://dl.fedoraproject.org/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm
[root@LB01 ~]# yum -y install haproxy


CentOS 6
We will need to set up access to the EPEL software repository to download haproxy on CentOS 6, but the address for the RPM is different from CentOS 5.  Run the commands:

[root@LB01 ~]# rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-6.noarch.rpm
[root@LB01 ~]# yum -y install haproxy


Install a base config
Once installed backup the HAProxy config file and download the managed cloud config:

[root@LB01 ~]# cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.bak
[root@LB01 ~]# wget http://c818095.r95.cf2.rackcdn.com/haproxy.cfg -O /etc/haproxy/haproxy.cfg
chkconfig haproxy on


Configuring HAProxy

Configuring HAProxy can only come after you have your web heads configured as you will need to utilize their 10.x service net IP address's. The reason we use the service net is because the customer will not be charged for bandwidth overage, and the service net is also faster in terms of throughput as shown in the chart at the top.

Editing /etc/haproxy/haproxy.cfg - There are a number of items that need to be changed in order to get HAProxy functional. These will be outlined below. Keep in mind you need to edit these values to reflect the server's IP's.

First and foremost change

listen webfarm 0.0.0.0:80
to

listen webfarm 127.0.0.1:80
Edit 127.0.0.1 to reflect your server's eth0 or public IP.


Now you can add your web servers. In the following you will want to replace the 10.0.0.X IP address with that of the eth1 or private IP address of web servers"

   server WWW1 10.0.0.1:80 check # Active in rotation
   server WWW2 10.0.0.2:80 check # Active in rotation
   server WWW3 10.0.0.3:80 check # Active in rotation
   server WWW4 10.0.0.4:80 check backup # Not active "sorry server" - this one comes live if all web heads are  down


Above is an example of what a four server config would look like. Once you have completed this portion you can then start HAProxy and start serving pages(assuming your web servers are ready).

service haproxy start

Below is the default configuration template for haproxy.cfg:

   #global options
   global
      
       #logging is designed to work with syslog facility's due to chrooted environment
       #log loghost    local0 info - By default this is commented out
      
       #chroot directory
       chroot /usr/share/haproxy
      
       #user/group id
       uid 99
       gid 99
      
       #running mode
       daemon
   defaults
      
       #HTTP Log format
       mode http
       #number of connection retries for the session
       retries 3
      
       #try another webhead if retry fails
       option redispatch
       #session settings - max connections, and session timeout values
       maxconn 10000
       contimeout 10000
       clitimeout 50000
       srvtimeout 50000
   #Define your farm
   #listen webfarm 0.0.0.0:80 - Pass only HTTP traffic and bind to port 80
   listen webfarm 0.0.0.0:80
      
       #HTTP Log format
       mode http
       #stats uri /haproxy - results in http://<load balancer ip>/haproxy (shows load balancer stats)
       stats uri /haproxy
       #balance roundrobin - Typical Round Robin
       #balance leastconn - Least Connections
       #balance static-rr - Static Round Robin - Same as round robin, but weights have no effect
       balance roundrobin
       #cookie <COOKIENAME> prefix - Used for cookie-based persistence
       cookie webpool insert
       #option httpclose - http connection closing
       option  httpclose
       #option forwardfor - best stated as "Enable insertion of the X-Forwarded-For header to requests sent to the web heads" aka send EU IP
       option forwardfor
          
       #Web Heads (Examples)
       #server WEB1 10.0.0.1:80 check - passes http traffic to this server and checks if its alive
       #server WEB1 10.0.0.1:80 check port 81 - same as above but checks port 81 to see if its alive (helps to remove servers from rotation)
       #server WEB1 10.0.0.1:80 check port 81 weight 100 - same as the above with weight specification (weights 1-256 / higher number higher weight)
       #server WEB1 10.0.0.1:80 check backup - defines this server as a backup for the other web heads
       #Working Example: *USE THIS HOSTNAME FORMAT*
       server WWW1 10.0.0.1:80 cookie webpool_WWW1 check port 81 # Active in rotation   
       server WWW2 10.0.0.2:80 cookie webpool_WWW2 check port 81 # Active in rotation
       server WWW3 10.0.0.3:80 check # Active in rotation
       server WWW4 10.0.0.4:80 check backup # Not active "sorry server" - this one comes live if all web heads are down
   #SSL farm example
   #listen https 0.0.0.0:443
   #    mode tcp
   #    server WEB1 10.0.0.1:443 check


Session Persistence with SSL

If the you wish to also balance SSL traffic, you will need to set the balance mode to "source" This setting takes a hash of the client's IP address and the number of servers in rotation, sending traffic from one IP address to the same web server consistently. The persistence will be reset if the number of servers is changed.:

listen https 0.0.0.0:443
mode tcp
balance source
server WEB1 10.0.0.1:443 check

 

Postfix as SMTP Relay


If you need to install Postfix and use it as a SMTP relay, you can follow these steps below:
Install Postfix and cyrus-sasl with your application manager of choice. If you're compiling from source, be sure to make Postfix with the -DUSE_SASL_AUTH flag for SASL support and -DUSE_TLS for TLS support.
$ yum install postfix cyrus-sasl
Stop the sendmail service
$ /etc/init.d/sendmail stop
Remove sendmail from the startup runlevels
$ chkconfig --del sendmail
Configre the Postfix as I have mentioned here:
Edit /etc/postfix/main.cf
# Set this to your server's fully qualified domain name.
# If you don't have a internet domain name,
# use the default or your email addy's domain - it'll keep
# postfix from generating warnings all the time in the logs
mydomain = local.domain
myhostname = host.local.domain
# Set this to your email provider's smtp server.
# A lot of ISP's (ie. Cox) block the default port 25
# for home users to prevent spamming.  So we'll use port 80
relayhost = yourisp.smtp.servername:80
smtpd_sasl_auth_enable = yes
smtpd_sasl_path = smtpd
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_type = cyrus
smtp_sasl_auth_enable = yes
# optional: necessary if email provider uses load balancing and
# forwards emails to another smtp server
# for delivery (ie: smtp.yahoo.com --> smtp.phx.1.yahoo.com)
smtp_cname_overrides_servername = no
# optional: necessary if email provider
# requires passwords sent in clear text
smtp_sasl_security_options = noanonymous
There's roughly a 99.9% chance that your email provider's SMTP server requires authentication. We need to set that up with the username and password given by your email provider.
Add the following line to /etc/postfix/sasl_passwd
yourisp.smtp.servername:80 username:password
The above server hostname and port must exactly match the value for "relayhost" in /etc/postfix/main.cf.
Generate a postfix lookup table from the previous file
$ postmap hash:/etc/postfix/sasl_passwd
Test the lookup table, if all is good then the following will return the specified username:password
$ postmap -q yourisp.smtp.servername:80 /etc/postfix/sasl_passwd
Make sure the sasl_passwd and sasl_passwd.db files are readable/writable only by root
$ chmod 600 /etc/postfix/sasl_passwd
$ chmod 600 /etc/postfix/sasl_passwd.db
Add postfix to be started at boot
$ chkconfig --add postfix
Fire up Postfix
$ /etc/init.d/postfix start
Test it out using sendmail alias from the command prompt
$ sendmail email@example.com
Postfix is good to go.

--The Below Steps are specifics. Please ignore if nor required.--

If you're attempting to relay mail using Gmail, then it will be necessary to use TLS with Postfix. You'll have to point Postfix at your server's trusted CA root certificate bundle. If that is the case then read below or else ignore.
First, double-check that Postfix was configured with SSL support (ie. ldd should return at least one line starting with libssl):
$ whereis -b postfix
postfix: /usr/sbin/postfix /etc/postfix /usr/libexec/postfix
$ ldd /usr/sbin/postfix
...
libssl.so.6 => /lib/libssl.so.6 (0x00111000)
...
Now we need to find your server's CA root certificate bundle, which is typically distributed with openssl. The bundle file is used by Postfix to verify Gmail's SSL certificate (signed by Thawte). On my CentOS server, this file was located at /etc/pki/tls/certs/ca-bundle.crt, but may be in a different location on your box (ie. /etc/ssl/certs).
$ locate ca-bundle.crt
/etc/pki/tls/certs/ca-bundle.crt
Edit /etc/postfix/main.cf with the following values:
relayhost = smtp.gmail.com:587
# your FQDN, or default value below
mydomain = local.domain
# your local machine name, or default value below
myhostname = host.local.domain
myorigin = $myhostname
# SASL
smtpd_sasl_path = smtpd
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_type = cyrus
smtp_sasl_auth_enable = yes
smtp_sasl_security_options = noanonymous
# TLS
smtp_sasl_tls_security_options = noanonymous
smtp_use_tls  = yes
smtp_tls_CAfile = /path/to/your/ca-bundle.crt
smtp_sasl_tls_security_options = noanonymous
If you haven't already, add the following to /etc/postfix/sasl_passwd
smtp.gmail.com:587 username:password
Generate a postfix lookup table from the previous file
$ postmap hash:/etc/postfix/sasl_passwd
Make sure the sasl_passwd and sasl_passwd.db files are readable/writable only by root
$ chmod 600 /etc/postfix/sasl_passwd
$ chmod 600 /etc/postfix/sasl_passwd.db
Restart postfix and send a test email
$ postfix reload
$ sendmail email@example.com
Test relay thru Gmail

If you need to do some debugging please read below or ignore:

Monitor postfix mail log in a separate session with the following command
$ tail -f /var/log/maillog
If the log is displaying the following error
(Authentication failed: cannot SASL authenticate to server ...: no mechanism available)
then set this variable in /etc/postfix/main.cf
smtp_sasl_security_options = noanonymous
If the log is displaying this error
553 Sorry, that domain isn't in my list of allowed rcpthosts. (in reply to RCPT TO command)
check your username and password in /etc/postfix/sasl_passwd. Your user name is usually your full email address. If you have to fix it, don't forget to use postmap to generate a new lookup table.

Load Balancing Techniques

Load balancing is a term that describes a method to distribute incoming socket connections to different servers. It’s not distributed computing, where jobs are broken up into a series of sub-jobs, so each server does a fraction of the overall work. It’s not that at all. Rather, incoming socket connections are spread out to different servers. Each incoming connection will communicate with the node it was delegated to, and the entire interaction will occur there. Each node is not aware of the other nodes existence.
Why do you need load balancing?
Simple answer: Scalability and Redundancy.
Scalability
If your application becomes busy, resource limits, such as bandwidth, cpu, memory, disk space, disk I/O, and more may reach its limits. In order to remedy such problem, you have two options: scale up, or scale out. Load balancing is a scale out technique. Rather than increasing server resources, you add cost effective, commodity servers, creating a “cluster” of servers that perform the same task. Scaling out is more cost effective, because commodity level hardware provides the most bang for the buck. High end super computers come at a premium, and can be avoided in many cases.
Redundancy
Servers crash, this is the rule, not the exception. Your architecture should be devised in a way to reduce or eliminate single points of failure (SPOF). Load balancing a cluster of servers that perform the same role provides room for a server to be taken out manually for maintenance tasks, without taking down the system. You can also withstand a server crashing. This is called High Availability, or HA for short. Load balancing is a tactic that assists with High Availability, but is not High Availability by itself. To achieve high availability, you need automated monitoring that checks the status of the applications in your cluster, and automates taking servers out of rotation, in response to failure detected. These tools are often bundled into Load Balancing software and appliances, but sometimes need to be programmed independently.
How to perform load balancing?
There are 3 well known ways:
  1. DNS based
  2. Hardware based
  3. Software based

DNS based
This is also known as round robin DNS. You can inject multiple A records for the same hostname. This creates a random distribution – requests for the hostname will receive the list in a random order. If you wish to weight it (say serverA can take 2x the number of requests that serverB can), you can simply add more A records for a particular IP.
Hardware based
There are many commercial vendors out there selling appliances to perform load balancing.
Hardware based load balancing is the best way to go, if you have budget for it. These appliances provide the latest features, with little fuss.
Software based
This is where it gets fun, if you’re a technology enthusiast. If your budget doesn’t allow a load balancing appliance, or if you just like doing things yourself, software based load balancing is for you. You can turn a Linux server into your own load balancing appliance. Presumably, you could also use a Windows server, maybe even a Mac, but this article doesn’t cover those. For RHEL based, the “piranha” package provides Linux Virtual Server (LVS) and piranha (an LVS management tool – web based gui). Just “yum install piranha” and you’ll have everything you need to get started. Other softwares include BalanceNG (commercial) and a basic freeware counterpart balance.
This was super simple to use. Just download, run the program. There are a few basic input parameters, and you can be load balancing in no time. This is a no frills binary program. There are no configuration files, no startup/shutdown programs, no logging or reporting. But it does have a nifty console that you can get runtime statistics from. You could create your own tools around “balance” to monitor and gather statistics.
LVS and piranha on RHEL or CentOS

piranha is a gui that makes configuring Linux Virtual Server (LVS) easy. Here are some of the virtual server scheduling features:
  • Round robin
  • Weighted least-connections
  • Weighted round robin
  • Least-connection
  • Locality-Based Least-Connection Scheduling
  • Locality-Based Least-Connection Scheduling (R)
  • Destination Hash Scheduling
  • Source Hash Scheduling
Courtesy: http://www.koopman.me

Directory Tree in Linux


If you want to get the hierarchy of the directory in a tree format there is the tree command for that. But here I have put the command in a small loop so that the output becomes more informative with the directory and file sizes also.

Try this command out in the bash shell and see the output:

#     for i in $(ls -d */); do tree  $i ; done

Note: If the directory structure is too long to caontain it in a screen you can use the following to save it to a file:

#     for i in $(ls -d */); do tree  $i ; done > result.txt

Find Total Number of Sub-directories and Files in a Directory


We have often wondered how many sub-directories and files are there recursively under a directory in Linux. Sometimes, we may also need to calculate the number so that we don't cross inode count in shared hosting servers.

The following code snippet will allow you to get the count of files & directories inside a folder. Please run this command after doing a 'cd' to inside the folder.

find $DIR -exec stat -c '%F' {} \; | sort | uniq -c | sort -rn

If you want a tree view of the directories with the count you can use this:

for i in $(ls -d */); do tree  $i ; done > result.txt

or try this one below:

for i in $(ls -d */); do tree $i | grep -v \\-\\-\  ; done

Quite simple but effective one-liners aren't they?  :-)

Configure NTP to Synchronize Server Time


The following steps are sufficient to make the NTP Sync.

Login as the root user

Type the following command to install NTP:
# yum install ntpTurn on service:
# chkconfig ntpd onSynchronize the system clock with 0.pool.ntp.org server:
# ntpdate pool.ntp.orgStart the NTP:
# /etc/init.d/ntpd start

Passwordless SSH Login

You want to access computer A securely from computer B without having to enter a password. This technique is very useful for automation, where you don't want to put passwords into scripts.

On computer B login as the user you want to use to access computer A, and generate yourself a pair of public private keys by issuing the following command, and simply press return to the prompts to use an
empty passphrase. On computer B login as the user you want to use to access computer A, and generate yourself a pair of public private keys by issuing the following command, and simply press return to the prompts to use an empty passphrase.

ssh-keygen -t rsa

This command created a public/private key pair in your home directory under the directory of ~/.ssh. Your public key is stored in the file 

~/.ssh/id_rsa.pub

Now we need to create an ~/.ssh directory on the remote computer B if it doesn't exist, under the username we're going to access it with, substitute the username your going to use in the following. you'll need
at this stage to enter the password for username@B

> ssh username@B mkdir -p .ssh
username@B's password:

Now use ssh to push your public key to computer B via the following command, you'll again need to enter username@b's password

> cat .ssh/id_rsa.pub | ssh username@B 'cat >> .ssh/authorized_keys'
username@B's password:

That's it !

From now on entering

ssh username@b

Will get you logged is as username on computer B without the need to
enter a password.

Analyse LAMP Server Slowness


Analysis of LAMP server slowness can be done basically by following these four steps:
  1. Track MySQL Queries Taking over a Second

    By default, mysql will log queries which take 10 seconds or longer. Depending on your installation, these may or may not be logged to a file. Certainly, if you have queries showing up here, at 10+ seconds, you should investigate them. In most cases, I expect that you'll be investigating performace before the point at which you have numerous queries taking over ten seconds. On a production site, the vast majority of your queries should be returning in significantly under a second.

    To lower your slow query threshold and enable logging, you'll want to modify your my.cnf (often /etc/mysql/my.cnf), and ensure that you have settings like this:
    set-variable=long_query_time=1
      log-slow-queries=/var/log/mysql/slow_query.log


    You may need to touch that file, and ensured that it is owned and writable by your mysql process. When you restart mysql, look in the mysql error log (often /var/log/mysql/mysql.err) for any errors along the lines of "Could not use /var/log/slow_query.log for logging (error 13)"_. If you see these, create the @slowquery.log@ and set it's ownership to that of your mysql user.

    Now, depending on the state of your system, slow_query.log will begin to accumulate queries. The actual format of the slow log is bit verbose, but mysqldumpslow, a perl script included with most mysql installations can parse it and produce some more meaningful output. It will take various integers in your queries (a user_id, thread_id, etc) and generalize them, so you can locate types, instead of specific queries.
    > mysqldumpslow -t=10 -s=t /var/log/slow_query.log
      Reading mysql slow query log from /var/log/slow_query.log
      Count: 46  Time=80.46s (3701s)  Lock=0.00s (0s)  Rows=512311 (117447821), bob[bob]@[10.0.0.32]
        SELECT * FROM forum_posts
    Count: 26  Time=68.26s (1775s)  Lock=0.00s (0s)  Rows=426 (117447821), bob[bob]@[10.0.0.32]
        SELECT * FROM forum_posts WHERE thread=N
    Count: 120  Time=3.52s (422s)  Lock=0.63s (76s)  Rows=58.0 (6960), bob[bob]@[10.0.0.32]
        SELECT authors FROM forum_posts WHERE lastpost > N
    ...

    The next step is analyzing this, likely throwing each of these into an EXPLAIN query (or asking yourself why you are selecting every row in the forum_posts table), adding some indexes, and rewriting some code. The scope of this article is finding the bottlenecks… fixing them is left as an exercise for the reader.
  2. Monitor PHP Memory Usage & Log Apache Delivery Times

    Out of the box, your apache install is likely using the NCSA extended/combined log format. You're going to take this format and add to pieces of data to it. The first will be the memory used by PHP during the rendering of each page. The second will be the time that apache spends delivering this page. Both of these values will be tacked onto the end of the log format. Many log processing scripts will ignore fields added onto the end of the line, so adding them here is least likely to break things.

    Unless you've mucked with it, your httpd.conf likely has lines like this:

    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
      LogFormat "%h %l %u %t \"%r\" %>s %b" common
    CustomLog /usr/local/apache/logs/access_log common
      </pre>
    
      Whichever log format is being used, this is the name at the end of the CustomLog directive, you're going to make a copy of that LogFormat, give it a name like "commondebug", and switch the CustomLog directive to use this format.
      
      The fields you will be adding are:
      * %T – The time taken to server the request in seconds
      * %{mod_php_memory_usage}n – Memory used by PHP in bytes
    
      
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %T %{mod_php_memory_usage}n" combineddebug
      LogFormat "%h %l %u %t \"%r\" %>s %b %T %{mod_php_memory_usage}n" commondebug
    CustomLog /usr/local/apache/logs/access_log commondebug
      </pre>
    
      At this point, you'll be collecting some great data in you apache logs.  You can get some good information with some quick shell magic, like so:
    
    
    
    > awk '{printf("%07d\t%d\t%s\n", $(NF), $(NF-1), $7)}' access_log  | sed 's/\?.*//' | sort -g -k1
      0001232 0       /baz.php
      0001232 0       /bar.php
      ...
    1712160 0       /foo.php
      1717640 0       /foo.php
      1907800 0       /foo.php
      2010840 0       /foo.php
      </pre>
    
      Replace, -k1 with -k2 to sort by the delivery times.  Keep in mind, delivery times will include the time to send the bytes over the network — http clients can do screwy things, and you'll occasionally see anomalous data including 120+ second connects where the client just stopped accepting packets, but didn't close the socket.
      
      From here, you'll want to examine each of the memory-hogging scripts, and anything which is consitently long-running.
  3. Log PHP Errors

    This is one that is obvious, but easy to miss. Many sites disable the display of errors, via php.ini on their production servers. (This is a good idea, preventing the revelation of any inappopriate information to end users.)

    You'll be modifying your php.ini to include lines such as these:
    error_reporting =       E_ALL & ~E_NOTICE         ; Show all errors except for notices
      display_errors  =       Off                       ; Do not print out errors (as a part of the HTML script)
      log_errors      =       On                        ; Log errors into a log file 
      error_log       =       "/var/log/php_error.log"  ; log errors to specified file


    By itself, this isn't likely to pinpoint any performance issues, but it may help you locate other issues. Although, if are really bad and you may see a slew of "maximum execution time of XX seconds exceeded" in various pages. You are more likely to see errors which correlate to long-running or memory-consuming scripts and queries identified earlier.
  4. Take Snapshots at an OS Level

    This is the area that makes the average developer wish they had a sysadmin on call. Unfortunately, for many small sites and projects, the developer is forced to wear that hat as well. At some point, if usage gets high enough, no amount of redesign or optimization will be enough to stretch your hardware further. So the question will be, what is the bottleneck? RAM, IO, CPU?

    On this topic, your options are pretty endless. Linux provides endless tools for monitoring resources, and different people will recommend different ones. Something incredibly simple, like below can be pretty informative.
    #!/usr/bin/perl
    my $URL_TO_TEST = 'http://test.test/test.test';
      my $THRESHOLD = 2;
    
    open(my $log, '>>status_log.txt');
      while (1) {
        my $start = time();
          `curl "$URL_TO_TEST" > /dev/null`;
        my $took = time() - $start;
          if ($took > $THRESHOLD) {
              print $log "$took seconds load ::";
            print $log `date`;
              print $log "\n\n";
            print $log `vmstat -a`;
              print $log "\n\n";
            print $log `uptime`;
              print $log "\n\n";
            print $log `ps awux`;
              print $log "\n\n";
            print $log `mysqladmin -hHOST -uUSER -pPASSWORD processlist`;
              print $log "----------------------------------------------\n\n";
        }
          sleep(30);
    }
After these tests you will get an idea of where the bottleneck lies and can put your strength together to resolve that.


Handy Mysql Commands

 These are a few handy Mysql commands I have always used. You will find them useful particularly if you are a newbie. Please add more of them in the comments so that it becomes a really useful thread.
 
Please note that below when you see # it means from the Linux/Unix shell. When you see mysql> it means from a MySQL prompt after logging into MySQL.


  • To login (from unix shell) use -h only if needed.

# [mysql dir]/bin/mysql -h hostname -u root -p
  • Create a database on the sql server.

mysql> create database [databasename];
  • List all databases on the sql server.

mysql> show databases;
  • Switch to a database.

mysql> use [db name];
  • To see all the tables in the db.

mysql> show tables;
  • To see database's field formats.

mysql> describe [table name];
  • To delete a db.

mysql> drop database [database name];
  • To delete a table.

mysql> drop table [table name];
  • Show all data in a table.

mysql> SELECT * FROM [table name];
  • Returns the columns and column information pertaining to the designated table.

mysql> show columns from [table name];
  • Show certain selected rows with the value "whatever".

mysql> SELECT * FROM [table name] WHERE [field name] = "whatever";
  • Show all records containing the name "Bob" AND the phone number '3444444'.

mysql> SELECT * FROM [table name] WHERE name = "Bob" AND phone_number = '3444444';
  • Show all records not containing the name "Bob" AND the phone number '3444444' order by the phone_number field.

mysql> SELECT * FROM [table name] WHERE name != "Bob" AND phone_number = '3444444' order by phone_number;
  • Show all records starting with the letters 'bob' AND the phone number '3444444'.

mysql> SELECT * FROM [table name] WHERE name like "Bob%" AND phone_number = '3444444';
  • Show all records starting with the letters 'bob' AND the phone number '3444444' limit to records 1 through 5.

mysql> SELECT * FROM [table name] WHERE name like "Bob%" AND phone_number = '3444444' limit 1,5;
  • Use a regular expression to find records. Use "REGEXP BINARY" to force case-sensitivity. This finds any record beginning with a.

mysql> SELECT * FROM [table name] WHERE rec RLIKE "^a";
  • Show unique records.

mysql> SELECT DISTINCT [column name] FROM [table name];
  • Show selected records sorted in an ascending (asc) or descending (desc).

mysql> SELECT [col1],[col2] FROM [table name] ORDER BY [col2] DESC;
  • Return number of rows.

mysql> SELECT COUNT(*) FROM [table name];
  • Sum column.

mysql> SELECT SUM(*) FROM [table name];
  • Join tables on common columns.

mysql> select lookup.illustrationid, lookup.personid,person.birthday from lookup left join person on lookup.personid=person.personid=statement to join birthday in person table with primary illustration id;
  • Creating a new user. Login as root. Switch to the MySQL db. Make the user. Update privs.

# mysql -u root -p
mysql> use mysql;
mysql> INSERT INTO user (Host,User,Password) VALUES('%','username',PASSWORD('password'));
mysql> flush privileges;
  • Change a users password from unix shell.

# [mysql dir]/bin/mysqladmin -u username -h hostname.blah.org -p password 'new-password'
  • Change a users password from MySQL prompt. Login as root. Set the password. Update privs.

# mysql -u root -p
mysql> SET PASSWORD FOR 'user'@'hostname' = PASSWORD('passwordhere');
mysql> flush privileges;
  • Recover a MySQL root password. Stop the MySQL server process. Start again with no grant tables. Login to MySQL as root. Set new password. Exit MySQL and restart MySQL server.

# /etc/init.d/mysql stop
# mysqld_safe --skip-grant-tables &
# mysql -u root
mysql> use mysql;
mysql> update user set password=PASSWORD("newrootpassword") where User='root';
mysql> flush privileges;
mysql> quit
# /etc/init.d/mysql stop
# /etc/init.d/mysql start
  • Set a root password if there is on root password.

# mysqladmin -u root password newpassword
  • Update a root password.

# mysqladmin -u root -p oldpassword newpassword
  • Allow the user "tom" to connect to the server from localhost using the password "passwd". Login as root. Switch to the MySQL db. Give privs. Update privs.

# mysql -u root -p
mysql> use mysql;
mysql> grant usage on *.* to tom@localhost identified by 'passwd';
mysql> flush privileges;
  • Give user privilages for a db. Login as root. Switch to the MySQL db. Grant privs. Update privs.

# mysql -u root -p
mysql> use mysql;
mysql> INSERT INTO db (Host,Db,User,Select_priv,Insert_priv,Update_priv,Delete_priv,Create_priv,Drop_priv) VALUES ('%','databasename','username','Y','Y','Y','Y','Y','N');
mysql> flush privileges;


or


mysql> grant all privileges on databasename.* to username@localhost;
mysql> flush privileges;
  • To update info already in a table.

mysql> UPDATE [table name] SET Select_priv = 'Y',Insert_priv = 'Y',Update_priv = 'Y' where [field name] = 'user';
  • Delete a row(s) from a table.

mysql> DELETE from [table name] where [field name] = 'whatever';
  • Update database permissions/privilages.

mysql> flush privileges;
  • Delete a column.

mysql> alter table [table name] drop column [column name];
  • Add a new column to db.

mysql> alter table [table name] add column [new column name] varchar (20);
  • Change column name.

mysql> alter table [table name] change [old column name] [new column name] varchar (50);
  • Make a unique column so you get no dupes.

mysql> alter table [table name] add unique ([column name]);
  • Make a column bigger.

mysql> alter table [table name] modify [column name] VARCHAR(3);
  • Delete unique from table.

mysql> alter table [table name] drop index [colmn name];
  • Load a CSV file into a table.

mysql> LOAD DATA INFILE '/tmp/filename.csv' replace INTO TABLE [table name] FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' (field1,field2,field3);
  • Dump all databases for backup. Backup file is sql commands to recreate all db's.

# [mysql dir]/bin/mysqldump -u root -ppassword --opt >/tmp/alldatabases.sql

or # [mysql dir]/bin/mysqldump -u root -ppassword --all-databases >/tmp/alldatabases.sql
  • Dump one database for backup.

# [mysql dir]/bin/mysqldump -u username -ppassword --databases databasename >/tmp/databasename.sql
  • Dump a table from a database.

# [mysql dir]/bin/mysqldump -c -u username -ppassword databasename tablename > /tmp/databasename.tablename.sql
  • Restore database (or database table) from backup.

# [mysql dir]/bin/mysql -u username -ppassword databasename < /tmp/databasename.sql 

  • Restore all databases from backup with --all-databases option.

# [mysql dir]/bin/mysql -u root -ppassword < /tmp/alldatabases.sql
  • Create Table Example 1.

mysql> CREATE TABLE [table name] (firstname VARCHAR(20), middleinitial VARCHAR(3), lastname VARCHAR(35),suffix VARCHAR(3),officeid VARCHAR(10),userid VARCHAR(15),username VARCHAR(8),email VARCHAR(35),phone VARCHAR(25), groups VARCHAR(15),datestamp DATE,timestamp time,pgpemail VARCHAR(255));
  • Create Table Example 2.

mysql> create table [table name] (personid int(50) not null auto_increment primary key,firstname varchar(35),middlename varchar(50),lastnamevarchar(50) default 'bato');

MYSQL Statements and clauses
ALTER DATABASE

ALTER TABLE

ALTER VIEW

ANALYZE TABLE

BACKUP TABLE

CACHE INDEX

CHANGE MASTER TO

CHECK TABLE

CHECKSUM TABLE

COMMIT

CREATE DATABASE

CREATE INDEX

CREATE TABLE

CREATE VIEW

DELETE

DESCRIBE

DO

DROP DATABASE

DROP INDEX

DROP TABLE

DROP USER

DROP VIEW

EXPLAIN

FLUSH

GRANT

HANDLER

INSERT

JOIN

KILL

LOAD DATA FROM MASTER

LOAD DATA INFILE

LOAD INDEX INTO CACHE

LOAD TABLE...FROM MASTER

LOCK TABLES

OPTIMIZE TABLE

PURGE MASTER LOGS

RENAME TABLE

REPAIR TABLE

REPLACE

RESET

RESET MASTER

RESET SLAVE

RESTORE TABLE

REVOKE

ROLLBACK

ROLLBACK TO SAVEPOINT

SAVEPOINT

SELECT

SET

SET PASSWORD

SET SQL_LOG_BIN

SET TRANSACTION

SHOW BINLOG EVENTS

SHOW CHARACTER SET

SHOW COLLATION

SHOW COLUMNS

SHOW CREATE DATABASE

SHOW CREATE TABLE

SHOW CREATE VIEW

SHOW DATABASES

SHOW ENGINES

SHOW ERRORS

SHOW GRANTS

SHOW INDEX

SHOW INNODB STATUS

SHOW LOGS

SHOW MASTER LOGS

SHOW MASTER STATUS

SHOW PRIVILEGES

SHOW PROCESSLIST

SHOW SLAVE HOSTS

SHOW SLAVE STATUS

SHOW STATUS

SHOW TABLE STATUS

SHOW TABLES

SHOW VARIABLES

SHOW WARNINGS

START SLAVE

START TRANSACTION

STOP SLAVE

TRUNCATE TABLE

UNION

UNLOCK TABLES

USE

String Functions
AES_DECRYPT

AES_ENCRYPT

ASCII

BIN

BINARY

BIT_LENGTH

CHAR

CHAR_LENGTH

CHARACTER_LENGTH

COMPRESS

CONCAT

CONCAT_WS

CONV

DECODE

DES_DECRYPT

DES_ENCRYPT

ELT

ENCODE

ENCRYPT

EXPORT_SET

FIELD

FIND_IN_SET

HEX

INET_ATON

INET_NTOA

INSERT

INSTR

LCASE

LEFT

LENGTH

LOAD_FILE

LOCATE

LOWER

LPAD

LTRIM

MAKE_SET

MATCH    AGAINST

MD5

MID

OCT

OCTET_LENGTH

OLD_PASSWORD

ORD

PASSWORD

POSITION

QUOTE

REPEAT

REPLACE

REVERSE

RIGHT

RPAD

RTRIM

SHA

SHA1

SOUNDEX

SPACE

STRCMP

SUBSTRING

SUBSTRING_INDEX

TRIM

UCASE

UNCOMPRESS

UNCOMPRESSED_LENGTH

UNHEX

UPPER

Date and Time Functions
ADDDATE

ADDTIME

CONVERT_TZ

CURDATE

CURRENT_DATE

CURRENT_TIME

CURRENT_TIMESTAMP

CURTIME

DATE

DATE_ADD

DATE_FORMAT

DATE_SUB

DATEDIFF

DAY

DAYNAME

DAYOFMONTH

DAYOFWEEK

DAYOFYEAR

EXTRACT

FROM_DAYS

FROM_UNIXTIME

GET_FORMAT

HOUR

LAST_DAY

LOCALTIME

LOCALTIMESTAMP

MAKEDATE

MAKETIME

MICROSECOND

MINUTE

MONTH

MONTHNAME

NOW

PERIOD_ADD

PERIOD_DIFF

QUARTER

SEC_TO_TIME

SECOND

STR_TO_DATE

SUBDATE

SUBTIME

SYSDATE

TIME

TIMEDIFF

TIMESTAMP

TIMESTAMPDIFF

TIMESTAMPADD

TIME_FORMAT

TIME_TO_SEC

TO_DAYS

UNIX_TIMESTAMP

UTC_DATE

UTC_TIME

UTC_TIMESTAMP

WEEK

WEEKDAY

WEEKOFYEAR

YEAR

YEARWEEK

Mathematical and Aggregate Functions
ABS

ACOS

ASIN

ATAN

ATAN2

AVG

BIT_AND

BIT_OR

BIT_XOR

CEIL

CEILING

COS

COT

COUNT

CRC32

DEGREES

EXP

FLOOR

FORMAT

GREATEST

GROUP_CONCAT

LEAST

LN

LOG

LOG2

LOG10

MAX

MIN

MOD

PI

POW

POWER

RADIANS

RAND

ROUND

SIGN

SIN

SQRT

STD

STDDEV

SUM

TAN

TRUNCATE

VARIANCE

Flow Control Functions
CASE

IF

IFNULL

NULLIF

Command-Line Utilities
comp_err

isamchk

make_binary_distribution

msql2mysql

my_print_defaults

myisamchk

myisamlog

myisampack

mysqlaccess

mysqladmin

mysqlbinlog

mysqlbug

mysqlcheck

mysqldump

mysqldumpslow

mysqlhotcopy

mysqlimport

mysqlshow

perror

Perl API - using functions and methods built into the Perl DBI with MySQL
available_drivers

begin_work

bind_col

bind_columns

bind_param

bind_param_array

bind_param_inout

can

clone

column_info

commit

connect

connect_cached

data_sources

disconnect

do

dump_results

err

errstr

execute

execute_array

execute_for_fetch

fetch

fetchall_arrayref

fetchall_hashref

fetchrow_array

fetchrow_arrayref

fetchrow_hashref

finish

foreign_key_info

func

get_info

installed_versions


last_insert_id

looks_like_number

neat

neat_list

parse_dsn

parse_trace_flag

parse_trace_flags

ping

prepare

prepare_cached

primary_key

primary_key_info

quote

quote_identifier

rollback

rows

selectall_arrayref

selectall_hashref

selectcol_arrayref

selectrow_array

selectrow_arrayref

selectrow_hashref

set_err

state

table_info

table_info_all

tables

trace

trace_msg

type_info

type_info_all

Attributes for Handles

PHP API - using functions built into PHP with MySQL
mysql_affected_rows

mysql_change_user

mysql_client_encoding

mysql_close

mysql_connect

mysql_create_db

mysql_data_seek

mysql_db_name

mysql_db_query

mysql_drop_db

mysql_errno

mysql_error

mysql_escape_string

mysql_fetch_array

mysql_fetch_assoc

mysql_fetch_field

mysql_fetch_lengths

mysql_fetch_object

mysql_fetch_row

mysql_field_flags

mysql_field_len

mysql_field_name

mysql_field_seek

mysql_field_table

mysql_field_type

mysql_free_result

mysql_get_client_info

mysql_get_host_info

mysql_get_proto_info

mysql_get_server_info

mysql_info

mysql_insert_id

mysql_list_dbs

mysql_list_fields

mysql_list_processes

mysql_list_tables

mysql_num_fields

mysql_num_rows

mysql_pconnect

mysql_ping

mysql_query

mysql_real_escape_string

mysql_result

mysql_select_db

mysql_stat

mysql_tablename

mysql_thread_id

mysql_unbuffered_query