Run command ‘free -m’ note value under ‘-/+ buffers/cache’ and find out how much memory your vps is currently using ? After that please proceed the following steps:

1) Stopping SpamAssassin:

This program isn’t overly important to the stopping of spam. To turn it off do the following:

Go to WHM » Server Configuration » Tweak Settings » Disable Spamd

And:

Go to WHM » Service Manager » Disable Spamd

2) Stopping Mailman:

If you or none of your customers aren’t running any mail listings through cPanel, it would be a good idea to disable Mailman.

Go to WHM » Server Configuration » Tweak Settings » Disable Mailman

3) Stop ClamAV:

Never use ClamAV on a VPS server.You can use maldet instead of ClamAV.You can disable ClamAV from:

Go to WHM » Home » cPanel » Manage Plugins » Uncheck clamavconnector and save

4) Never Use Tomcat on a VPS:

Never run TomCat on a VPS because of it’s high RAM needs.

5) Disable CatchAll :

If you receive incoming spam mail to a large extend then catchall setup will be reason.Try to catch-all from :

Go to WHM » Server Configuration » Tweak Settings » Change to :fail:

6) Tweak Apache:

Apache need to tweaked compulsory to reduce memory usage, do the following:

WHM » Home » Service Configuration » Apache Configuration » Global Configuration

Change the following settings:

MinSpareServers 2
MaxSpareServers 5
StartServers 3
Keep-Alive Off

Save the values and rebuild apache.

7) Optimize MySQL.

Please replace your existing my.cnf to the following:

[mysqld]
local-infile=0
max_allowed_packet = 1M
max_connections = 300
max_user_connections = 35
wait_timeout=40
connect_timeout=10
[mysqldump]
quick max_allowed_packet = 16M

8)InnoDB

InnoDB requires more RAM and it in not good for a VPS server. Better prefer MyISAM tables.So i suggest you to convert InnoDB tables to MyISAM.

9) Optimize PHP

Reduce PHP memory limit from php.ini.

10) Backup Type :

Never use compressed backup type.Instead of compress backup use incremental backup.You can configure it from WHM » Home » Backup » Backup Configuration

11) Disable Unwanted Services.

pcscd
portmap
netfs
nfslock
rpcidmapd
yum-updatesd
avahi-daemon
autofs
acpid
atd
gpm
haldaemon
hidd
irqbalance
auditd
xfs
bluetooth
anacron
pcmcia
sgi_fam
rhnsd
hald
hcid or sdpd
hpiod or hpssd.py
dbus-daemon
cupsd

All that should reduce your memory usage by a large amount.

Please reboot your VPS for one time.After server is up run ‘free -m’ command and feel the difference in memory usage.