This will not cover every part of server hardening, since this would be way too long.
Let´s start with the Serverpart:
1.) In your php.ini disable dangerous functions. Here is a complete list:
apache_child_terminate apache_setenv define_syslog_variables escapeshellarg escapeshellcmd eval exec fp fput ftp_connect ftp_exec ftp_get ftp_login ftp_nb_fput ftp_put ftp_raw ftp_rawlist highlight_file ini_alter ini_get_all ini_restore inject_code mysql_pconnect openlog passthru php_uname phpAds_remoteInfo phpAds_XmlRpc phpAds_xmlrpcDecode phpAds_xmlrpcEncode popen posix_getpwuid posix_kill posix_mkfifo posix_setpgid posix_setsid posix_setuid posix_setuid posix_uname proc_close proc_get_status proc_nice proc_open proc_terminate shell_exec syslog system xmlrpc_entity_decode2.) If you use a SSL certificate (which I recommend), you should enable PHP Secure Flags.
3.) Don´t set the max_post_size too high, it can use alot of ressources and lead to a DoS attack.
4.) Always use the latest PHP version available on the standard repository. (Non standard repositories can include malicious code)
5.) I assume you use Apache - so first check which modules apache has enabled. Use apache2ctl -M | sort to display those on a Debian based system (I didn´t lookup other OS yet).
Once done, disable all modules which you don´t need - but make sure you know what you disable, it can cause real damage.
6.) If you use an SSL Certificate make sure you disable port 80 and run apache on port 443 only.
7.) Use .htaccess Basic Authentication to secure sensitive parts of your server, i. e. phpmyadmin or your adminpanel for IPB.
8.) Change your SSH Port to a non standard one and install a software like fail2ban to limit the request to specified port. - I recommend limiting the login to one IP only, but some of you might use a dynamic IP.
9.) Do not allow external MySQL logins, also disable anonymous logins!
10.) Let a security researcher/pentester check your server for issues, trust me, they can always help make things better.
Let´s go to the Forumpart:
1.) Did you look at the Security Center in your ACP? Did you really follow every step? I recommend to do so.
2.) Create a whitelist for image hosting sites. External pages can cause real damage/trouble by enabling basic authentications and logging usernames/passwords or by simply logging IP Adresses.
3.) If you can, restrict your ACP by IP (which can be done with code editing or server configuration / .htaccess). This adds an extra layer of security.
4.) If you have external untrusted javascript included in your script, save it locally and serve it from there, it can always happen that someone changes his javascript to log keystrokes.
5.) If you use SSL, use https versions of links included in your IPB Design.
6.) Don´t install plugins/applications from 3rd party sites, they might be backdoored.
7.) Check your forum directory for unusual files/filesizes and check weekly if you can find functions like eval() or system() in your files, since those usually indicate that someone uploaded a shell, or the code of your application is unusual.
8.) Do not host other software on your server, always seperate your software unless you want the data of all your applications to be leaked.
9.) Choose strong passwords for your users, unfortunately there are still people using 123456 as passwords.
10.) Always upgrade to the newest version of your Software!
Now to the weakest part in this chain:
You! - Trust me, in 75% of the cases, full access was provided by the admin himself. Thats caused by Social Engineering tactics used to fool admins into giving out info. Please never give out info unless it is absolutely necessary.