PHP fatal error. Causes Dashboard only to load 68%

juesor · September 7, 2016, 8:03pm

My Dashboard gets to 68% and stops then i get an error message is /var/log/httpd/error_log

Here is an output of fwconsole ma list and fwconsole dbug

[root@AnotherNewPBX httpd]# fwconsole ma list
No repos specified, using: [commercial,standard,extended,unsupported] from last GUI settings

hotelwakeup (Cur v. 2.11.3) should be upgraded to v. 13.0.15 to fix security issues: SEC-2016-005

OUT > ==> /var/log/asterisk/freepbx_dbug <==

2016-Aug-16 19:04:09 /var/www/html/admin/libraries/modulefunctions.class.php:181
[NOTICE]: Undefined index: highreleasetrackver

2016-Aug-16 19:04:09 /var/www/html/admin/libraries/modulefunctions.class.php:1366
[NOTICE]: fopen(): Content-type not specified assuming application/x-www-form-urlencoded

2016-Aug-16 19:04:11 /var/www/html/admin/libraries/cronmanager.class.php:92
[NOTICE]: Undefined variable: db

==> /var/log/httpd/error_log <==
chmod: cannot operate on dangling symlink /var/lib/asterisk/bin/freepbx_engine_hook_restapps' chmod: cannot operate on dangling symlink/var/lib/asterisk/bin/freepbx_engine_hook_ucp’
chmod: cannot operate on dangling symlink /var/lib/asterisk/bin/freepbx_engine_hook_vqplus' chmod: cannot operate on dangling symlink/var/lib/asterisk/bin/freepbx_engine_hook_xmpp’
chmod: cannot operate on dangling symlink /var/lib/asterisk/bin/getservices' chmod: cannot operate on dangling symlink/var/lib/asterisk/bin/pagepro-startup.php’
[Wed Sep 07 14:27:06 2016] [error] [client 111.111.111.111] PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 32 bytes) in /var/www/html/admin/libraries/BMO/DB_Helper.class.php on line 158, referer: http://205.185.158.159/admin/config.php?display=index
[Wed Sep 07 14:27:54 2016] [error] [client 111.111.111.111] PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 110 bytes) in /var/www/html/admin/libraries/BMO/DB_Helper.class.php on line 158, referer: http://205.185.158.159/admin/config.php?display=index
[Wed Sep 07 14:39:11 2016] [error] [client 111.111.111.111] PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 111 bytes) in /var/www/html/admin/libraries/BMO/DB_Helper.class.php on line 158, referer: http://205.185.158.159/admin/config.php?display=index
[Wed Sep 07 14:47:09 2016] [error] [client 111.111.111.111] PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 9 bytes) in /var/www/html/admin/libraries/BMO/DB_Helper.class.php on line 158, referer: http://205.185.158.159/admin/config.php?display=index

==> /var/log/asterisk/freepbx_security.log <==
[2015-02-03 08:24:13] Authentication failure for admin from 111.111.111.112
[2016-04-26 09:56:47] Authentication failure for trussinic from 111.111.111.112
[2016-04-26 09:57:49] Authentication failure for trussinc from 111.111.111.112
[2016-04-26 09:57:55] Authentication failure for trussinc from 111.111.111.112
[2016-06-14 07:27:04] Authentication failure for trussinc from 111.111.111.112
[2016-06-14 07:27:26] Authentication failure for trussinc from 111.111.111.112
[2016-06-14 07:27:42] Authentication failure for trussinc from 111.111.111.112
[2016-06-14 07:32:26] Authentication failure for trussinc from 111.111.111.112
[2016-08-17 08:35:05] Authentication failure for admin from 111.111.111.112
[2016-08-25 15:56:10] Authentication failure for jeffs from 111.111.111.112

==> /var/log/asterisk/freepbx.log <==
)
[2016-Sep-07 14:45:56] [WARNING] (core/functions.inc.php:6441) - Depreciated Function core_users_list detected in /var/www/html/admin/libraries/modulefunctions.class.php on line 2893
[2016-Sep-07 14:45:58] [SECURITY] (BMO/Notifications.class.php:487) - [NOTIFICATION]-[freepbx]-[VULNERABILITIES] - There is 1 module vulnerable to security threats (hotelwakeup (Cur v. 2.11.3) should be upgraded to v. 13.0.15 to fix security issues: SEC-2016-005
)
[2016-Sep-07 14:45:58] [INFO] (extensionroutes/uninstall.php:10) - dropping table extensionroutes…done
[2016-Sep-07 14:45:58] [WARNING] (core/functions.inc.php:6441) - Depreciated Function core_users_list detected in /var/www/html/admin/modules/extensionroutes/uninstall.php on line 15
[2016-Sep-07 14:45:58] [INFO] (extensionroutes/uninstall.php:21) - removing AMPUSER outroute field…done
[2016-Sep-07 14:46:02] [WARNING] (core/functions.inc.php:6441) - Depreciated Function core_users_list detected in /var/www/html/admin/libraries/modulefunctions.class.php on line 2893
[2016-Sep-07 14:46:04] [SECURITY] (BMO/Notifications.class.php:487) - [NOTIFICATION]-[freepbx]-[VULNERABILITIES] - There is 1 module vulnerable to security threats (hotelwakeup (Cur v. 2.11.3) should be upgraded to v. 13.0.15 to fix security issues: SEC-2016-005
)
OUT >
==> /var/log/httpd/error_log <==
[Wed Sep 07 14:50:39 2016] [error] [client 208.74.240.253] PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 9 bytes) in /var/www/html/admin/libraries/BMO/DB_Helper.class.php on line 158, referer: http://111.111.111.111/admin/config.php?display=index
OUT > [Wed Sep 07 14:52:40 2016] [error] [client 208.74.240.254] PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 111 bytes) in /var/www/html/admin/libraries/BMO/DB_Helper.class.php on line 158, referer: http://111.111.111.111/admin/config.php
OUT > [Wed Sep 07 14:54:20 2016] [error] [client 111.111.111.112] PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 9 bytes) in /var/www/html/admin/libraries/BMO/DB_Helper.class.php on line 158, referer: http://111.111.111.111/admin/config.php?display=index
^C

xrobau · September 7, 2016, 10:00pm

You should upgrade everything, as it says.

fwconsole ma upgradeall

juesor · September 8, 2016, 2:54pm

hotelwakeup is disabled i don’t use it i ran fwconsole ma upgradeall and i ran module updates from the web interface already.

xrobau · September 8, 2016, 2:56pm

Are things still playing up with the latest of everything?

juesor · October 3, 2016, 10:44pm

I had rebooted and then asterisk wouldn’t start.

It was after hours so i restored from a snapshot prior to making changes.

Today i upgraded the memory on the VM and rebooted things are ok but Dashboard is SLOW to load.

juesor · October 3, 2016, 10:58pm

[Mon Oct 03 17:57:05 2016] [error] [client 1.2.2.4] PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 101 bytes) in /var/www/html/admin/libraries/BMO/DB_Helper.class.php on line 158, referer: http://CompanyPBX.Domain.com/admin/config.php?display=index

That displays every time i try to load the dashboard.

juesor · October 3, 2016, 10:59pm

PBX httpd]# php -version
PHP 5.3.28 (cli) (built: Mar 31 2014 19:09:21)
Copyright © 1997-2013 The PHP Group
Zend Engine v2.3.0, Copyright © 1998-2013 Zend Technologies
with Zend Guard Loader v3.3, Copyright © 1998-2010, by Zend Technologies

juesor · October 4, 2016, 1:37pm

Still getting failures

tonyclewis · October 4, 2016, 1:42pm

You are running out of memory in php. Go increase your memory in php.ini

juesor · October 4, 2016, 1:47pm

I was thinking more of a php leak but i edited /etc/php.ini

Previous settting

memory_limit = 512M

New settting
memory_limit = 1024M

service httpd restart

PBX BMO]# tail -f /var/log/httpd/error_log
[Tue Oct 04 08:44:36 2016] [notice] caught SIGTERM, shutting down
[Tue Oct 04 08:44:37 2016] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue Oct 04 08:44:37 2016] [notice] Digest: generating secret for digest authentication …
[Tue Oct 04 08:44:37 2016] [notice] Digest: done
[Tue Oct 04 08:44:37 2016] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.28 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips configured – resuming normal operations

Now i refresh the dashboard and…

Loading 68%

Now it has 1 GB of memory to exhaust so it will take some time. But here you go…

[Tue Oct 04 08:47:00 2016] [error] [client 10.1.1.54] PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 111 bytes) in /var/www/html/admin/libraries/BMO/DB_Helper.class.php on line 158, referer: http://CompanyPBX.Domain.com/admin/config.php?display=index

juesor · October 4, 2016, 1:58pm

Any idea as to what is running amock in php?

[Tue Oct 04 08:53:49 2016] [error] [client 10.1.1.54] PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 111 bytes) in /var/www/html/admin/libraries/BMO/DB_Helper.class.php on line 158, referer: http://CompanyPBX.Domain.com/admin/config.php?display=index

juesor · October 4, 2016, 2:15pm

So this looks off to me.

mysql> select count() from kvstore limit 5;
±---------+
| count() |
±---------+
| 506770 |
±---------+
1 row in set (0.00 sec)

Two of the other servers have a count of less than 500

ideas???

Tony Lewis???

juesor · October 4, 2016, 2:34pm

We did a mysql dump of the asterisk.kvstore… 4 GB!!!

Then truncated with the following command TRUNCATE kvstore;

Now it’s running like champ.

I lost historical info in the graphs but for no php exhaustion i’ll take it

juesor · October 4, 2016, 2:35pm

One thing i noticed

Summary
Asterisk
MySQL
Web Server
Fail2Ban
System Registration
SysInfo updated 11 seconds ago

System Alerts

I’m not seeing RestApps show up there?