PHP fatal error. Causes Dashboard only to load 68%

My Dashboard gets to 68% and stops then i get an error message is /var/log/httpd/error_log

Here is an output of fwconsole ma list and fwconsole dbug

[[email protected] httpd]# fwconsole ma list
No repos specified, using: [commercial,standard,extended,unsupported] from last GUI settings

±---------------------±-----------±----------------------------------±-----------+
| Module | Version | Status | License |
±---------------------±-----------±----------------------------------±-----------+
| accountcodepreserve | 13.0.2 | Enabled | GPLv2 |
| announcement | 13.0.6 | Enabled | GPLv3+ |
| areminder | 2.11.6 | Disabled | Commercial |
| arimanager | | Not Installed (Locally available) | GPLv3+ |
| asterisk-cli | 13.0.4 | Enabled | GPLv3+ |
| asteriskinfo | 13.0.7 | Enabled | GPLv3+ |
| backup | 13.0.25 | Enabled | GPLv3+ |
| blacklist | 13.0.14 | Enabled | GPLv3+ |
| bria | 2.11.0.14 | Disabled | Commercial |
| broadcast | 2.11.7 | Disabled | Commercial |
| builtin | | Enabled | |
| bulkdids | 2.11.1.4 | Enabled | GPLv3+ |
| bulkextensions | 2.11.0.7 | Enabled | GPLv3+ |
| bulkhandler | 13.0.13 | Enabled | GPLv3+ |
| callback | 2.11.0.4 | Disabled | GPLv3+ |
| callerid | 2.11.0.7 | Disabled | Commercial |
| callforward | 13.0.4 | Enabled | AGPLv3+ |
| calllimit | 13.0.3 | Disabled | Commercial |
| callrecording | 13.0.11 | Enabled | AGPLv3+ |
| callwaiting | 12.0.3 | Disabled | GPLv3+ |
| campon | 2.11.0.2 | Disabled | GPLv3+ |
| cdr | 13.0.29.8 | Enabled | GPLv3+ |
| cel | 13.0.25 | Enabled | GPLv3+ |
| certman | 13.0.34 | Enabled | AGPLv3+ |
| cidlookup | 2.11.1.11 | Disabled | GPLv3+ |
| conferences | 13.0.22 | Enabled | GPLv3+ |
| conferencespro | 13.0.25 | Disabled | Commercial |
| configedit | 13.0.7 | Disabled | AGPLv3+ |
| contactmanager | 13.0.33 | Enabled | GPLv3+ |
| core | 13.0.112 | Enabled | GPLv3+ |
| cos | 13.0.9 | Disabled | Commercial |
| customappsreg | 13.0.5 | Enabled | GPLv3+ |
| cxpanel | 4.1.18 | Disabled | GPLv3 |
| dahdiconfig | 2.11.47 | Disabled | GPLv3+ |
| dashboard | 13.0.24 | Enabled | AGPLv3+ |
| daynight | 13.0.13 | Enabled | GPLv3+ |
| dictate | 2.11.0.3 | Disabled | GPLv3+ |
| digium_phones | 2.11.2.1 | Disabled | GPLv2 |
| digiumaddoninstaller | 2.11.0.10 | Disabled | GPLv2 |
| directory | 13.0.16 | Enabled | GPLv3+ |
| disa | 12.0.2 | Disabled | AGPLv3+ |
| donotdisturb | 13.0.3 | Enabled | GPLv3+ |
| dundicheck | 2.11.0.3 | Disabled | GPLv3+ |
| endpoint | 13.0.84.3 | Enabled | Commercial |
| extensionroutes | | Not Installed (Locally available) | Commercial |
| extensionsettings | 13.0.4 | Enabled | GPLv3+ |
| fax | 12.0.7 | Disabled | GPLv3+ |
| faxpro | 12.0.33 | Disabled | Commercial |
| featurecodeadmin | 13.0.6 | Enabled | GPLv3+ |
| findmefollow | 13.0.37 | Enabled | GPLv3+ |
| framework | 13.0.179 | Enabled | GPLv2+ |
| freepbx_ha | 12.0.2.1 | Disabled | Commercial |
| fw_langpacks | 12.0.7 | Enabled | GPLv3+ |
| hotelwakeup | 2.11.3 | Disabled | GPLv2 |
| iaxsettings | 13.0.5 | Enabled | AGPLv3 |
| infoservices | 13.0.1 | Enabled | GPLv2+ |
| irc | 2.11.0.4 | Disabled | GPLv3+ |
| ivr | 13.0.25 | Enabled | GPLv3+ |
| javassh | 2.11.2 | Disabled | AGPLv3+ |
| languages | 2.11.0.2 | Disabled | GPLv3+ |
| logfiles | 13.0.10 | Enabled | GPLv3+ |
| manager | 13.0.2.5 | Enabled | GPLv2+ |
| miscapps | 13.0.2.4 | Enabled | GPLv3+ |
| miscdests | 13.0.4 | Enabled | GPLv3+ |
| music | 13.0.22 | Enabled | GPLv3+ |
| outroutemsg | 13.0.2 | Enabled | GPLv3+ |
| paging | 13.0.24 | Enabled | GPLv3+ |
| pagingpro | 13.0.17 | Disabled | Commercial |
| parking | 13.0.16 | Enabled | GPLv3+ |
| parkpro | 12.0.5 | Disabled | Commercial |
| pbdirectory | 2.11.0.5 | Enabled | GPLv3+ |
| phonebook | 13.0.5.5 | Enabled | GPLv3+ |
| phpagiconf | 2.11.0.2 | Enabled | GPLv3+ |
| phpinfo | 13.0.2 | Enabled | GPLv2+ |
| pinsets | 2.11.0.9 | Disabled | GPLv3+ |
| pinsetspro | 2.11.0.11 | Disabled | Commercial |
| presencestate | 13.0.7 | Enabled | GPLv3+ |
| printextensions | 13.0.3 | Enabled | GPLv3+ |
| queueprio | 13.0.2 | Disabled | GPLv3+ |
| queues | 13.0.30 | Enabled | GPLv2+ |
| qxact_reports | 2.11.0.5 | Disabled | Commercial |
| recording_report | 2.11.0.9 | Disabled | Commercial |
| recordings | 13.0.28 | Enabled | GPLv3+ |
| restapi | 13.0.19 | Enabled | AGPLv3 |
| restapps | 13.0.76 | Enabled | Commercial |
| ringgroups | 13.0.21 | Enabled | GPLv3+ |
| setcid | 2.11.0.4 | Disabled | GPLv3+ |
| sipsettings | 13.0.23.7 | Enabled | AGPLv3+ |
| sipstation | 13.0.13.13 | Disabled | Commercial |
| sms | 13.0.9 | Disabled | Commercial |
| sng_mcu | 13.0.5 | Disabled | Commercial |
| soundlang | 13.0.17 | Enabled | GPLv3+ |
| speeddial | 2.11.0.4 | Enabled | GPLv3+ |
| superfecta | 13.0.3.18 | Disabled | GPLv2+ |
| sysadmin | 13.0.66 | Enabled | Commercial |
| timeconditions | 13.0.32 | Enabled | GPLv3+ |
| tts | 13.0.8 | Disabled | GPLv3+ |
| ttsengines | 13.0.6 | Disabled | AGPLv3 |
| ucp | 13.0.41 | Enabled | AGPLv3+ |
| ucpnode | 13.0.31 | Enabled | Commercial |
| userman | 13.0.72 | Enabled | AGPLv3+ |
| vmblast | 13.0.8 | Enabled | GPLv3+ |
| vmnotify | 13.0.15 | Disabled | Commercial |
| voicemail | 13.0.53 | Enabled | GPLv3+ |
| voicemail_report | 13.0.12 | Disabled | Commercial |
| vqplus | 13.0.24 | Disabled | Commercial |
| weakpasswords | 12.0.3 | Disabled | GPLv3+ |
| webcallback | 13.0.10 | Disabled | Commercial |
| webrtc | 13.0.28 | Disabled | GPLv3+ |
| xmpp | 13.0.14 | Disabled | AGPLv3 |
| zulu | 13.0.43 | Disabled | Commercial |
±---------------------±-----------±----------------------------------±-----------+
[[email protected] httpd]# fwconsole dbug
±----------------------+
| FreePBX Notifications |
±----------------------+

hotelwakeup (Cur v. 2.11.3) should be upgraded to v. 13.0.15 to fix security issues: SEC-2016-005

OUT > ==> /var/log/asterisk/freepbx_dbug <==

2016-Aug-16 19:04:09 /var/www/html/admin/libraries/modulefunctions.class.php:181
[NOTICE]: Undefined index: highreleasetrackver

2016-Aug-16 19:04:09 /var/www/html/admin/libraries/modulefunctions.class.php:1366
[NOTICE]: fopen(): Content-type not specified assuming application/x-www-form-urlencoded

2016-Aug-16 19:04:11 /var/www/html/admin/libraries/cronmanager.class.php:92
[NOTICE]: Undefined variable: db

==> /var/log/httpd/error_log <==
chmod: cannot operate on dangling symlink /var/lib/asterisk/bin/freepbx_engine_hook_restapps' chmod: cannot operate on dangling symlink/var/lib/asterisk/bin/freepbx_engine_hook_ucp’
chmod: cannot operate on dangling symlink /var/lib/asterisk/bin/freepbx_engine_hook_vqplus' chmod: cannot operate on dangling symlink/var/lib/asterisk/bin/freepbx_engine_hook_xmpp’
chmod: cannot operate on dangling symlink /var/lib/asterisk/bin/getservices' chmod: cannot operate on dangling symlink/var/lib/asterisk/bin/pagepro-startup.php’
[Wed Sep 07 14:27:06 2016] [error] [client 111.111.111.111] PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 32 bytes) in /var/www/html/admin/libraries/BMO/DB_Helper.class.php on line 158, referer: http://205.185.158.159/admin/config.php?display=index
[Wed Sep 07 14:27:54 2016] [error] [client 111.111.111.111] PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 110 bytes) in /var/www/html/admin/libraries/BMO/DB_Helper.class.php on line 158, referer: http://205.185.158.159/admin/config.php?display=index
[Wed Sep 07 14:39:11 2016] [error] [client 111.111.111.111] PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 111 bytes) in /var/www/html/admin/libraries/BMO/DB_Helper.class.php on line 158, referer: http://205.185.158.159/admin/config.php?display=index
[Wed Sep 07 14:47:09 2016] [error] [client 111.111.111.111] PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 9 bytes) in /var/www/html/admin/libraries/BMO/DB_Helper.class.php on line 158, referer: http://205.185.158.159/admin/config.php?display=index

==> /var/log/asterisk/freepbx_security.log <==
[2015-02-03 08:24:13] Authentication failure for admin from 111.111.111.112
[2016-04-26 09:56:47] Authentication failure for trussinic from 111.111.111.112
[2016-04-26 09:57:49] Authentication failure for trussinc from 111.111.111.112
[2016-04-26 09:57:55] Authentication failure for trussinc from 111.111.111.112
[2016-06-14 07:27:04] Authentication failure for trussinc from 111.111.111.112
[2016-06-14 07:27:26] Authentication failure for trussinc from 111.111.111.112
[2016-06-14 07:27:42] Authentication failure for trussinc from 111.111.111.112
[2016-06-14 07:32:26] Authentication failure for trussinc from 111.111.111.112
[2016-08-17 08:35:05] Authentication failure for admin from 111.111.111.112
[2016-08-25 15:56:10] Authentication failure for jeffs from 111.111.111.112

==> /var/log/asterisk/freepbx.log <==
)
[2016-Sep-07 14:45:56] [WARNING] (core/functions.inc.php:6441) - Depreciated Function core_users_list detected in /var/www/html/admin/libraries/modulefunctions.class.php on line 2893
[2016-Sep-07 14:45:58] [SECURITY] (BMO/Notifications.class.php:487) - [NOTIFICATION]-[freepbx]-[VULNERABILITIES] - There is 1 module vulnerable to security threats (hotelwakeup (Cur v. 2.11.3) should be upgraded to v. 13.0.15 to fix security issues: SEC-2016-005
)
[2016-Sep-07 14:45:58] [INFO] (extensionroutes/uninstall.php:10) - dropping table extensionroutes…done
[2016-Sep-07 14:45:58] [WARNING] (core/functions.inc.php:6441) - Depreciated Function core_users_list detected in /var/www/html/admin/modules/extensionroutes/uninstall.php on line 15
[2016-Sep-07 14:45:58] [INFO] (extensionroutes/uninstall.php:21) - removing AMPUSER outroute field…done
[2016-Sep-07 14:46:02] [WARNING] (core/functions.inc.php:6441) - Depreciated Function core_users_list detected in /var/www/html/admin/libraries/modulefunctions.class.php on line 2893
[2016-Sep-07 14:46:04] [SECURITY] (BMO/Notifications.class.php:487) - [NOTIFICATION]-[freepbx]-[VULNERABILITIES] - There is 1 module vulnerable to security threats (hotelwakeup (Cur v. 2.11.3) should be upgraded to v. 13.0.15 to fix security issues: SEC-2016-005
)
OUT >
==> /var/log/httpd/error_log <==
[Wed Sep 07 14:50:39 2016] [error] [client 208.74.240.253] PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 9 bytes) in /var/www/html/admin/libraries/BMO/DB_Helper.class.php on line 158, referer: http://111.111.111.111/admin/config.php?display=index
OUT > [Wed Sep 07 14:52:40 2016] [error] [client 208.74.240.254] PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 111 bytes) in /var/www/html/admin/libraries/BMO/DB_Helper.class.php on line 158, referer: http://111.111.111.111/admin/config.php
OUT > [Wed Sep 07 14:54:20 2016] [error] [client 111.111.111.112] PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 9 bytes) in /var/www/html/admin/libraries/BMO/DB_Helper.class.php on line 158, referer: http://111.111.111.111/admin/config.php?display=index
^C

You should upgrade everything, as it says.

fwconsole ma upgradeall

1 Like

hotelwakeup is disabled i don’t use it i ran fwconsole ma upgradeall and i ran module updates from the web interface already.

Are things still playing up with the latest of everything?

I had rebooted and then asterisk wouldn’t start.

It was after hours so i restored from a snapshot prior to making changes.

Today i upgraded the memory on the VM and rebooted things are ok but Dashboard is SLOW to load.

[Mon Oct 03 17:57:05 2016] [error] [client 1.2.2.4] PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 101 bytes) in /var/www/html/admin/libraries/BMO/DB_Helper.class.php on line 158, referer: http://CompanyPBX.Domain.com/admin/config.php?display=index

That displays every time i try to load the dashboard.

PBX httpd]# php -version
PHP 5.3.28 (cli) (built: Mar 31 2014 19:09:21)
Copyright © 1997-2013 The PHP Group
Zend Engine v2.3.0, Copyright © 1998-2013 Zend Technologies
with Zend Guard Loader v3.3, Copyright © 1998-2010, by Zend Technologies

Still getting failures

You are running out of memory in php. Go increase your memory in php.ini

I was thinking more of a php leak but i edited /etc/php.ini

Previous settting

memory_limit = 512M

New settting
memory_limit = 1024M

service httpd restart

PBX BMO]# tail -f /var/log/httpd/error_log
[Tue Oct 04 08:44:36 2016] [notice] caught SIGTERM, shutting down
[Tue Oct 04 08:44:37 2016] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue Oct 04 08:44:37 2016] [notice] Digest: generating secret for digest authentication …
[Tue Oct 04 08:44:37 2016] [notice] Digest: done
[Tue Oct 04 08:44:37 2016] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.28 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips configured – resuming normal operations

Now i refresh the dashboard and…

Loading 68%

Now it has 1 GB of memory to exhaust so it will take some time. But here you go…

[Tue Oct 04 08:47:00 2016] [error] [client 10.1.1.54] PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 111 bytes) in /var/www/html/admin/libraries/BMO/DB_Helper.class.php on line 158, referer: http://CompanyPBX.Domain.com/admin/config.php?display=index

Any idea as to what is running amock in php?

[Tue Oct 04 08:53:49 2016] [error] [client 10.1.1.54] PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 111 bytes) in /var/www/html/admin/libraries/BMO/DB_Helper.class.php on line 158, referer: http://CompanyPBX.Domain.com/admin/config.php?display=index

So this looks off to me.

mysql> select count() from kvstore limit 5;
±---------+
| count(
) |
±---------+
| 506770 |
±---------+
1 row in set (0.00 sec)

Two of the other servers have a count of less than 500

ideas???

Tony Lewis???

We did a mysql dump of the asterisk.kvstore… 4 GB!!!

Then truncated with the following command TRUNCATE kvstore;

Now it’s running like champ.

I lost historical info in the graphs but for no php exhaustion i’ll take it

One thing i noticed

Summary
Asterisk
MySQL
Web Server
Fail2Ban
System Registration
SysInfo updated 11 seconds ago

System Alerts

I’m not seeing RestApps show up there?