Latest System Admin (2.11.0.38) produce new strange PHP-WARNINGs regarding (unused) DDNS feature

FreePBX Distro 4.211.64-10 up to date with System Admin 2.11.0.38.

After that Module update I started to notice new recurring (hourly? not always) PHP-WARNINGs like the one reported below:

[2014-Jan-13 19:00:20] [PHP-WARNING] (/var/www/html/admin/modules/sysadmin/functions.inc/ddns.php:70) - file_get_contents(http://ddns2.schmoozecom.net/ddnsserver.php?server=NNNNNNNN&ver=4.211.64-10&fs=1.0.0.0&av=See+https%3A%2F%2Fwiki.asterisk.org%2Fwiki%2Fx%2F1gKfAQ+for+more+details.&kv=2.6.32-358.14.1.el6.x86_64&n=&e=14&st=X.X.X.X): failed to open stream: Connection timed out

where I manually replaced the Public IP Address of the system with X.X.X.X and the ID of the system (as per DDNS setting page, setting that weren’t touched since I installed the system months ago) with NNNNNNNN.

I also noticed a sort of slowness in opening the DDNS Settings page. Other menus of System Admin main menu open fast but once the DDNS menu is selected that web page takes a long time before opening and, while opening, it seems to refuse other navigation inputs.

For instance when I select the DDNS menu the short log sequence below is then generated into freepbx.log:

[2014-Jan-13 19:15:20] [PHP-NOTICE] (/var/www/html/admin/modules/endpoint/functions.inc/functions_common.php:1102) - Undefined index: action [2014-Jan-13 19:16:20] [PHP-WARNING] (/var/www/html/admin/modules/sysadmin/functions.inc/ddns.php:70) - file_get_contents(http://ddns2.schmoozecom.net/ddnsserver.php?server=NNNNNNNN&ver=4.211.64-10&fs=1.0.0.0&av=See+https%3A%2F%2Fwiki.asterisk.org%2Fwiki%2Fx%2F1gKfAQ+for+more+details.&kv=2.6.32-358.14.1.el6.x86_64&n=&e=14&st=): failed to open stream: Connection timed out [2014-Jan-13 19:16:22] [PHP-NOTICE] (/var/www/html/admin/libraries/view.functions.php:370) - Undefined variable: mod_version_tag

Look at 60 seconds to open the DDNS page (PHP-WARNING).
To note: the Public IP address (X.X.X.X) doesn’t always appear.

The DDNS service is not important to me (never used).

Any idea?

It looks like your system is having a hard time reaching our ddns server. Can you do an nslookup to ddns2.schmoozecom.net and let me know the results.

Here the result of nslookup against ddns2.schmoozecom.net (done Tue Jan 14 07:39:43 CET 2014):

Server: 127.0.0.1 Address: 127.0.0.1#53

Non-authoritative answer:
Name: ddns2.schmoozecom.net
Address: 199.102.239.146
Name: ddns2.schmoozecom.net
Address: 199.102.239.149

Latest PHP-WARNINGs’ timestamp about connection timing out (so not exactly hourly):

2014-Jan-14 03:40:02
2014-Jan-14 04:41:01
2014-Jan-14 05:19:02
2014-Jan-14 06:17:01

Below a dig against the same FQDN:

; <> DiG 9.8.2rc1-RedHat-9.8.2-0.17.rc1.el6_4.6 <> ddns2.schmoozecom.net ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 24268 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;ddns2.schmoozecom.net. IN A

;; ANSWER SECTION:
ddns2.schmoozecom.net. 2251 IN A 199.102.239.149
ddns2.schmoozecom.net. 2251 IN A 199.102.239.146

;; Query time: 1 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Tue Jan 14 07:42:31 2014
;; MSG SIZE rcvd: 71

Tracing route against .149 and .146 looks similar and I can see packets “jumping over” the Atlantic via seabone.net, cogentco.com and cyberlynk.net routers.

Internet glitches in your opinion?

What DNS servers are you using as 199.102.239.149 has not been used in months for ddns2

That’s quite strange, the FreePBX Distro points to a pfSense 2.1 Firewall in which the DNS order is (and was):

1st: 8.8.8.8
2nd: 8.8.4.4
3rd: 208.67.222.222
4th: 208.67.220.220

The Firewall has an uptime of 24 days only and the FreePBX Distro system just 23 days…so I think it couldn’t be something related to a DNS caching mechanism.

I’ll try to perform a nslookup from the Firewall. Done. The FQDN ddns2.schmoozecom.net resolves into 199.102.239.146 with a first query delay of:

161 ms (8.8.8.8)
41 ms (8.8.8.4)
41 ms (208.67.222.222)
41 ms (208.67.220.220)

Sequential DNS lookups have a lower average (40/45 ms) on all listed DNS servers (caching).

And just to be complete, below the /etc/resolv.conf:

#Configuration automatically generated via the PBXact Utility #DO NOT HAND MODIFY THIS FILE! #generated: Wed, 07 Aug 2013 10:12:39 +0200 nameserver 127.0.0.1 nameserver 192.168.0.200 search localdomain

last touched August, 7th 2013. Naturally 192.168.0.200 is the Default Gateway IP address (the pfSense Firewall).

Ok well now its showing you are just resolving the 199.102.239.146 which is correct. Wonder if your firewall had something cached. Can you try DDNS now?

Now the DDNS Setting web page navigation is pretty fast as usual…I didn’t touched the Firewall nor the FreePBX (just looked into and I don’t think a cache will ever survived a reboot 24 days ago) but…surprise! ehy! when I look now at System Admin page it reports my system (a FreePBX Distro 4.211.64-10) as a 5.211.65-1! Oh gosh that’s impossible…what’s happened? how is it possible that /etc/schmooze/pbx-version was (over)written (touched on 11 Jan 14 02:24)?!

That’s very strange…I updated Sysadmin Module on 10 Jan…so it’s not Sysadmin Module update relate…how could be possible that pbx-version file “changed” on 11 Jan?

What’s more interesting is that freepbx.log doesn’t report anything special between 02:00 and 03:00 on 11 Jan 14:

[2014-Jan-11 02:09:02] [PHP-WARNING] (/var/www/html/admin/modules/sysadmin/functions.inc/ddns.php:70) - file_get_contents(http://ddns2.schmoozecom.net/ddnsserver.php?server=NNNNNNNN&ver=4.211.64-10&fs=1.0.0.0&av=See+https%3A%2F%2Fwiki.asterisk.org%2Fwiki%2Fx%2F1gKfAQ+for+more+details.&kv=2.6.32-358.14.1.el6.x86_64&n=&e=14&st=X.X.X.X): failed to open stream: Connection timed out [2014-Jan-11 03:00:02] [PHP-NOTICE] (/var/www/html/admin/modules/sysadmin/functions.inc/storage.php:96) - Undefined offset: 1 [2014-Jan-11 03:00:02] [PHP-NOTICE] (/var/www/html/admin/modules/sysadmin/functions.inc/storage.php:97) - Undefined offset: 2 [2014-Jan-11 03:00:02] [PHP-NOTICE] (/var/www/html/admin/modules/sysadmin/functions.inc/storage.php:98) - Undefined offset: 3 [2014-Jan-11 03:00:02] [PHP-NOTICE] (/var/www/html/admin/modules/sysadmin/functions.inc/storage.php:99) - Undefined offset: 4 [2014-Jan-11 03:00:02] [PHP-NOTICE] (/var/www/html/admin/modules/sysadmin/functions.inc/storage.php:100) - Undefined offset: 5 [2014-Jan-11 03:11:02] [PHP-WARNING] (/var/www/html/admin/modules/sysadmin/functions.inc/ddns.php:70) - file_get_contents(http://ddns2.schmoozecom.net/ddnsserver.php?server=NNNNNNNN&ver=4.211.64-10&fs=1.0.0.0&av=See+https%3A%2F%2Fwiki.asterisk.org%2Fwiki%2Fx%2F1gKfAQ+for+more+details.&kv=2.6.32-358.14.1.el6.x86_64&n=&e=14&st=X.X.X.X): failed to open stream: Connection timed out

System (Public) IP and System ID removed.

Nothing special from the Full log too during the same period of time.

Well something changed it. No clue what but no modules in FreePBX ever change that version file. Only upgrade scripts do.

I did an echo “4.211.64-10” > /etc/schmooze/pbx-version to fix the mismatch but, still, I don’t understand how this (unwanted/unattended) change technically happened. Truly paranormal activities.

I’m 100% sure I not applied any upgrade/update script (latest was to update -9 to -10 time ago), just updated various modules by hand (via CLI). What is very strange is the timestamp of that change…I truly sleep at night so seems impossible to me perform a manual touch at 2 AM (local time) in the morning…

Oh Oh…looking at /var/log/pbx/upgrade folder:

-rw-r--r-- 1 root root 2.4K Jan 14 02:24 5.211.65-1

What’s that? I NEVER downloaded any upgrade script to 5.211.65-1!

And look here:

[[email protected] upgrade]# cat 5.211.65-1 Tue Jan 14 02:16:18 CET 2014 Check to make sure this is a FreePBX Distro system before executing Tue Jan 14 02:16:18 CET 2014 This appears to be a FreePBX Distro system as it has a Distro Version of 4.211.64-10 Tue Jan 14 02:16:18 CET 2014 Make sure this is version 4.211.64-9 or greater before executing Tue Jan 14 02:16:18 CET 2014 This appears to be FreePBX Distro version 4.211.64.10 so we can now upgrade you to the 5.211.65 track as version 5.211.65-1 VARIABLES SET FOR UPGRADE asterisk=Asterisk 11.7.0 built by root @ jenkins-el6-64.schmoozecom.net on a x86_64 running Linux on 2013-12-18 22:17:47 UTC Privilege escalation protection disabled! See https://wiki.asterisk.org/wiki/x/1gKfAQ for more details. kernel=Linux freepbx.shirt.internal 2.6.32-358.14.1.el6.x86_64 #1 SMP Tue Jul 16 23:51:20 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux version=4.211.64.10 brand=FreePBXDistro host=freepbx.shirt.internal upgradeversion=5.211.65-1 virtual=

Tue Jan 14 02:16:18 CET 2014 Your FreePBX Distro System is being upgraded to 5.211.65-1. Please standby…

Tue Jan 14 02:16:18 CET 2014 STAGE 1 STARTING - GUI Modules
Tue Jan 14 02:16:18 CET 2014 Upgrade All FreePBX GUI Modules
Tue Jan 14 02:16:39 CET 2014 STAGE 1 COMPLETED - GUI Modules - Moving to Stage 2

Tue Jan 14 02:16:40 CET 2014 STAGE 2 STARTING - RPM’s
Tue Jan 14 02:16:40 CET 2014 Replace repos with only FreePBX Distro since some people have added other repos which can break updates
Tue Jan 14 02:16:40 CET 2014 Cleanup old Kernels to make space in boot
Tue Jan 14 02:17:07 CET 2014 Updating all remaining RPMS now to Centos 6.5
Tue Jan 14 02:24:35 CET 2014 STAGE 2 COMPLETED - Misc Items - Moving to Stage 4

Tue Jan 14 02:24:35 CET 2014 STAGE 3 STARTING - Misc Items
Tue Jan 14 02:24:35 CET 2014 STAGE 3 COMPLETED - Misc Items - Moving to Stage 4

Tue Jan 14 02:24:35 CET 2014 STAGE 4 STARTING - Clean Up
Tue Jan 14 02:24:36 CET 2014 updatedb for locate command
Tue Jan 14 02:24:37 CET 2014 Restart incron to be safe
Tue Jan 14 02:24:37 CET 2014 STAGE 4 COMPLETED - Clean Up - Moving to Stage 5

Tue Jan 14 02:24:37 CET 2014 STAGE 5 STARTING - Final Verifications
Tue Jan 14 02:24:37 CET 2014 STAGE 5 COMPLETED - Final Verifications - Moving to Stage 6

Tue Jan 14 02:24:37 CET 2014 UPGRADE 100% COMPLETED
Tue Jan 14 02:24:38 CET 2014 REBOOT YOUR BOX NOW FOR NEW KERNEL, DAHDI AND ASTERISK

How it is possible IF System Admin -> Updates shows 4.211.64-100 as AN update script available and Update Log show only the latest done (4.211.64-10)?

And the system is (still) naturally on CentOS 6.4 as I expect it is.

An edit: it’s clear at this point that the FreePBX.repo was touched too…

ls -alh /etc/yum.repos.d/FreePBX.repo -rw-r--r-- 1 root root 1.9K Jan 14 02:16 /etc/yum.repos.d/FreePBX.repo

Yesterday 2 AM localtime…so something happened for sure.

No I need to revert back the FreePBX.repo file to point to (CentOS) 6.4 because I noticed all the repos are pointing to (CentOS) 6.5 and 5.211.65…

DO you have auto updates set in sysadmin?

Wait Tony, just a moment.

What are you speaking of exactly?

(A) FreePBX Module updates? (B) FreePBX Distro system updates (minor releases)? (C) FreePBX Distro system upgrades (major releases)?

The answer to your question is: yes, I have “Update Schedule” set on “Weekly” BUT FreePBX Distro NEVER updated (nor upgraded) its FreePBX modules or just itself automatically before.

I can confirm that I only receive informational mails about FreePBX modules updates availability.

Nothing about FreePBX Distro major upgrades (e.g. from Track 4 to 5) and nothing automatically. Never (or never before this point in time). Sure 100%.

Since first installation (FreePBX Distro was Track 3) I always used myself to perform FreePBX module updates and system updates (minor releases) and/or upgrades (major releases) manually via CLI or GUI.

Nothing (never) happened automatically BEFORE the last FreePBX modules updates batch (I mean the one with “Sysadmin 2.11.0.38”).

Latest update notices received via e-Mail:

10.01.2014:

UPDATE NOTICE: There are 5 modules available for online upgrades
framework 2.11.0.12 (current: 2.11.0.11)
dahdiconfig 2.11.29 (current: 2.11.28)
core 2.11.0.10 (current: 2.11.0.9)
sysadmin 2.11.0.38 (current: 2.11.0.37)
fw_ari 2.11.0.4 (current: 2.11.0.0)

09.01.2014:

UPDATE NOTICE: There are 14 modules available for online upgrades
framework 2.11.0.12 (current: 2.11.0.11)
backup 2.11.0.6 (current: 2.11.0.5)
isymphony 1.4.9.3 (current: 1.4.9.2)
queues 2.11.0.12 (current: 2.11.0.11)
bulkdids 2.11.1.0 (current: 2.11.0.0)
endpoint 2.11.0.1.28 (current: 2.11.0.1.27)
directory 2.11.0.3 (current: 2.11.0.2)
dahdiconfig 2.11.29 (current: 2.11.28)
parking 2.11.0.13 (current: 2.11.0.12)
core 2.11.0.10 (current: 2.11.0.9)
asteriskinfo 2.11.0.7 (current: 2.11.0.2)
sysadmin 2.11.0.38 (current: 2.11.0.37)
fw_ari 2.11.0.4 (current: 2.11.0.0)
blacklist 2.11.0.3 (current: 2.11.0.2)

Updates I did manually (not in the sequence reported and not all during the same update session).

I really never paid much attention at the System Admin - Updates / Update Schedule function even if it is (and was) enabled on my system…simply because it never effectively did any update of any sort and, in the better (or worst?) case, I bet it should have eventually automatically “updated” only FreePBX modules and not the entire Major Release version of the system.

Isn’t it?

P.S.
Another interesting argument is that, despite the presence of that 5.211.65-1 upgrade log, no new packages of Track 5 were really installed on the system (checked e.g. yum list dahdi* or kernel* and no DAHDI 2.8.0 - Track 4 has the 2.7.0 - nor new Kernel was installed). So the system IS really still in Track 4 and SO that upgrade log is lying.

The new sysadmin pro updates the entire distro.

Hope I don’t misunderstood what you wrote:

“The new sysadmin pro updates the entire distro.”

If that is true, this “new feature” should have been advertised very well before!

I don’t see the words: unattended and automatic…

I’m quite sure I never ever read about such “new feature” of Sysadmin Pro module nor in the Forum neither in the Blog or on the JIRA. Do you? Is it really that?

Looking at the Change Log between sysadmin 2.11.0.37 and 2.11.0.38 (if you mean that 2.11.0.38 is the NEW one that incorporates THAT “new feature”) there are two supposedly harmless notices (see below the whole 2.11.0.x list):

2.11.0.38 Add more module keys to the subsystem
2.11.0.37 Prep for FreePBX 12 Release

2.11.0.38 Add more module keys to the subsystem 2.11.0.37 Prep for FreePBX 12 Release 2.11.0.36 Fix logger for fail2ban on asterisk greater than or equal to 11 2.11.0.35 Fix UPS device save issue 2.11.0.34 Re-write config for storage and ups when notifications is updated 2.11.0.33 Add license view for restapi, restapps, ha 2.11.0.32 Packaging of ver 2.11.0.32 2.11.0.31 Fix for no Apply Changes bar on install 2.11.0.30 Update Sysadmin 2.11.0.29 Packaging of ver 2.11.0.29 2.11.0.28 Incron restart support 2.11.0.26 Add support for HA stuff 2.11.0.25 NOT Available / NEVER officially released 2.11.0.24 Bomb out if we can't determine sysadmin version 2.11.0.23 Packaging of ver 2.11.0.23 2.11.0.22 Use fpbx_which for lsof if available 2.11.0.21 After unused ports 2.11.0.20 Check sysadmin rpm version and release on install 2.11.0.19 Packaging of ver 2.11.0.19 2.11.0.18 Packaging of ver 2.11.0.18 2.11.0.17 Packaging of ver 2.11.0.17 2.11.0.16 Packaging of ver 2.11.0.16 2.11.0.15 Packaging of ver 2.11.0.15 2.11.0.14 Add missing ports for portmgmt on install 2.11.0.13 Packaging of ver 2.11.0.13 2.11.0.12 Add Port Management for RESTful Phone Apps and RESTful API 2.11.0.11 NOT Available / NEVER officially released 2.11.0.10 NOT Available / NEVER officially released 2.11.0.9 Add restapps restart 2.11.0.8 Remove update from rnav for other distros 2.11.0.7 Packaging of ver 2.11.0.7 2.11.0.6 fix for restarting httpd during ajax post of registration 2.11.0.5 Fix Hardware Address save issue and intrusion email change 2.11.0.4 Fix spelling error 2.11.0.3 Packaging of ver 2.11.0.3 2.11.0.2 Packaging of ver 2.11.0.2 2.11.0.1 Allow location name to be set during registration 2.11.0.0 Bump for 2.11

Again, if true and if I don’t misunderstood what you wrote, someone then should learn to be a little more communicative/informative about that and, even if we will agree blindly with this new way of managing thing (if I don’t want my system be upgraded between Major Versions?), then there is something broken in this new “automagical whole system upgrade mechanism” of FreePBX Distro (at best: no notification and no administrator clearance, at worst: it didn’t worked at all resulting in a non finished upgrade)!

Isn’t it?

Sorry for my emphasis but I started with a strange recurring DDNS (PHP-WARNING) message and I finished learning that my system is doing a thing I never suspect it was ever able to do!

No System Admin Pro handles core updates.

When you run a upgrade script manually or use sysadmin pro it pulls a upgrade script which includes all RPMs and FreePBX modules. It has done this for 2 years now and not something new. A upgrade script handles the whole PBX including updating all available modules for the track you are on.

As you stated you have Sysadmin Pro set to check for weekly updates. This means every week it will look for Distro upgrades and apply them if they are available. As stated this has been the case for over 2 years now and its a feature of the System Admin Pro paid module. You can always turn off auto updates.

OK, so the statement of SkyKingOH is not fully pertinent because “new” doesn’t mean nothing at the moment.

Wait Tony…so you’re saying that I was always faster than my System Admin Pro Update module in discovering, downloading and applying the FreePBX Distro update scripts (Minor Releases) and so I always won in applying them first than Update scheduler (which always checked them weekly)?

Possible but improbable. OK…let we say the behaviour is this one…then the question is: I never received an informative mail of (new) Minor/Major Release availability. Isn’t that strange?

I still don’t understand how it’s possible that System Admin Pro Updates will perform a Major Release upgrade automatically which, in my opinion, is not “like” a Minor Release update.

There is something strange: in which way I could ever discover the system has automatically applied the 4.211.64-100 upgrade script (Track 4 to 5) if it (in the log only) requested a reboot to complete? It never completed! where are notifications of such procedure?

And…more…if I left the “Update Schedule” set to something different that “Never” (e.g. Weekly) then I’m going to expect next week the system will try to do an unattended upgrade to Track 5 again? really?

IMO first such type of feature should be divided to manage differently:

(1) FreePBX Distro system updates (Minor releases)
(2) FreePBX Distro system upgrades (Major releases)

Because a Major Release upgrade is not always wanted, on the contrary Minor Release may be accepted and applied with more confidence.

And then a series of informative notifications should be provided (never see one regarding core updates/upgrades availability from my system…in 1 year since 3.211.63-6 up to 4.211.64-10!).

Sysadmin does not send emails about updates. It just handles whatever you have set for a schedule.

Yes it will even upgrade to the next track as the release version of 4.211.64-100 whole job is to take you from track 4 to 5.

See comment above…maybe I miss the point (engineered in this way is such mechanism a good mechanism?): made in this way…it looks scaring at my eyes.

I can only say at this point that I was very Lucky to see it in action the very first time only after that 15 upgrade scripts were applied by me manually before the 16th!

Still the Track 4 to 5 never completed…and this is another story (I saw the repositories were changed but no RPMs were queried, downloaded or installed). Luckily.