Latest update ruins XMPP

I have upgraded a PBX deployment this morning that have been happy running Prosody as the chat server. I had some custom scripts etc running that managed the contact list and contact cards.

Since updating to the latest distro version 10.13.66-20 it has forcibly updated my system to use the new XMPP server lets chat. In doing so my custom scripts etc have become useless and there is very little information around about the FreePBX and lets chat integration. So now all of the users have lost their contact information which is used frequently for extension and contact numbers!

Also, a massive issue I am seeing is that users are connecting and disconnecting every minute or so. The XMPP client I have this site using is logging in and out continuously.

I am unprepared for this change over as I have been putting off the upgrade of the XMPP module. However doing my usual maintenance in module upgrades today there were many that required a distro update to complete. The upgrade forced the upgrade of XMPPā€¦ Not happy about this at all.

So it looks like LetsChat is just constantly crashing due to a java script error.

2017-06-07 10:55 +12:00: TypeError: undefined is not a function
at MongooseError.ValidationError.inspect (/var/www/html/admin/modules/xmpp/node/node_modules/lets-chat/node_modules/mongoose/lib/error/validation.js:63:17)
at formatValue (util.js:222:21)
at inspect (util.js:147:10)
at Console.exports.format (util.js:55:20)
at Console.log (console.js:55:34)
at /var/www/html/admin/modules/xmpp/node/node_modules/lets-chat/node_modules/lets-chat-freepbx-auth/lib/auth.js:55:21
at /var/www/html/admin/modules/xmpp/node/node_modules/lets-chat/node_modules/mongoose/lib/model.js:3748:16
at /var/www/html/admin/modules/xmpp/node/node_modules/lets-chat/node_modules/mongoose/lib/services/model/applyHooks.js:153:17
at process._tickDomainCallback (node.js:381:11)
2017-06-07 10:55 +12:00: Mongoose: mpromise (mongooseā€™s default promise library) is deprecated, plug in your own promise library instead: Mongoose v8.1.0: Promises

For now I have soled the problem by rolling back the XMPP module and reinstalling Prosody.

It looks like the problem is related to MongoDB and not being setup correctly. Here is a snippet from the error log of mongoDB:

Wed Jun  7 09:46:35.751 [conn4] Assertion: 16633:text search not enabled
0x86ccf40 0x86a62f6 0x868ff89 0x837f2fb 0x83da43b 0x848740c 0x8488465 0x83f8243 0x83f8bf2 0x83fde4c 0x81aa735 0x86b86cc 0x289bc9 0xcd5c9e
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x30) [0x86ccf40]
 /usr/bin/mongod(_ZN5mongo10logContextEPKc+0xa6) [0x86a62f6]
 /usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0xb9) [0x868ff89]
 /usr/bin/mongod(_ZNK5mongo3fts14FTSIndexPlugin15adjustIndexSpecERKNS_7BSONObjE+0xcb) [0x837f2fb]
 /usr/bin/mongod(_ZN5mongo19prepareToBuildIndexERKNS_7BSONObjEbbRSsRPNS_16NamespaceDetailsERS0_+0x107b) [0x83da43b]
 /usr/bin/mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibbbPb+0x1ec) [0x848740c]
 /usr/bin/mongod(_ZN5mongo11DataFileMgr16insertWithObjModEPKcRNS_7BSONObjEbb+0x65) [0x8488465]
 /usr/bin/mongod(_ZN5mongo14checkAndInsertEPKcRNS_7BSONObjE+0x1f3) [0x83f8243]
 /usr/bin/mongod(_ZN5mongo14receivedInsertERNS_7MessageERNS_5CurOpE+0x752) [0x83f8bf2]
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x9ac) [0x83fde4c]
 /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x95) [0x81aa735]
 /usr/bin/mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x49c) [0x86b86cc]
 /lib/libpthread.so.0(+0x6bc9) [0x289bc9]
 /lib/libc.so.6(clone+0x5e) [0xcd5c9e]
Wed Jun  7 09:46:35.754 [conn4] insert letschat.system.indexes keyUpdates:0 exception: text search not enabled code:16633 locks(micros) w:3229 3ms
Wed Jun  7 09:46:43.319 [conn8] Assertion: 16633:text search not enabled
0x86ccf40 0x86a62f6 0x868ff89 0x837f2fb 0x83da43b 0x848740c 0x8488465 0x83f8243 0x83f8bf2 0x83fde4c 0x81aa735 0x86b86cc 0x289bc9 0xcd5c9e
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x30) [0x86ccf40]
 /usr/bin/mongod(_ZN5mongo10logContextEPKc+0xa6) [0x86a62f6]
 /usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0xb9) [0x868ff89]
 /usr/bin/mongod(_ZNK5mongo3fts14FTSIndexPlugin15adjustIndexSpecERKNS_7BSONObjE+0xcb) [0x837f2fb]
 /usr/bin/mongod(_ZN5mongo19prepareToBuildIndexERKNS_7BSONObjEbbRSsRPNS_16NamespaceDetailsERS0_+0x107b) [0x83da43b]
 /usr/bin/mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibbbPb+0x1ec) [0x848740c]
 /usr/bin/mongod(_ZN5mongo11DataFileMgr16insertWithObjModEPKcRNS_7BSONObjEbb+0x65) [0x8488465]
 /usr/bin/mongod(_ZN5mongo14checkAndInsertEPKcRNS_7BSONObjE+0x1f3) [0x83f8243]
 /usr/bin/mongod(_ZN5mongo14receivedInsertERNS_7MessageERNS_5CurOpE+0x752) [0x83f8bf2]
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x9ac) [0x83fde4c]
 /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x95) [0x81aa735]
 /usr/bin/mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x49c) [0x86b86cc]
 /lib/libpthread.so.0(+0x6bc9) [0x289bc9]

MongoDB is crashing on your system, it looks corrupt.

As for XMPP. My best advice is you should fork the XMPP project and run it with your forked version as we have no plans to go back to Prosody. Then you can disable XMPP. This will allow you to continue to use prosody in the end.

Thanks @tm1000

Would this corruption have happened during the distro update? I have done everything though the webUI for the upgrades etcā€¦ so not sure what would have gone wrong here.

I will look into forking the XMPP module and maintaining my own version.

Looking ahead, is there any way to populate the vcard information using letschat, and as well as adding in the avatars rather than having to rely on Gravitar for that?

Thanks,

First and foremost sorry about that. We never built the XMPP module in a way that would have allowed for changes like the ones you made to persist, and unfortunately moved away from prosody to be able to allow for better integration.

As for MongoDB I havenā€™t seen a corruption with MongoDB happen as a result of a distro update yet, so Iā€™m interested in getting as much information as possible on this. Assuming you havenā€™t moved back to prosody just yet, can you tell me what happens if you try to restart mongod? Also whatā€™s the architecture of this system and the output of rpm -q | grep mongodb? Also how much free disk space do you have, as MongoDB is a bit of a hog, without some tuning which could eventually have other performance impacts.

As for Letā€™s Chat, Iā€™m unaware of it being able to handle the vcard information. However for handling avatars, I know it has a pull request that was contributed by a member of their community which can be found at Letā€™s Chat Pull Request 521. Keep in mind that this has not been pulled into the mainline branch and currently has merge conflicts, so your mileage with it may vary.

Hi @GameGamer43 Thanks for the reply.

Unfortunately I have already rolled back to using Prosody as this is in one of my production sites with about 60 people using and relying on the chat system for day to day business operations.

I have not uninstalled letschat or mongodb yet, however the output of the command you requested results in: ā€˜rpm: no arguments given for queryā€™. Prosody is holding the port for XMPP at this point which I think be preventing letschat from starting up. I am unsure of the procedure to remove this from the PBX (I am not familiar with node.js). I am happy to preform some tests on the system or provide more information if you like but I would want to do this in a way that will not impact the function of Prosody at this point. After, I can look into removing LetsChat from this PBX.

I donā€™t believe the corruption would have happened because of the distro update, but could have happened at any time when the module was updated.

This is the FreePBX distro for FreePBX 13, so I believe its running on CentOS6 on 64bit hardware. As for disk space, I have only used about 30% of my whole drive, with the most used partition using 13% of its capacity.

Bellow is a screenshot from my client showing how I am using the vCard information. At the moment I just have a script that pulls this information from the AD server (including avatar) and populates the prosody database with the appropriate information. This is a very handy feature as there is is also acting as their internal address book to transfer and chat between each other.

I am going to start looking into how letschat works with FreePBX and it would be good to get some information put up somewhere about it as there is very little information around about this at the moment. All of the FreePBX wiki pages are still referencing Prosody as the XMPP server for FreePBX.

Letā€™s chat gets installed as a result of the xmpp module, so if you rolled back the filesystem it used is gone as it should be self contained inside the module.

Iā€™ve also opened a ticket to fix our documentation (FREEPBX-15014).

Thanks @GameGamer43 Another question I have is why do the distro updates reinstall all of the modules in the available list? This is quite painful as I remove the modules that are not being used and when it comes to do distro updates they are all reinstated. Is there any reason why they are all reinstalled, and can this be stopped?

That question is complex and has complex answers. To solve the issue just disable any module you donā€™t want updated.

Actually ( and hopefully I will bypass unnessary wrath here), the question was really not complex, the answer might well be a good 'work around" though and one I personally script :slight_smile:

I was having what appears to be the same problem with MongoDB / LetsCat (Sat Aug 5 12:35:26.567 [conn34] Assertion: 16633:text search not enabled), so this looks to not be a one-off issue. Crawling down the rabbit hole, I had to enable full text search by adding this line to mongod.conf:

setParameter=textSearchEnabled=true

I then restarted MongoDB:

/etc/init.d/mongod restart

Then verified that Full Text indexing was enabled and working:

tail /var/log/mongodb/mongodb.log

Tue Aug 15 09:38:24.977 [conn2] build index letschat.messages { _fts: ā€œtextā€, _ftsx: 1, room: 1, posted: -1, _id: 1 } background
Tue Aug 15 09:38:24.978 [conn2] build index done. scanned 0 total records. 0 secs

And then restarted LetsChat:

fwconsole pm2 --restart xmpp

And after all thatā€¦ I donā€™t think it works with Active Directory users. I created a test user on the internal directory and that seems to work OK.

Letā€™s Chat/Xmpp dont know the difference between Active Directory and Internal users (they query against User Manager not the directory), most-likely your directory is misconfigured in User Manager.

Odd. Active Directory is working fine for everything else (user / group creation and deletion, UCP authentication, etc.). Active Directory users donā€™t get the XMPP icon in UCP; local users do (yes, the per-user XMPP settings were checked). I donā€™t even know where to begin troubleshooting this issueā€¦ if itā€™s too crazy to address in the forums, let me know and Iā€™ll buy some support time.

EDIT - Never mind - latest updates fixed it. Seems OK now.

1 Like