Inactive Stasis app 'hey' missed message

FreePBX Applications / Modules
1

I’m seeing this on two of my servers after the latest updates to core and framework. Oddly though I don’t see this on other servers. The two in question are both running Zulu, Sangoma connect and phone apps. These are also the only ones running VPN for remote phones.

I do have a couple others running the same without the VPN component and they do not have this message spamming the console 3 or so times a second. Does anyone know what this app hey is?

2

Might be a new exploit that is somehow taking advantage of ARI.

... bunch of normal logs and then ...
[2023-01-05 15:43:26] VERBOSE[24988] stasis/app.c: Creating Stasis app 'hey'
[2023-01-05 15:43:26] VERBOSE[24988] res_http_websocket.c: WebSocket connection from '139.144.35.125:63531' for protocol '' accepted using version '13'

(ip listed is some random Linode vm)

Later:

[2023-01-05 15:56:28] ERROR[24988] res_http_websocket.c: Error reading from web socket: Connection reset by peer
[2023-01-05 15:56:28] WARNING[24988] ari/ari_websockets.c: WebSocket read error: Connection reset by peer
[2023-01-05 15:56:28] VERBOSE[24988] stasis/app.c: Deactivating Stasis app 'hey'

and then you start getting

[2023-01-05 15:56:39] VERBOSE[24990] stasis/app.c: Inactive Stasis app 'hey' missed message

in Asterisk:

*CLI> stasis show topics

Name                                                             Detail                                                          
ari:application/hey    
...

*CLI> stasis show topic ari:application/hey 
Name: ari:application/hey
Detail: 
Subscribers count: 1
Forwarding topic count: 2
Duration time: 190:19:38

and

*CLI> ari show app hey 
Name: hey
  Debug: Yes
  Subscription Model: Global Resource Subscription
  Subscriptions: 3
    Channels:
      __AST_CHANNEL_ALL_TOPIC (1)
    Bridges:
      __AST_BRIDGE_ALL_TOPIC (1)
    Endpoints:
      __AST_ENDPOINT_ALL_TOPIC (1)

I’m unfamiliar with these components of Asterisk but it looks like at least an attempt at exfiltrating information from the PBX. Not sure what all can be gained here, but best to close this off until the root cause is figured out.

3

In order for an external host to create an ARI app, the asterisk http/https services would need to be reachable, they should be blocked by default in the firewall. They would also need to know ARI credentials.

4

On the server I am investigating, Settings → Asterisk REST Interface Users has no users listed.

The ari_additional.conf file has only a freepbxuser with encrypted password.

I’m puzzled at the moment. @wwenthin do you have any more clues?

5

This show nothing for me BUT… early this morning I executed fwconsole reload The messages seemed to have stopped with that. I also pushed asterisk up to 18 certified LTS on both systems.
After this I started seeing this at least once a second until I firewalled them to oblivion: WARNING[235581] res_pjsip_registrar.c: Endpoint ‘anonymous’ (135.181.233.61:53051) has no configured AORs

Not really sure if that was someone that is sitting at home now wondering where they phones went or part of the previous fun.

6

Hello, got 4 PBX hacked here too, they are somehow bypassing ARI authentication and placing calls this way
Also somehow on some of the PBXs i found 8088 port open to the internet even if firewall was active and set to trust only local IPs

I see it being open because of a rule named “webrtc”

7

If you are using a certified version of Asterisk, you should be using the support contract for which you obtained it. If you don’t have such a contract, you should be using the latest, non-certified version.

8

I have in the meanwhile sent an abuse complaint to Linode, i’ve checked and the IP in my case is the same as posted by @billsimon
For now everyone should absolutely check that 8088 is actually closed from outside and preferably use a separate firewall in front of it while it is figured out how it happened

In my case it has been used to place 0.60€ / min calls to Africa

9

Same problem here on a freepbx box of a client… posting here so will get email if something is posted…

10

same problem on nearly 100 hosted Freepbx. Each comes from a master so they have the same ARI password. When I change the password, it stopps as if I disable ARI in advanced settings.

BUT what seems not normal is THAT my firewall is correctly configured. The only thing wich was with INTERNET selected were the OpenVPN server.

I think there is a security breach somewhere @lgaetz

@tizbac tizbac my 8088 is closed, that’s not the issue

11

Port 8089 is the TLS port for the same services (asterisk HTTP).

12

@billsimon neither 8088 nor 8089 is open on my server.
I checked this with Port Checker - Check Open Ports Online

13

strange, I have desactivate the ARI and modify the admin password for ARI in the full log file, I still have twice a minute : stasis/app.c: Inactive Stasis app ‘hey’ missed message

is that normal ?

UPDATE : after REBOOT, no more message

14

Hi
we had the same with one of our clients the last night - all servers have different ARI password.

FreePBX 16 asterisk 18 updated a few days a go.

res_http_websocket.c: WebSocket connection from ‘139.144.35.125:58761’ for protocol ‘’ accepted using version ‘13’

In FreePBX Firewall normally nothing is open , only OpenVPN

We find two “mistakes” , server has no Sangoma connect or zulu.

Sip settings pjsip ws and wss yes

Firewall services Sip Protocol Internet was selected (This protocol is being managed by the Responsive Firewall. You should not enable access from the ‘Internet’ zone, or Responsive Firewall will be bypassed.)

We saw

CLI> ari show apps

Application Name

=========================

hey

after changing above an restarting asterisk

CLI> ari show apps

Application Name

=========================

I am not familiar with ARI, so I don‘t understand how they could do this

15

It is sufficient to bind Asterisk HTTP services to 127.0.0.1 so that internal communications between FreePBX apps and ARI or webrtc still work but Asterisk HTTP won’t be listening on a network interface. You can make the change in Advanced Settings or fwconsole setting…

fwconsole setting HTTPTLSBINDADDRESS "127.0.0.1"
fwconsole setting HTTPBINDADDRESS "127.0.0.1"
fwconsole r

note well that this will block webrtc connections. However, Sangoma Phone desktop still works because its webrtc is proxied through a different port. I don’t know about Zulu. UCP’s webrtc widget uses 8089 and would be blocked.

1 Like
16

THX
we have done this to all pbxs.
But how could they get the password for ari,there was no extra user only the in advanced settings

17

We can also confirm this, 2 Hosted and one onPremise (with Static IP)

Example Log

  1. [2023-01-23 16:43:30] VERBOSE[1631] stasis/app.c: Creating Stasis app ‘hey’
    [2023-01-23 16:43:30] VERBOSE[1631] res_http_websocket.c: WebSocket connection from ‘139.144.35.125:49205’ for protocol ‘’ accepted using version ‘13’
    [2023-01-23 16:43:30] VERBOSE[1631] stasis/app.c: Activating Stasis app ‘hey’

And after this messages, spam of outgoing calls started to expensive numbers.

Same IP from attacker. Like @marcelo and @billsimon posted.

18

Do you have a list of those numbers?

19

I messaged you @wwenthin

And yes, I have this numbers in “Call Event Logging” when I go entry and go “Details Show”.

2 of 3 Systems got lucky, the Trunk Provider used had an Cost Protection and the customer was still alerted by trunk customer service about it.
The last one (well Telekom Germany) Informed the client veeery late about this unusual calls, many of them was not blocked.

20

Hello,

Our solution have been closing the 8088 port, but the question is: how they knew ARI credentials? I think the ARI default random password is strong enough, isn’t it?