Please confirm that with Verify Server turned off, it still fails in the same way.
Does packetcaputere not capture TLS due to the encryption? or does packet capture need certain commands to capture what we are looking for?
Gentlemen. I need to run to school for a weekend class. So I need to duck out for most of the day. Thank you for the dynamic interaction here.
Did you capture the right traffic. Yes, you can still capture the packets going through the interface…may need a cert to view the actual data but we should still see the traffic. How did you actually capture this traffic? What command did you use?
This is probably the incoming INVITE:
50 19:47:38.858099 192.76.120.10 45.32.129.200 TLSv1.2 1860 Application Data
and this is probably the Trying:
52 19:47:38.860875 45.32.129.200 192.76.120.10 TLSv1.2 724 Application Data
This is probably the OK:
118 19:47:42.636068 45.32.129.200 192.76.120.10 TLSv1.2 1266 Application Data
and this, the ACK:
120 19:47:42.812007 192.76.120.10 45.32.129.200 TLSv1.2 607 Application Data
You will need the session keys, not just the certificate, to decrypt them. The certificate is present, in clear, in the capture, and all certificates are public information.
I half remember there being options to output the session keys in a form usable by wireshark, but the reason that I suggested matched time stamps is to avoid the need for that, although I didn’t bother matching them up on time, in the end, as it was clear what was failing.
The slight problem with the capture is when you started it, not what was on the command line. There should be lines with SYN for the above TLS session. Note that this session could have been reused from previous incoming calls. Also there aren’t enough receive transmit pairs to account for the full SIP dialogue.
Note there seems to be about 59m 59s difference between the time stamps.
Previous reports are that Twilio didn’t have the problem:
This is the page in the GUI that contains the Verify Server setting. I’m not sure how to navigate to it, as I’m an Asterisk person, not a FreePBX one, and I’ve not been able to find a web based user guide since the great documentation reorganisation.
Im back from school so catching up on messages here.
@BlazeStudios - The command I used to capture the packets was:
tcpdump -i eth0 -s0 -w /tmp/pcap.pcap
@david55 - Is there a SSH command line I can use to export the session keys?
@david55 - I have found this issue now with both Telnyx and Twilio TLS trunks. Both trunk providers are having the same issue.
@david55 @Stewart1 - Yesterday I found this thread (post link below), that also was having TLS issues, and for them turning off “Verify Server” worked. I tried this yesterday, for me “Verify Server” was on so I then turned it off and had no noticeable different result.
I just ran a test and everything is working.
I went back and turned “Verify Server” to Yes. And the problem came back.
I then set “Verify Server” to No. And the problem remained
I then did a CLI Core Restart Now and the problem went away.
It seems like when you set “Verify Server” to No, a core restart is necessary.
Yes, certain parts of transports when changed require a restart of Asterisk.
But the whole things smells like your certification is bad, and not verifying the server kind of defeats the point of most of the security provided by TLS checking
The problem is with Telnyx’s certification, not the OP’s. As I’ve said several times, a machine they have told the OP to contact as 192.76.120.10 is claiming to be sip.telnyx.com in its certificate, and is not providing 192.76.120.10 as an alternative name, so Asterisk is unable to verify that it is the real 192.76.120.10.
Turning off Verify Server makes the OP vulnerable to a man in middle attack on outgoing TLS sessions, both new calls, and when they act as UAC in a dialogue which started with them as UAS. They need to understand the implications of doing this, and make a risk assessment, based on their specific threats, which might lead them to the conclusion that they need to use a provider that has a good understanding of the Public Key Infrastructure.
Note to the OP, if you follow up on any of the other threads on this topic, please make sure you include warnings similar to the above paragraph. Too often people tell people to do something that disables security to get round security based failures, without explaining the implications.
I can think of heuristics that Asterisk might use, if this problem becomes common, but they would be feature requests, not bugs, and would have to be off, by default, as they still compromise security mechanisms. Keeping the security compromise complete and, hopefully obvious, is probably safe (although judging by the number of people who blindly copied “insecure=verify”, I’m not sure that obvious is enough).
I copy and pasted conversations here from @david55 in my latest response with Telnyx. Here is my latest response to Telnyx.
Hi Gentlemen,
I have some additional information. If in asterisk I set “Server Verification” to “NO”, it turns off verification telnyx servers and the issue goes away and the system functions as required. Unfortunately turning off verification defeats part of the security benefits of TLS and leaves open the possibility for a “man in the middle” attack.
There are two possible issues that are happening here.
(1) 192.76.120.10 is claiming to be sip.telnyx.com in its certificate, and is not providing 192.76.120.10 as an alternative name, so Asterisk is unable to verify that it is the real 192.76.120.10.
(2) Telnyx’s response to the BYE command is not using the record - route, but attempting to make a new connection for purposes of issuing the response to the BYE command. When this response hits Asterisk, it is rejected because it is not coming from a verified server. If the response to the BYE command used the record-route path, it would be accepted by Asterisk.
FYI- Twilio has the exact same issue. I’m guessing the two of you use the same or similar backend which is causing the issue.
Hope this helps!
David
The IP address the service is bound to is never required for the TLS cert. Your cert will have the Common Name (the domain you are securing) and it can have additional SANs. Unless you are specifically signing a cert against an IP as the Common Name or one of the SANs, the doesn’t matter. TLS uses a public-key method. The IP of the server is not really what is checked since a server can have multiple IPs using the same cert.
How did you come up with this? You said that Telnyx never saw the BYE coming from Asterisk? If Telnyx never saw the BYE and then how can reply not using the record-route? Nothing you said in point two matches what I explained. The short version was:
- Asterisk sends BYE to Telnyx
- There’s no response to the BYE from Telnyx (again, you said they never see it)
- A BYE is sent from Telnyx because the other side ends the call…almost 60 seconds later
- Asterisk sends a 481 Call/Leg Transaction Doesn’t Exist back to Telnyx because the leg no longer exists in Asterisk.
The verify server is happening when Asterisk sends the BYE to Telnyx meaning the connection never gets verified completed it would seem. But none of that has to do with the IP of the server.
How Server Verification is done:
In RSA key exchange, the client generates a random sequence of bytes and performs RSA encryption using the public key from the server’s certificate. Then the client sends the resulting ciphertext to the server and expects the server to decrypt it (using the private key corresponding to the public key from the certificate) and use the random value in a KDF, together with other values, to generate symmetric keys and send a Finished message encrypted with the resulting symmetric keys. The client verifies the Finished message. The server can only succeed in generating the expected symmetric keys by decryption RSA encrypted message. RFC 5246: The Transport Layer Security (TLS) Protocol Version 1.2
In DHE/ECDHE key exchange with PFS, the server signs its ephemeral key using the private key corresponding to the public key in the certificate and sends this in ServerKeyExchange. The client verifies the signature using the public key from the certificate. RFC 5246: The Transport Layer Security (TLS) Protocol Version 1.2
So yes, when you turn off verify server it skips that whole process of using the private key on the server side. What this really sounds like, a private key is missing on Telnyx’s server you are using.
Again, server verification has nothing to do with the IPs unless that IP is specifically being secured by the certificate. If I have a website that I assigned a new IP for…the TLS cert would not have to be updated because I changed IPs.
Telnyx are not providing a name in their Record-Route header; they are only providing an IP address, so the IP address is the only thing that can be checked against the certificate.
Asterisk tries to send BYE, but the transport layer aborts, due to the server authentication failure, before it is actually sent.
This is the common name part of the certificate sent by Telnyx:
But that is not the domain name part of the URI that Asterisk has been told to connect back to. It has been told to connect back to the one in the Record Route header, which is:
Record-Route: <sip:192.76.120.10:5061;transport=tls;lr;r2=on;ftag=XyBXH90cyy91g>
The hostname part of which is 192.76.120.10. “192.76.120.10” !=
so server verification fails, and the TLS session has to be abandoned, unused.
The full certificate, from Telnyx, as decoded by wireshark, is:
Certificate: 3082065930820541a003020102021100a666b33f0c81f28ea1a204898ac944b4300d0609… (id-at-commonName=sip.telnyx.com)
signedCertificate
version: v3 (2)
serialNumber: 0x00a666b33f0c81f28ea1a204898ac944b4
signature (sha256WithRSAEncryption)
Algorithm Id: 1.2.840.113549.1.1.11 (sha256WithRSAEncryption)
issuer: rdnSequence (0)
rdnSequence: 5 items (id-at-commonName=Sectigo RSA Domain Validation Secure Server CA,id-at-organizationName=Sectigo Limited,id-at-localityName=Salford,id-at-stateOrProvinceName=Greater Manchester,id-at-countryName=GB)
RDNSequence item: 1 item (id-at-countryName=GB)
RelativeDistinguishedName item (id-at-countryName=GB)
Object Id: 2.5.4.6 (id-at-countryName)
CountryName: GB
RDNSequence item: 1 item (id-at-stateOrProvinceName=Greater Manchester)
RelativeDistinguishedName item (id-at-stateOrProvinceName=Greater Manchester)
Object Id: 2.5.4.8 (id-at-stateOrProvinceName)
DirectoryString: printableString (1)
printableString: Greater Manchester
RDNSequence item: 1 item (id-at-localityName=Salford)
RelativeDistinguishedName item (id-at-localityName=Salford)
Object Id: 2.5.4.7 (id-at-localityName)
DirectoryString: printableString (1)
printableString: Salford
RDNSequence item: 1 item (id-at-organizationName=Sectigo Limited)
RelativeDistinguishedName item (id-at-organizationName=Sectigo Limited)
Object Id: 2.5.4.10 (id-at-organizationName)
DirectoryString: printableString (1)
printableString: Sectigo Limited
RDNSequence item: 1 item (id-at-commonName=Sectigo RSA Domain Validation Secure Server CA)
RelativeDistinguishedName item (id-at-commonName=Sectigo RSA Domain Validation Secure Server CA)
Object Id: 2.5.4.3 (id-at-commonName)
DirectoryString: printableString (1)
printableString: Sectigo RSA Domain Validation Secure Server CA
validity
notBefore: utcTime (0)
utcTime: 2023-04-13 00:00:00 (UTC)
notAfter: utcTime (0)
utcTime: 2024-05-12 23:59:59 (UTC)
subject: rdnSequence (0)
rdnSequence: 1 item (id-at-commonName=sip.telnyx.com)
RDNSequence item: 1 item (id-at-commonName=sip.telnyx.com)
RelativeDistinguishedName item (id-at-commonName=sip.telnyx.com)
Object Id: 2.5.4.3 (id-at-commonName)
DirectoryString: printableString (1)
printableString: sip.telnyx.com
subjectPublicKeyInfo
algorithm (rsaEncryption)
Algorithm Id: 1.2.840.113549.1.1.1 (rsaEncryption)
subjectPublicKey: 3082010a0282010100bcdd08ccceef76c6bb38f4f9624af3c1a07af60fcc86a1dd91e5e0…
modulus: 0x00bcdd08ccceef76c6bb38f4f9624af3c1a07af60fcc86a1dd91e5e012d157412d0176fc…
publicExponent: 65537
extensions: 9 items
Extension (id-ce-authorityKeyIdentifier)
Extension Id: 2.5.29.35 (id-ce-authorityKeyIdentifier)
AuthorityKeyIdentifier
keyIdentifier: 8d8c5ec454ad8ae177e99bf99b05e1b8018d61e1
Extension (id-ce-subjectKeyIdentifier)
Extension Id: 2.5.29.14 (id-ce-subjectKeyIdentifier)
SubjectKeyIdentifier: cfb2d329a762daec86c62b9806ae866bd468f830
Extension (id-ce-keyUsage)
Extension Id: 2.5.29.15 (id-ce-keyUsage)
critical: True
Padding: 5
KeyUsage: a0
1... .... = digitalSignature: True
.0.. .... = contentCommitment: False
..1. .... = keyEncipherment: True
...0 .... = dataEncipherment: False
.... 0... = keyAgreement: False
.... .0.. = keyCertSign: False
.... ..0. = cRLSign: False
.... ...0 = encipherOnly: False
0... .... = decipherOnly: False
Extension (id-ce-basicConstraints)
Extension Id: 2.5.29.19 (id-ce-basicConstraints)
critical: True
BasicConstraintsSyntax [0 length]
Extension (id-ce-extKeyUsage)
Extension Id: 2.5.29.37 (id-ce-extKeyUsage)
KeyPurposeIDs: 2 items
KeyPurposeId: 1.3.6.1.5.5.7.3.1 (id-kp-serverAuth)
KeyPurposeId: 1.3.6.1.5.5.7.3.2 (id-kp-clientAuth)
Extension (id-ce-certificatePolicies)
Extension Id: 2.5.29.32 (id-ce-certificatePolicies)
CertificatePoliciesSyntax: 2 items
PolicyInformation
policyIdentifier: 1.3.6.1.4.1.6449.1.2.2.7 (iso.3.6.1.4.1.6449.1.2.2.7)
policyQualifiers: 1 item
PolicyQualifierInfo
Id: 1.3.6.1.5.5.7.2.1 (id-qt-cps)
DirectoryString: https://sectigo.com/CPS
PolicyInformation
policyIdentifier: 2.23.140.1.2.1 (joint-iso-itu-t.23.140.1.2.1)
Extension (id-pe-authorityInfoAccess)
Extension Id: 1.3.6.1.5.5.7.1.1 (id-pe-authorityInfoAccess)
AuthorityInfoAccessSyntax: 2 items
AccessDescription
accessMethod: 1.3.6.1.5.5.7.48.2 (id-ad-caIssuers)
accessLocation: 6
uniformResourceIdentifier: http://crt.sectigo.com/SectigoRSADomainValidationSecureServerCA.crt
AccessDescription
accessMethod: 1.3.6.1.5.5.7.48.1 (id-ad-ocsp)
accessLocation: 6
uniformResourceIdentifier: http://ocsp.sectigo.com
Extension (SignedCertificateTimestampList)
Extension Id: 1.3.6.1.4.1.11129.2.4.2 (SignedCertificateTimestampList)
Serialized SCT List Length: 361
Signed Certificate Timestamp (Google 'Xenon2024' log)
Serialized SCT Length: 119
SCT Version: 0
Log ID: 76ff883f0ab6fb9551c261ccf587ba34b4a4cdbb29dc68420a9fe6674c5a3a74
Timestamp: Apr 13, 2023 10:58:42.516000000 UTC
Extensions length: 0
Signature Algorithm: ecdsa_secp256r1_sha256 (0x0403)
Signature Hash Algorithm Hash: SHA256 (4)
Signature Hash Algorithm Signature: ECDSA (3)
Signature Length: 72
Signature: 3046022100db0401b5005a633c5f35a992836607def14132fbb75abb47831a363718845d…
Signed Certificate Timestamp (Cloudflare 'Nimbus2024' Log)
Serialized SCT Length: 118
SCT Version: 0
Log ID: dab6bf6b3fb5b6229f9bc2bb5c6be87091716cbb51848534bda43d3048d7fbab
Timestamp: Apr 13, 2023 10:58:42.616000000 UTC
Extensions length: 0
Signature Algorithm: ecdsa_secp256r1_sha256 (0x0403)
Signature Hash Algorithm Hash: SHA256 (4)
Signature Hash Algorithm Signature: ECDSA (3)
Signature Length: 71
Signature: 304502201504d582ebd0f2bb9516c9c29690e64f841c2606f7c08dc083377005cddca0e0…
Signed Certificate Timestamp (Google 'Argon2024' log)
Serialized SCT Length: 118
SCT Version: 0
Log ID: eecdd064d5db1acec55cb79db4cd13a23287467cbcecdec351485946711fb59b
Timestamp: Apr 13, 2023 10:58:42.553000000 UTC
Extensions length: 0
Signature Algorithm: ecdsa_secp256r1_sha256 (0x0403)
Signature Hash Algorithm Hash: SHA256 (4)
Signature Hash Algorithm Signature: ECDSA (3)
Signature Length: 71
Signature: 304502200668e9604a85a91b53fe76d7b89616457436d811f89d5eae34acddeee22c3438…
Extension (id-ce-subjectAltName)
Extension Id: 2.5.29.17 (id-ce-subjectAltName)
GeneralNames: 3 items
GeneralName: dNSName (2)
dNSName: sip.telnyx.com
GeneralName: dNSName (2)
dNSName: sip-anycast1.telnyx.com
GeneralName: dNSName (2)
dNSName: sip-anycast2.telnyx.com
algorithmIdentifier (sha256WithRSAEncryption)
Algorithm Id: 1.2.840.113549.1.1.11 (sha256WithRSAEncryption)
Padding: 0
encrypted: 5c61879a1fd972c67dd57c47d8848f43665c99d8a40e9f35140760e0985c3bd16762e9bf…
Not sure that is doing what you think it is doing considering that a record-route is added through each hop in the path between source and destination. So if telnyx had a SIP proxy between that server and the client, the last record-route would be the proxy. If the OP had a proxy between Telnyx and the PBX, when the PBX got the requests it would be the same thing. Proxy is the top record-route.
Since there is only one hop between Telnyx and the OP in this case, there’s only one record-route. But that’s not always the case.
TLS and TLS certificates are hop by hop at the transport and IP layers, not end to end.
I was puzzling over this and had posted a thought that Asterisk was not doing the right thing in-dialog. But after checking out the code a bit (asterisk/main/tcptls.c at 18 · asterisk/asterisk · GitHub is where the server validation code is found) it seems like it keeps track of the session when Asterisk is the UAC and makes a valid connection to a UAS even if things like Route headers point to IPs.
But when Asterisk is the server and receives a TLS connection, I don’t see how the verify logic would typically work, since the information the provider puts in the Record-Route or Contact header is almost always going to be an IP address, not a DNS name. (Indeed, even Asterisk only puts IP addresses into Contact headers, so this scenario should also fail when connecting two Asterisk servers together, even if both have valid TLS certificates in place)
one more edit, at the risk of being obnoxious
The code shows a flag AST_SSL_IGNORE_COMMON_NAME which seems like it would be helpful but I don’t see anywhere in config that this flag can be set.
Here is the reply from Telnyx.
Hello David,
- sip.telnyx.com is the fqdn which resolves to 192.76.120.10 and is correct.
- Per RFC, we are following the Record Route header logs to be presented once in a SIP dialog
RFC 5658 - Addressing Record-Route Issues in the Session Initiation Protocol (SIP)