CID-Superfecta: problem with ä,ö,ü and ß - ISO-8859-15 to UTF-8 conversion not working - workaround for german provided

stindl · January 17, 2019, 6:00pm

freePBX 14.0.5.25
CID-Superfecta 14.0.7
Herold-Austria-template

When Superfecta does a lookup, it correctly displays the german umlaute, but once it does the utf-8 converting, they are gone.
Instead of Schäcke you get Sch?cke on the phone display and in the CDR lists. I just want to report it, I dont have a problem with it…if it cannot get fixed

Here’s one example:
> …Debug is on and set at level: 1
> The Original Number: 4350121018
> The Scheme: Default
> Scheme Type: SINGLEFECTA
> Debugging Enabled, will not stop after first result
>
> Scheme Asked is: Default
> The DID is: 5555555555
> The CNUM is: 4350121018
> The CNAME is: CID Superfecta!
>
> Starting scheme Default
> Matched CID Rule: 0+43|ZXXXXX. with 050121018
> Changed CNUM to: 050121018
> Executing Send to email
> Searching Send to Email …
> result took 0.0018 seconds.
>
> Executing Superfecta Cache
> Searching Superfecta Cache …
> not found
> result took 0.0029 seconds.
>
> Executing LDAP
> Searching LDAP for number: 050121018
> Attempting to bind, using ad:‘Resource id #9’ UID:’’ …
> Searching ou=people,dc=example,dc=com … 0 entries returned
> not found
> result took 0.0260 seconds.
>
> Executing FreePBX Contactmanager
> result took 0.1067 seconds.
>
> Executing Asterisk Phonebook
> Searching Asterisk Phonebook …
> not found
> result took 0.0286 seconds.
>
> Executing Herold Austria
> Searchinghttps://www.herold.at/ …
> ‘Schäcke, eine Marke der Rexel Austria GmbH’
> result took 0.5588 seconds.
>
> Executing TelefonABC Austria
> Searchinghttp://www.telefonabc.at …
> The requested URL returned error: 404 Not Found
> result took 0.3114 seconds.
>
> Converting result to UTF-8
> Post CID retrieval processing
> Executing Send to email
> Email sent to [email protected] Incoming call from Sch?cke, eine Marke der Rexel Austria GmbH at 050121018
> Executing Superfecta Cache
> Caller ID data added to Superfecta_Cache.
> Done
> This scheme would set the caller id to: Sch?cke, eine Marke der Rexel Austria GmbH
>
> Returned Result would be:Sch?cke, eine Marke der Rexel Austria GmbH
> result took 1.1436159610748 seconds

The problem is not related to the lookup-template, it also happens when the name with a german umlaut is correctly stored in the superfecta cache…

cynjut · January 17, 2019, 6:05pm

Can the phone handle umlauts?

stindl · January 17, 2019, 6:10pm

yes, they can…because of this, I think…

<userLocale>
        <name>German_Germany</name>
        <uid></uid>
        <langCode>de</langCode>
        <version></version>
        <winCharSet>ISO-8859-1</winCharSet>
    </userLocale>
    <networkLocale>Austria</networkLocale>
    <networkLocaleInfo>
        <name>Austria</name>
        <version></version>
    </networkLocaleInfo>

You can see in the log of superfecta (see above) that the umlaut “ä” gets converted to “?” WITHIN superfecta…

stindl · January 18, 2019, 9:24am

If the UTF-8 conversion cannot be fixed, because the problem lies somewhere else in freePBX (I read of similar problems in cdr-reports)…maybe an option to switch it off in the CID-Superfecta template would help…

like this

CID-Superfecta-UTF-8

tm1000 · January 18, 2019, 4:57pm

The problems in cdr have been fixed for a very long time.

stindl · January 18, 2019, 5:55pm

Ok…I filed a bug report…
https://issues.freepbx.org/browse/FREEPBX-18998

tm1000 · January 18, 2019, 5:57pm

Perfect thank you!

stindl · January 18, 2019, 8:25pm

One of my extensions is Büro-2…and when I make an outgoing call, in the CDR Reports it is displayed as

Mon, 14 Jan 2019 20:23 [1547493806.26]“BÃ¼ro-2” <19>

…so it cannot be a bug of CID-Superfecta, since Superfecta is not involved in the naming of extensions. It must be a problem of freePBX and UTF-8.

tm1000 · January 18, 2019, 10:12pm

Nope. As that is how it’s shown in the database. Example:

If you look at asteriskcdrdb and table cdr. You will see it is inserted wrong in there. That is not an issue with FreePBX since freepbx does not insert values directly into CDR nor is it an issue with Asterisk

stindl · January 18, 2019, 10:26pm

why does it look like this in cdr reports
buero
…on my system?

Is it possible that there is a difference, between you inserting a german umlaut (on a US-computer) into the freePBX webgui and me doing it on a Mac/Windows machine with a german keyboard layout?

tm1000 · January 18, 2019, 10:27pm

No there is no difference. UTF-8 is UTF-8. Is this an Official FreePBX Distro system?

stindl · January 18, 2019, 10:28pm

yes, I downloaded the image file a few months ago from the freePBX website and installed it as a virtualbox guest system. Where do I see the distro info?

tm1000 · January 18, 2019, 10:29pm

Then you will need to follow the steps in this ticket https://issues.freepbx.org/browse/FREEPBX-15268

Most likely the second to the last response

rpm -e mysql-connector-odbc --nodeps
yum -y install mariadb-connector-odbc
fwconsole restart

stindl · January 18, 2019, 10:31pm

Thanks a lot…I will try…and report

tm1000 · January 18, 2019, 10:37pm

This will most likely work for you as there are Russian, German and Israeli members contributing in that ticket.

stindl · January 19, 2019, 12:17am

When I tried it, my cdr reports stopped working at all. Isnt mysql-connector-odbc the default in a freePBX 14 distro? Why remove it and replace it with mariaDB?

Anyway…I went back in virtualbox to the previous version and cdr reports work again…I don’t care about the german umlaute anymore

stindl · January 19, 2019, 12:35am

Sorry, my fault…I did something else before, which broke it.

Tried the 15268-fix again, and it worked. In CDR-Reports the german umlaute get displayed!
Thanks @tm1000
Buero-2

So, you were right…the umlaut problem in superfecta is a separate problem…nothing changed, it is still the same issue I posted.

…
Converting result to UTF-8
Post CID retrieval processing
Executing Send to email
Email sent to [email protected] Incoming call from Kr?uterapotheke at 0186712xxx
Executing Superfecta Cache
Caller ID data added to Superfecta_Cache.
Done
This scheme would set the caller id to: Kr?uterapotheke

Returned Result would be:Kr?uterapotheke
result took 3.8335919380188 seconds

tm1000 · January 19, 2019, 4:47am

Mariadb is actually the sql server in freepbx but it uses the mysql odbc connector. As you can see there are issues with that. So we have people try the mariadb one. It’s not live because I’ve seen it crash asterisk in one case so I need more testing.

stindl · January 19, 2019, 9:07am

Interesting…which connector does superfecta use? Maybe I have to switch it too…to mariaDB-connector…

stindl · January 19, 2019, 9:39am

I was searching for relevant code in the superfecta files…this looks promising to me…
superfecta_base.php

	function stripAccents($string) {
		$string = html_entity_decode($string);
		$string = strtr($string, "äåéöúûü•µ¿¡¬√ƒ≈∆«»… ÀÃÕŒœ–—“”‘’÷ÿŸ⁄€‹›ﬂ‡·‚„‰ÂÊÁËÈÍÎÏÌÓÔÒÚÛÙıˆ¯˘˙˚¸˝ˇ", "SOZsozYYuAAAAAAACEEEEIIIIDNOOOOOOUUUUYsaaaaaaaceeeeiiiionoooooouuuuyy");
		$string = str_replace(chr(160), ' ', $string);
		return $string;
	}

	function isutf8($string) {
		if (!function_exists('mb_detect_encoding')) {
			return false;
		} else {
			return (mb_detect_encoding($string . "e") == "UTF-8"); // added a character to the string to avoid the mb detect bug
		}
	}

	function _utf8_decode($string) {
		$string = html_entity_decode($string);
		$tmp = $string;
		$count = 0;
		while ($this->isutf8($tmp)) {
			$tmp = utf8_decode($tmp);
			$count++;
		}

		for ($i = 0; $i < $count - 1; $i++) {
			$string = utf8_decode($string);
		}
		return $string;
	}