Dear all, Superfecta does not seem to work anymore with Austrian/German databases (FreePBX 16.0.40.10 / Asterisk 20.5.2).
These are the error messages:
Debug is on and set at level: 1
The DID: 5555555555
The Original Number: +49 30 65212888
Das Schema: Oesterreich
Scheme Type: SINGLEFECTA
Debugging Enabled, will not stop after first result
Scheme Asked is: Oesterreich
The DID is: 5555555555
The CNUM is: +49 30 65212888
The CNAME is: CID Superfecta!
Starte Schema Oesterreich
Executing FreePBX Contactmanager
result took 0.2925 seconds.
Executing Asterisk Phonebook
Searching Asterisk Phonebook ...
not found
result took 0.0459 seconds.
Executing Herold Austria
Searchinghttps://www.herold.at/ ...
The requested URL returned error: 400 Bad Request
The requested URL returned error: 400 Bad Request
result took 0.4248 seconds.
Executing TelefonABC Austria
Searchinghttp://www.telefonabc.at ...
The requested URL returned error: 400 Bad request: request protocol version denied
result took 0.1145 seconds.
Executing DasTelefonBuch Germany
Searchinghttps://www.dastelefonbuch.de ...
The requested URL returned error: 410 Gone
result took 0.2980 seconds.
Executing DeTeMedien DE
Searching http://www.detemedien.de/ for +49 30 65212888...
The requested URL returned error: 404 Not Found
result took 0.6558 seconds.
Executing Infobel
Searching Infobel...
None : country source not found
result took 0.0014 seconds.
I can confirm that Source-Herold.at is broken (again).
The URL for the search changed, it is now:
$res_rul = "https://www.herold.at/telefonbuch/suche/?term=".$thenumber."/";// url for searching residential listings
$bus_url = "https://www.herold.at/gelbe-seiten/suche/?term=".$thenumber."/";// url for searching business listings
They also changed the code…and that’s the difficult part. When you search for a residential number, it gives you something like this for first and last name: <meta itemProp="givenName" content="Christian" q:id="2a"><meta itemProp="familyName" content="Müller" q:id="2b">
or <h2 class="font-satoshi text-xl font-bold md:text-2xl"><!--t=2i-->MĂĽller Christian<!----></h2>
…but I dont know how to extract the information…
The problem is even more sophisticated with the second URL (business). Herold shows you four recommended entries, which have nothing to do with the number you searched for. Only the last entry shows you the correct result.
I have no clue …but it seems I somehow fixed the source Herold.at for residential numbers. The business url cannot be used anymore, it always gives you 4 recommended entries in front of the actual search result…so I don’t know how to fix that.
Many companies are listed in the Telefonbuch (residential numbers) anyway…so it is ok…at least for me.
source-Herold_Austria.module
/*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
* Developer Notes:
*
* herold.at Terms of Service:
* As summarized in the post here on 2014-06-27
* https://github.com/POSSA/Caller-ID-Superfecta/issues/131#issuecomment-47354154
* herold.at TOS do not explicitly prohibit automated lookups
*
* Version History:
* 2014-06-28 Initial migration from 2.2.x
* 2014-07-02 Add business lookups and change urls to mobile site
* 2016-05-03 Rewrite Numbers correct for Herold and change Query for new Mobile Web Template
* 2017-03-20 Again Herold has a new Theme. We need to change the regexp again
* 2017-11-09 If the Caller ID is 'unknown', 'anonymous' or empty then skip the check
* 2020-09-12 update regex per https://community.freepbx.org/t/superfecta-herold-austria/42997/15
* 2022-06-10 general update per https://community.freepbx.org/t/superfecta-herold-austria-not-working-anymore-they-changed-the-code/83830
* 2024-09-22 quick & dirty fix for residential numbers, removal of the business url&code by Charles_Darwin, freePBX community
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***/
class Herold_Austria extends superfecta_base {
public $description;
public $version_requirement = "2.11";
public function __construct() {
$this->description = "https://www.herold.at/ - "._("These listings include data for Austria.");
}
function get_caller_id($thenumber, $run_param=array()) {
$this->DebugPrint(_("Searching"). "https://www.herold.at/ ... ");
if($thenumber !== 'anonymous' && $thenumber !== 'unknown' && $thenumber !== ''){
if (substr($thenumber, 0, 1) ==! '0') {
$thenumber=trim($thenumber,' ');
$thenumber="00" . $thenumber;
}
if (substr($thenumber, 0, 2) === '00') {
$thenumber=trim($thenumber,' ');
}
// Set the urls we're searching for
$res_rul = "https://www.herold.at/telefonbuch/suche/?term=".$thenumber."/";// url for searching residential listings
// regex patterns to search for
$regexp = array(
'~<h2 class="font-satoshi text-xl font-bold md:text-2xl"><!--t=2l-->(.+?)<!----></h2>~',
'~<h2 class="font-satoshi text-xl font-bold md:text-2xl"><!--t=2k-->(.+?)<!----></h2>~',
'~<h2 class="font-satoshi text-xl font-bold md:text-2xl"><!--t=2i-->(.+?)<!----></h2>~'
);
// first search for Residential match
if ($this->SearchURL($res_rul, $regexp, $match)) {
$caller_id = $this->ExtractMatch($match);
}
$caller_id = isset($caller_id)?$caller_id:'';
return($caller_id);
}
}
}
Since most of our calls come from other businesses, could you guide me on how to get a temporary result using the Business URL, with the 4 recommended entries appearing before the actual result?
Wir können auch deutsch reden. Das Problem ist, dass Herold scheinbar nicht will, dass wir auf ihren Server zugreifen.
Der Code ist nicht das Problem. Das Script sucht nach einen eindeutigen Eintrag und extrahiert dann den Text. Problem ist aber, Herold scheint keine eindeutigen Einträge mehr bereitzustellen. Da wechseln irgendwelche Variablen…entweder jede Woche (beim Telefonbuch) oder bei der Businessseite werden irgendwelche Zufalls-IDs vergeben.
Kenn mich ĂĽberhaupt nicht aus mit Programmierung, diese Skripte sind aber relativ simple, z.B.
sucht nach dem Eintrag h2 class="font etc und nimmt dann nur den Text (.+?) der dort an dieser Stelle steht.
Ich habe mit Firefox nach einer Telefonnummer bei Herold.at gesucht und dann klickst du mit der rechten Maustaste auf den Hintergrund der Seite…und wählst “Quelltext anzeigen” oder so ähnlich. Den gesamten Quelltext kopierst du in einen Editor und suchst dort nach dem Namen. Das hat bisher bei Herold immer funktioniert, aber dieses Mal brauchts wohl einen Experten Wenn du dir die obige Zeile anschaust, dann steht dort t=2l…vor einer Woche stand bei einer Suchanfrage t=2i…bei einer anderen Suchanfrage gestern stand t=2k. Ich befürchte, dass Herold hier eine Variable eingefügt hat, die immer wechselt. Bisher hab ich nur i,k,l berücksichtigt.
Here is an improved version of the Herold.at source for residential numbers (Telefonbuchsuche, white-pages) only.
source-Herold_Austria.module
<?php
/*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
* Developer Notes:
*
* herold.at Terms of Service:
* As summarized in the post here on 2014-06-27
* https://github.com/POSSA/Caller-ID-Superfecta/issues/131#issuecomment-47354154
* herold.at TOS do not explicitly prohibit automated lookups
*
* Version History:
* 2014-06-28 Initial migration from 2.2.x
* 2014-07-02 Add business lookups and change urls to mobile site
* 2016-05-03 Rewrite Numbers correct for Herold and change Query for new Mobile Web Template
* 2017-03-20 Again Herold has a new Theme. We need to change the regexp again
* 2017-11-09 If the Caller ID is 'unknown', 'anonymous' or empty then skip the check
* 2020-09-12 update regex per https://community.freepbx.org/t/superfecta-herold-austria/42997/15
* 2022-06-10 general update per https://community.freepbx.org/t/superfecta-herold-austria-not-working-anymore-they-changed-the-code/83830
* 2024-09-22 quick & dirty fix for residential numbers, removal of the business url&code by Charles_Darwin, freePBX community
* 2024-09-23 update by C. Darwin
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***/
class Herold_Austria extends superfecta_base {
public $description;
public $version_requirement = "2.11";
public function __construct() {
$this->description = "https://www.herold.at/ - "._("These listings include data for Austria.");
}
function get_caller_id($thenumber, $run_param=array()) {
$this->DebugPrint(_("Searching"). "https://www.herold.at/ ... ");
if($thenumber !== 'anonymous' && $thenumber !== 'unknown' && $thenumber !== ''){
if (substr($thenumber, 0, 1) ==! '0') {
$thenumber=trim($thenumber,' ');
$thenumber="00" . $thenumber;
}
if (substr($thenumber, 0, 2) === '00') {
$thenumber=trim($thenumber,' ');
}
// Set the urls we're searching for
$res_rul = "https://www.herold.at/telefonbuch/suche/?term=".$thenumber."/";// url for searching residential listings
// regex patterns to search for
$regexp = array(
'~<h2 class="font-satoshi text-xl font-bold md:text-2xl"><!--t=[0-9][a-z]-->(.+?)<!----></h2>~'
);
// first search for Residential match
if ($this->SearchURL($res_rul, $regexp, $match)) {
$caller_id = $this->ExtractMatch($match);
}
$caller_id = isset($caller_id)?$caller_id:'';
return($caller_id);
}
}
}
Clearly…Herold does not want us to use their database. The yellow-pages (gelbe-seiten) subpage has a built-in 3s delay. I adapted the code…for the business numbers too…but the problem is it takes more than 3s to get the name for a business number.
source-Herold_Austria.module (residential & business numbers working)
<?php
/*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
* Developer Notes:
*
* herold.at Terms of Service:
* As summarized in the post here on 2014-06-27
* https://github.com/POSSA/Caller-ID-Superfecta/issues/131#issuecomment-47354154
* herold.at TOS do not explicitly prohibit automated lookups
*
* Version History:
* 2014-06-28 Initial migration from 2.2.x
* 2014-07-02 Add business lookups and change urls to mobile site
* 2016-05-03 Rewrite Numbers correct for Herold and change Query for new Mobile Web Template
* 2017-03-20 Again Herold has a new Theme. We need to change the regexp again
* 2017-11-09 If the Caller ID is 'unknown', 'anonymous' or empty then skip the check
* 2020-09-12 update regex per https://community.freepbx.org/t/superfecta-herold-austria/42997/15
* 2022-06-10 general update per https://community.freepbx.org/t/superfecta-herold-austria-not-working-anymore-they-changed-the-code/83830
* 2024-09-24 fixed residential & business numbers by Charles_Darwin, freePBX community, yet herold gelbe-seiten page has a built-in 3s delay
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***/
class Herold_Austria extends superfecta_base {
public $description;
public $version_requirement = "2.11";
public function __construct() {
$this->description = "https://www.herold.at/ - "._("These listings include data for Austria.");
}
function get_caller_id($thenumber, $run_param=array()) {
$this->DebugPrint(_("Searching"). "https://www.herold.at/ ... ");
if($thenumber !== 'anonymous' && $thenumber !== 'unknown' && $thenumber !== ''){
if (substr($thenumber, 0, 1) ==! '0') {
$thenumber=trim($thenumber,' ');
$thenumber="00" . $thenumber;
}
if (substr($thenumber, 0, 2) === '00') {
$thenumber=trim($thenumber,' ');
}
// Set the urls we're searching for
$res_rul = "https://www.herold.at/telefonbuch/suche/?term=".$thenumber."/";// url for searching residential listings
$bus_url = "https://www.herold.at/gelbe-seiten/suche/?term=".$thenumber."/";// url for searching business listings
// regex patterns to search for
$regexp = array(
'~<h2 class="font-satoshi text-xl font-bold md:text-2xl"><!--t=[0-9][a-z]-->(.+?)<!----></h2>~',
'~<meta itemProp="position" content="5" q:id="6[a-z]"><meta itemProp="name" content="(.+?)" q:id="6[a-z]">~'
);
// first search for Residential match
if ($this->SearchURL($res_rul, $regexp, $match)) {
$caller_id = $this->ExtractMatch($match);
}
// if no residential match found, search business
if ($this->SearchURL($bus_url, $regexp, $match)) {
$caller_id = $this->ExtractMatch($match);
}
$caller_id = isset($caller_id)?$caller_id:'';
return($caller_id);
}
}
}
Unfortunately, the Yellow Pages feature is not functioning as expected. I tested it with a number that exists only in the Yellow Pages, and the result returned the code “t=2e”, which should be considered in your updated code.
However, I’m not clear on the 3-second delay - could you clarify if this presents a problem for FreePBX?
“t=2e” would be ok, if it is the result of a search of the white-pages. Are you sure? Did you use Firefox for your test? How does the url in the address field look like? Does it contain “telefonbuch” or “gelbe-seiten”? If my code is correct and you want to search the yellow-pages, you would have to increase the lookup timeout of the superfecta scheme to 3.5s.
Edit: There is a second timeout value too…you have to increase it as well.
The code for the yellow pages is ok…there is another problem…in addition to the 3s delay, Herold_gelbe-seiten seems to block multiple requests from the same IP. You have to wait several minutes, until it works again. When I do a debug, all of a sudden there is no html code, the two windows are empty. I may be wrong, but in this case, gelbe-seiten is not useable anymore, just the white-pages (Telefonbuch).
EDIT: Yes, there is something weird with the yellow-pages…it suddenly returns no html code…and after some time, it works again. So I keep the residential-numbers-only template
I was able to get the name for the number you suggested, but it only worked once or twice. Anyways…a delay of 3s for a reverse lookup is not good and Herold somehow blocks the access to the yellow-pages. When you do a superfecta debug, you get two empty windows with no html code and no response from Herold.
Maybe someone else can fix it…I don’t
On my machines I now created a separate file, Herold_Austria_wp.module, and put it in the Superfecta source folder. The advantage is that freePBX does not recognize it as an unsigned module. You can easily choose this file in the Superfecta scheme settings. Works well with white pages of Herold Austria.
<?php
/*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
* Developer Notes:
*
* herold.at Terms of Service:
* As summarized in the post here on 2014-06-27
* https://github.com/POSSA/Caller-ID-Superfecta/issues/131#issuecomment-47354154
* herold.at TOS do not explicitly prohibit automated lookups
*
* Version History:
* 2014-06-28 Initial migration from 2.2.x
* 2014-07-02 Add business lookups and change urls to mobile site
* 2016-05-03 Rewrite Numbers correct for Herold and change Query for new Mobile Web Template
* 2017-03-20 Again Herold has a new Theme. We need to change the regexp again
* 2017-11-09 If the Caller ID is 'unknown', 'anonymous' or empty then skip the check
* 2020-09-12 update regex per https://community.freepbx.org/t/superfecta-herold-austria/42997/15
* 2022-06-10 general update per https://community.freepbx.org/t/superfecta-herold-austria-not-working-anymore-they-changed-the-code/83830
* 2024-09-25 fixed residential numbers by Charles_Darwin, freePBX community, removed yellow-pages
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***/
class Herold_Austria_wp extends superfecta_base {
public $description;
public $version_requirement = "2.11";
public function __construct() {
$this->description = "https://www.herold.at/ - "._("These listings include data for White Pages Austria.");
}
function get_caller_id($thenumber, $run_param=array()) {
$this->DebugPrint(_("Searching"). "https://www.herold.at/ ... ");
if($thenumber !== 'anonymous' && $thenumber !== 'unknown' && $thenumber !== ''){
if (substr($thenumber, 0, 1) ==! '0') {
$thenumber=trim($thenumber,' ');
$thenumber="00" . $thenumber;
}
if (substr($thenumber, 0, 2) === '00') {
$thenumber=trim($thenumber,' ');
}
// Set the urls we're searching for
$res_rul = "https://www.herold.at/telefonbuch/suche/?term=".$thenumber."/";// url for searching residential listings
// regex patterns to search for
$regexp = array(
'~<h2 class="font-satoshi text-xl font-bold md:text-2xl"><!--t=[0-9][a-z]-->(.+?)<!----></h2>~',
);
// first search for Residential match
if ($this->SearchURL($res_rul, $regexp, $match)) {
$caller_id = $this->ExtractMatch($match);
}
$caller_id = isset($caller_id)?$caller_id:'';
return($caller_id);
}
}
}
Wenn du dir die obige Zeile anschaust, dann steht dort t=2l…vor einer Woche stand bei einer Suchanfrage t=2i…bei einer anderen Suchanfrage gestern stand t=2k. Ich befürchte, dass Herold hier eine Variable eingefügt hat, die immer wechselt. Bisher hab ich nur i,k,l berücksichtigt.
