Possible new source to block "bad" numbers (telemarketers, etc.)

If you are tired of getting “expiring vehicle warranty” scam calls and similar total wastes of your time, you might find this page interesting:

http://www.everycall.us/top_100_phone_number_comments

At this page you can download a list (which is updated daily) that lists the top 100 commented numbers on this site (which usually indicates the number is associated with “junk calls” of some type). The list is in .csv format, which I assume means it could be directly loaded into a database.

Now, what I sort of wish is that someone who has figured out how to write FreePBX modules would create one that would have the ability to download this list nightly, optionally weed out the listings that are not in NANP format, and insert the numbers into a database along with the current date. Numbers that already in the database should be overwritten (so the date is refreshed), not duplicated. This list would then act as a secondary blacklist, and if a call comes in with one of the Caller ID’s from this list it would get the blocked call treatment. When the module does its nightly update, it would also check the database for any numbers that haven’t been “seen” on the list in the past n days or months (can be set by the user; I’d default to six months or 180 days) and delete those, so you don’t wind up with a database that is years out of date.

There are several other services that offer lists of this type; I’m not saying this is the only one or even necessarily the best.

I’m pretty sure I could do this in perl and shove it into a _custom.conf file pretty easily. Not so familiar with modules, though.

Also, if you trust the list you can just ditch the old one and re-read in a new one every time, no need to have it try to remember anything.

Hello everyone, my name is Ben and I own EveryCall.us. I was looking at my server logs and saw a couple of visits from freepbx.org and I thought I would check it out. I think your idea is a great one and I wanted to let you know that I would actively support any projects that you guys wanted to take on to configure some kind of active call blocking into the PBX software.

I could create another CSV file that is updated nightly with all of the telephone numbers and a score, then the PBX can be configured to block any number that are higher than the user defined allowable score. Or I could write a script that would allow for active passing of telephone numbers from the PBX to EveryCall.us through the web and return a score.

What do you guys think? Feel free to email me if you’d like, I think it would be a lot of fun to come up with something that really works well, my email address is admin [at] everycall.us

Ben

Ben, sorry for not responding sooner, but here is my thought. Let’s suppose you could do a data dip in real time. For example, the incoming caller ID is something like 8005552368 (which could also be in the format 18005552368, or even +18005552368). It would then be sent to you as part of a HTTP request such as this:

http://www.everycall.us/query?8005552368 (<— This is just a sample, any format that would work is fine as long as the number could be sent as part of the HTTP request).

The “page” returned should probably be one very simple line of plain text (or maybe two, see below). It should contain a score OR some other character(s) to indicate that the number is not in one of the allowable formats (meaning it might be an international call). However, if the number is not in a usual format but has still been associated with “spam” calls then a score should be returned anyway, if you have that information in your database.

The biggest issue would be that the server would have to be reasonably fast, so that call processing is not delayed significantly. You may also want to somehow keep records of the number of data dips per month sorted via IP address so that if anyone is abusing the service, you can block their IP address and/or request they make some contribution toward your server expenses. If all you are returning is a single line of text containing a one or two digit score then this may never become an issue, but still it seems that whenever anyone offers a free service there is always someone else who will figure out a way to abuse it. And then there’s also the issue that someone’s bad code might get stuck in a loop and continually make the same request every second or two for hours on end.

Anyway, the single line of text should be reasonably easy to handle in an AGI perl script. The biggest issue in FreePBX would be figuring out how to intercept the incoming call and then direct it based on the result of the data dip. I can think of at least one way to do it - change the context in a trunk definition from “from-trunk” to a new context in extensions_custom.conf, then in that custom context call the AGI script and based on what that script returns, either pass the call along to the normal “from-trunk” context or send it elsewhere (maybe a congestion tone). This is not something I could whip out in ten minutes but on the other hand it’s probably fairly trivial to code (especially for anyone who regularly writes code in perl or php). The biggest concern would be making sure that in the AGI script there is a short timeout on the HTTP request, so that if your site is cannot be accessed for some reason the call flow is not delayed by more than a second or two.

If your service were to become really popular than maybe the developers would consider adding a field on the inbound route configuration pages that would optionally do the data dip and then send the call to a “black hole” (or some other alternate) destination if the score were above a specified level.

Of course, in my ideal world your service would return two lines of plain-text information: the score as discussed above on the first line, and a best guess at the Caller ID name on the second line. That way, if the Caller ID name field on the incoming call were already populated we could ignore the second line, otherwise we could actually use it to provide some information about the call. The bonus is that for known spam callers you might actually associate a company name (or some other descriptive information about the type of call, such as “WARRANTY SCAM”, although you’d have to be very careful about doing something like that so as not to get sued for misidentifying a legitimate company), so that the name or information could be made to appear in the system log even if the original incoming Caller ID name were blank. I’m obviously not saying you should go paying the phone companies to do a real Caller ID data dip, but for numbers you know nothing about you could always return the state or province associated with the area code, like some cell phone companies do (e.g. NEW YORK CALL). But, that second line might add complexity and would also increase the load on your servers, so it’s entirely up to you if you’d want to do something like that.

I’ve been checking this thread once a week or so and I’m glad that you were able to respond. I really like your ideas and I think that we could come up with something very useful pretty quickly. I see this happening in two (or more) phases, phase 1 would be getting the score function going on EveryCall.us and then phase 2 would be working on returning the caller name if it exists in my database. Of course there will probably be a host of other great ideas that come up as we work on this and I’m pretty open to help developing anything that makes the freepbx product better using EveryCall.us.

I wanted to make sure and point out that I have no knowledge about the FreePBX product other than a cursory understanding of how it works. I propose that the programming of the FreePBX software be done by you or others interested in contributing, and then myself and my team would work on anything on the EveryCall.us side of the equation. Does that make sense? Are their other developers that would be willing to contribute to the FreepPBX side of things?

Also regarding the potential for load on my servers, can you give me an idea of how many freepbx systems are out there and how many might use the spam interface that we’re discussing?

For now I will get working on returning a score for queried numbers with my developers. I’d be interested in getting your thoughts on calculating the score, variables to consider, etc.

It is better for me to communicate via email admin [at] everycall.us, however I’ll check back to this forum to see if you’ve responded.

Just to be clear, I don’t really consider myself a programmer, but now and then I will try to hack together a short code segment in Perl. It’s difficult because I use Perl so infrequently that I forget most of what I know about it from one time to the next (too bad I didn’t learn it when I was in my 20’s; I still remember all the old BASIC commands and syntax I learned back then). Nevertheless, I THINK I could make this work through an AGI perl script, since I have done things that are fairly similar in other code segments. What I cannot do (but perhaps someone else could) is properly integrate it into FreePBX. For whatever reason, the FreePBX developers have pretty much enforced a defacto requirement that all “official” FreePBX modules be written in PHP, which I don’t know at all (the closer a language gets to “C”, the less likely I will ever touch it with the proverbial ten foot pole - my brain just can’t wrap itself around some of the newer languages).

However, using an AGI script is not necessarily a bad thing; there are several other AGI scripts (both Perl and PHP, and possibly in other languages as well) floating around that are successfully used with FreePBX. And if this becomes popular I suspect that the functionality could fairly easily be added to the inbound route module, or a placed in a separate FreePBX module, if someone with more programming experience than I cares to do it.

The only real downside of using a Perl script is that you wind up calling Perl modules, and almost invariably one or more of those will not installed on the machine that runs FreePBX. So the user has to install the Perl module(s), which is easy enough to do using yum (in CentOS) or Webmin, and probably second nature for anyone who’s ever used Perl. But it does mean that it would not be as easy as installing the typical FreePBX module (not point-and-click, in other words).

On the flip side, you can do what needs to be done on your end by simply making the http request work. You wouldn’t really need to know anything about FreePBX or Asterisk at all. But you would need to have the http requests working on your end before any code could be written to utilize them (obviously).

I cannot comment on how many people might actually use this. Clearly a lot more would use it if the ability to make the http request were incorporated into an “official” FreePBX module, but I can’t predict how many people would take the 15 minutes or so necessary to install an AGI script (and any associated Perl modules) and add some lines to extension_custom.conf and make a couple other tweaks.

As far as calculating the score, I guess I don’t know enough about that to really comment. Obviously you’d want to avoid “false positives” and also people reporting a number for malicious reasons (for example, members of one political party reporting the robo-callers of the other party - while those are definitely a nuisance and many of us would want to block those calls anyway, you can see how that’s in a different class from a warranty scam call). So the numbers really fall into two categories - “telephone spam” and scams, and then what we might call “annoyance” calls, which are technically neither spam nor scams. Annoyances might include political calls, solicitations from charities (the ones not known to be fake) and perhaps similar types of calls. I’d give the spam/scams a higher score than the “annoyance” calls, but the annoyances should definitely return a non-zero value (I, for one, don’t want either type of call, but maybe there is someone who really wants those annoying political calls that are mostly lies anyway). With that said, maybe others might have some ideas on scoring. I’d like to see some input from other people in this thread anyway; if only two or three people have any interest in this then I don’t know if it’s worth our time to try and make it work. That’s also why I’m posting publicly rather than going to e-mail - I keep hoping someone else will chime in on this thread.

Is anyone interested in assisting with developing the FreePBX side of this? If so please comment in the forum.

www.whocalled.us. The AGI code even on their webpage.
I use it and it works great.

I’m not knocking the everycall guys… The more the merrier.

Not only do they require registration to use their scripts, but you have to send your user name and password (in the clear) with every lookup request made via their AGI script. So they could, if they wanted to, build a database of every number you’ve looked up (which for many people, might mean every incoming call). If you didn’t like Trixbox’s “phoning home” trick then you probably should not be in any big rush to use whocalled.us - it’s about the same principle in my book.

And, the AGI scripts they provide are intended for use with plain Asterisk, and would doubtless have to be tweaked for use with FreePBX. Anyway, if I were going to use a service like this, I’d prefer one that does not require registration details to be sent with every lookup (also, when milliseconds count, I wonder how long it takes them to verify the login before they return the data?).

The call tracking part bothers me too.
Another problem is that there is no mechanism to remove a number from their list if there was a mistake.
There is also NO contact link to contact the website maintainer. None.

However, I am using it with FreePBX. And the lookup speed is so fast I can’t tell it is happening. And it WORKS!

I really enjoy looking at the log files and seeing all the vermin who tried to get through…
I remember a time when I got 32 telemarketers a day. Now, its about 2 a year.

I love the smell of telemarketers hitting the blacklist in the morning.

I have to ask, how did you interface it with FreePBX? No sense in everyone else trying to reinvent the wheel, and your method might be easily adaptable to the other service (if they ever get their mods in).

I always figured this should be handled Upstream (providers)

With VOIP it should be a no brainer…
We should all ask our providers to allow us to opt in to a blacklist
no need to worry about the lookup time nor the ‘tracking’ as they do that now.

I know many folks who would pay to have this feature handled upstream
I know folks who port out POTS lines to a VOIP provider who could offer this, just because of those “warranty” calls…

My dialplan is very complex, but I cut out a snippet and shortened it to show just a few lines related to calling WhoCalled.US.
Have your inbound route go to “custom-spam-lookup”.
Install “junk_lookup.agi” in your agi bin folder, and set the userid and password.

[custom-spam-lookup] exten => s,1,GotoIf(${DB_EXISTS(cidname/${CALLERID(NUM)})}?whitelisted) exten => s,n,AGI(junk_lookup.agi|${CALLERID(number)}|"${CALLERID(name)}") exten => s,n,NoOp(whocalled.us returned spammer set to ${spammer}) exten => s,n,GotoIf($["${spammer}" = "yes"]?blacklisted) exten => s,n(whitelisted),Goto(app-daynight,0,1) exten => s,n(blacklisted),Answer exten => s,n,Wait(1) exten => s,n,Zapateller() exten => s,n,Playback(ss-noservice) exten => s,n,Hangup

That’s great that whocalled.us offers a similar kind of service to their database, I was unaware of that. However I think there are many improvements that can be made if we’re up for it. I agree that having to send a username and password is not a good thing, although it does prevent abuse. My thinking is to require a registration and use some other form of checking the registration other than a username/password situation, maybe a secret code or something. It is a good thing to prevent abuse, keeps the system fast, lessens bandwidth usage, etc.

I also think that each local PBX system should have an allow list, so that even if a call is flagged as spam in the EveryCall.us database, it will be allowed to prevent against false positives. I’d also like to see an easy way to report the call to EveryCall.us, maybe #EVERYCALL or something similar will report the telephone number on the caller id to EveryCall.us?

I’m currently working on the developing the scoring system, shouldn’t be but a couple of weeks before I have something up and running. Meanwhile any comments you guys have would be great.

It shouldn’t take weeks. It’s just a couple of lines of code.

  1. You provide a URL for user to lookup a number. The results of that http get should be a number, the number would be how many people reported them as a telemarketer. If I get a zero back, I let the call pass. 1 or more, the call goes to Jordan… He’ll play with them as long as they keep talking. (Jordan is a BOT).
    The URL could look like: http://www.everycall.us/query?2025551212
    It would simply return the number of hits.

  2. The allow list. Better known as a “whitelist”. I use my asterisk phonebook as the whitelist. Anyone I talk to regularly is in my phonebook, primarily so the callerid name displays correctly (instead of just saying ANNAPOLIS MD, for example).
    In my context, I just see if the incoming number is in my phonebook. If it is, I skip the blacklist lookup and pass the call.
    Example:

    exten => s,1,GotoIf(${DB_EXISTS(cidname/${CALLERID(NUM)})}?whitelisted)

There is nothing to it. Done.

Oh, and you MUST use a username and password to register telemarketer phone numbers. And provide a mechanism to dispute mistakes.
Otherwise it will be abused to the point of being useless in a matter of hours.

Basildane, thanks for the feedback. I will definitely incorporate an authentication system to prevent abuse, I agree that without it there is a large potential for the system becoming unusable.

Regarding the scoring system, I don’t think it is adequate enough to return a simple 0 or 1 result for your system to make a decision on. Here’s why.

Anyone in the world can go to EveryCall.us and report a telephone number. So let’s say that someone reports your cell number to the database as a telemarketer, a simple 0 or 1 result would incorrectly classify your number as a spam number.

What I am trying to do is to come up with a more intelligent classification system that will score each telephone number based upon a number of different factors that we can derive from the report data we have to eliminate false positives as much as possible.

The other advantage with a 0-100 score is that now you can setup a low, medium, high threat level in your PBX system so that you can then play with the tolerance levels of the scores to fine tune calls that are blocked. For instance, a number with only one call report from two years ago probably isn’t going to be a threat, whereas a number that has been reported 425 times in the past week should definitely be blocked.

Hence why it’s going to take a week or two to get the scoring mechanism in place, but once we do I think it will be a very powerful tool which will better allow your system to intelligently block spam telephone numbers.

What do you think?

But to return the number of hits, then we can make a decision ourselves. For example, some may want to block if > 10.
Anyway, I agree with what you are saying, and I can offer my help. At the least I’ll test stuff for you. But I am an experienced software developer too.

Anything that takes a hit at telemarketers is good in my book. Let’s get to it!

I misunderstood you then, we’re on the same page :slight_smile: I’d welcome any assistance you’re willing to give, I should have something ready tomorrow or at the latest Friday… can you email me your contact info so that I can give you access to test our service? admin [at] everycall.us.

Thanks!

Just came across this one, haven’t had a chance to really look it over yet…

PhoneSpamFilter.com [Report and Stop Annoying Telemarketers]

Installing PhoneSpamFilter on Your Asterisk PBX

No idea how it compares to the others, just thought I’d mention it as a possible additional option.

Server CONSTANTLY busy and returned errors.
Maybe they fixed it, but I already switched.