Threads at the top of the forum right now are concerned about broken fail2ban and firewall modules. Given that these are related to security, this should be seen by everyone as a fairly big deal.
That said it is not on the shoulders of an individual developer who makes changes to the firewall module to test every possible scenario. (in my opinion) Especially community contributors.
Bugs like what are being reported right now should be found and squished during edge phase but what incentive does anyone have to work in edge mode when they simply want a working PBX?
Just putting this topic out there to fish for some ideas.
There is a much larger QA discussion that is out of our hands unfortunately.
I have only skimmed the fail2ban issue but if it is happening to multiple people that isnât an edge case.
There is basic functionality that has to be maintained and that is what needs to be tested.
The development model in FreePBX is generally focused on internal development so the QA has to be the same. When you (sangoma/schmooze/bandwidth/colescent) take ownership and stewardship of a product you pretty much take ownership of documentation and testing for the project.
What their procedures for QA are now between them and god (or some other omnipotent being). Most development is now done behind the curtains as part of internal tickets. Frankly without transparency you canât possibly have community QA.
Hey a new version of Foo just dropped, lets see what changedâŚ
change log: FREEI-90210 Added Bar.
Well thatâs vague lets go look at FREEI-90210, oh that isnât public.
Well I am a smart guy lets go look at the difference between tags on git.
Well let me end to end test my entire universe because at this point it is probably faster than figuring out the divinci code.
tl;dr you need open development practices to have community QA. If a developer isnât going to put in the baseline effort to ensure their code doesnât break existing, expected functionality they should lose commit access.
I expect that all Sangoma developers have test plans. Right? (Right�)
Could those be shared on the wiki so that community members who want to do some development and contribute to FreePBX could also work through the test plans?
If I never use feature X but my change to module Y has somehow broken X, I donât feel that I would be responsible for that if I didnât have a test plan that covered the scenario.
The recent problem is firewall and that is purely a Schmooze/Sangoma issue with requiring SysAdmin when it is supposed to be AGPLv3+. Because SysAdmin is not, no one can look at crap there.
See the recent threads for examples of things in Firewall suddenly needing something in SysAdmin to be fixed. With @yois and or @jerrm coding half blind to try and make things better for the rest of us.
As of the time I left no. I was tasked to write them then they said the QA folks were going to do them and I was tasked with things that paid the bills. This happened when it came to doing a lot of necessary tasks that âdonât pay the billsâ so the execs donât give priority.
A community member could write test plans but leading a horse to water will not make it drink.
Some of this can be automated with unit testing. A lot of refactoring needs to be done and you have to convince the bean counters to allow the task. Again this could be community driven.
Not trying to be completely negative New code (not fixes) supervised by Matt Brooks which seems to be around graphql stuff for 16 is unit tested. So the devs that have worked on that should be generally comfortable with unit testing.
Also note the build tools do run unit tests when present and will fail to build if the tests fail. That was put in place about 4 years ago. Again without âgood testsâ it is a bit pointless.
FreePBX needs through official or community contribution
A ton of Unit Tests Written per module to cover basic functionality
A test plan written for each module that covers basic expected functionality and dependent functionality if known
This is super annoying. I had an old bug report for something in User Manager that was fixed. Never looked closer, until the other day.
I posted a new bug. Then had an idea for an improvement, so I posted a feature request for that. Then I recalled my old bug report and found out it was a FREEI not FREEPBX. Luckily, that wasnât a huge commit and I saw what changed and used that as a basis to submit my own PR on the FREEPBX ticket I just made.
This has been discussed in length. The only way to do what that module does is to use the hooks provided by sysadmin. The alternative would be to open source that part of sysadmin which simply wonât happen.
Can it be done without sysadmin? Yes, but baking it in would require an extra rpm/deb/other package and would give away some of the âsecret sauceâ
Iâm so happy this conversation is coming to light. As a community developer, I canât possibly test every use case, because I only think of the FreePBX project in the context that I use it.
But who then is responsible to QA? I think the community needs to get more involved⌠If something isnât working (like firewall that has been banning good IPs for what⌠Forever?) go out and fix it! If you donât watch JIRA and GIT, you are implicitly trusting those that are trying to fix your issues for FREE.
Just regarding Firewall specifically, QA is nearly impossible for a community developer since most of the code base will not run unless signed with FreePBX key. I have been unable to test my own patches without resigning myself to real hackery to even see if they will do what I want. I have to push to edge to get working versions that actually run. So itâs accurate that Iâm âcoding blindâ. Iâm not sure what can be done about this to encourage community involvement but this is a step than needs improvement.
Finally, on the topic of security, letâs just get the pink elephant on the table:
The security aspects of FreePBX have been completely neglected since @xrobau left Sangoma, and I suspect there is no one on staff that understands how any of this works. The fail2ban RPM hasnât been updated since late 2018 and is riddled with Asterisk filter bugs that have been patched upstream long ago, even on the 0.8 branch. There is absolutely no good reason why those changes cannot be incorporated today, unless thereâs a staffing issue at the top.
Sorry to anyone at Sangoma for being a little harsh, but security is more important to most of us than feature improvements⌠Letâs get the system safe and stable and only then add new features.
Slightly more than that but not rocket surgery. An aspiring rent a coder could figure it out⌠That said I am being purposely vague because for some of us there are always lawyers looking to make an hourly fee for what we sayâŚ
Thanks, not sure what is needed âslightly moreâ to manipulate iptables rules. but that is of course the âsecret sourceâ a lot of folks rely on for their sense of security.
Firewall is the heaviest OSS module using sysadmin user by far.
A âsysadmin-liteâ with just the ability to run hooks is a one day job. Such a module would open up firewall to non-distro users.
The problem is the only thread by which the hook scheme can claim to be the least bit secure is the combination of zend and module signing. Zend hasnât really been an option for non-distro installs.
Several OSS modules other than firewall use hooks, but for the most part they are limited to logrotate configuration. There are a couple of exceptions. Filestore needs a hook for ssh config, so ssh stores may be broken on non-distro installs(something to test). Xmpp installs some of itâs dependencies, something non-distro users are used to doing anyway.
There is no business case for this.
Sysadmin light is technically what you get when you activate. There are other features you pay for. In any case it would be a terrible business decision to allow people to move away from the distro.
Potentially, a âlighterâ sysadmin without all the system level config options, maybe just registration and hook support, could allow for (at least some) commercial modules on non-distro machines. Iâm mostly non-distro, but there are a few modules I would entertain if available non-distro. Itâs a (small) potential market expansion, but also big can of worms.
Community benefit at this point would otherwise be restricted to the firewall module. It would be pretty simple to get the fw module functional under debian and itâs derivatives.
My point was they will not dedicate engineering time without a business case. It is financially beneficial to keep people on the distro. So creating such a module is counter productive to the bottom line.
There is no altruism, all engineering cost must come with a business benefit
I agree with all the of the above. We are adding more and more unit testing to FreePBX, but the reality is that we really have to work around the limitations of a legacy project instead of doing what is best for each test. I see two things as really blocking us from really good Automated Unit testing in FreePBX:
There is limited test isolation in FreePBX. Itâs like this for multiple reasons. The way BOM was designed is part of the issue as it requires a huge framework to run and prevents us from being able to test small units. Secondly, all tests are run through the dev environment instead of a separate testing environment, which can mess up your config on your dev environment. I think I will go rogue one day to fix the latter, and hopefully this will give us some breathing room
The way we mange the project with git using release branches makes it rather difficult and complex to setup in CI and be able to use that in a release process. At some point, something will need to change, but the developers will need to be open for their process to change. Right now the only options are to increase the steps required to submit a change, which doesnât seem ideal. Weâre still exploring different ways to make some of the automatic merging and maybe it will work. I.E. PR submitted to 13 release needs to also auto merge into 14 and 15 and run the unit tests on ALL branches before allowing a PR to merge.
So the OSS firewall cant be worked on properly by contributors because it was designed to use SysAdmin. The BMO is an unwieldy module base that makes basic things hard to test. Both have one unifying factor. That factor might need to be addressed and see if it is the root cause.
Somewhere in the internal dev wikis I started documenting mocking BMO. I kind of wish a lot of the internal docs I did were public because they were more for me than anyone else.
You missed my point about isolation. We have no other option but to mock BMO in various places through out our unit tests. The problem comes down to dependency management. I want to test this widget that only requires the database as a dependency, but in order to test it, I have to load framework, all of the modules that I currently have installed on my system, and all of the other things that I have no idea what framework runs just to get it to run my test. Wouldnât it be nice to just have the database as the required dependency instead of having to pass this giant god object around that we have to mock?
I think youâve missed a design point of BMO - the major design of it is that itâs trivial to inject mocked components into it, specifically to make freepbx easier to test. The problem is, no one is deprecating and removing all the procedural code in functions.php any more. That I know of, anyway 8)
ton of Unit Tests Written per module to cover basic functionality