what version of FreePBX are you using? They should get automatically cleared in the hangup context. On a rare occasion an entry can get left in but if you have that many there might be an issue with your system (or version) that it is not getting cleared. As far as removing them, you should delete the entire tree when the system is idle.
I would have to see a lot more detail on how your system is setup, how calls come in, any customizations you might have, etc. It could be a bug that results in some usage pattern where the flag is not being deleted or related to something you have setup on your system. Without a close analysis I could not say more. There is no significant harm done as long as you trim it occasionally. You can execute the dbdeltree command above from a cron job nightly or weekly or you can dig into the system to see why they are being left. It is deleted in macro-hangupcall (if not deleted by the queue dialplan code itself), and in a few other places.