High Load Average on Transcoding sip/alaw ulaw to g729/iax

I am getting a strange problem with load average climbing to 30 or more and asterisk cpu at 97% with only one call in progress. The problem occurs when I have a call from a Sip phone (Snom) set as alaw or ulaw, with the other phone iax set to g729. If I try the same thing between two phones, both sip, but one alaw and the other g729, then no problem.

I am running Ubuntu 8.04 server (no gui), 512Mb ram, Pentium 4 2.4Ghz, SMP kernel, but HT disabled. Asterisk is 1.4.21.2, zaptel (ztdummy only) is 1.4.12. I have compiled ilbc into the asterisk. I am using FreePBX 2.5.1

Here is a top:
op - 08:52:03 up 32 min, 3 users, load average: 31.89, 7.43, 2.48
Tasks: 72 total, 2 running, 70 sleeping, 0 stopped, 0 zombie
Cpu(s): 19.0%us, 81.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 507320k total, 196480k used, 310840k free, 14768k buffers
Swap: 1485972k total, 0k used, 1485972k free, 77888k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4684 asterisk 20 0 51908 15m 6532 S 99.9 3.1 1:27.25 asterisk
4391 mysql 20 0 125m 17m 5096 S 0.7 3.6 0:02.07 mysqld
4708 asterisk 20 0 11244 8460 1784 S 0.3 1.7 0:09.19 op_server.pl
4872 root 20 0 2308 1080 856 R 0.3 0.2 0:00.48 top
4905 root 20 0 2308 1084 856 S 0.3 0.2 0:00.38 top
1 root 20 0 2844 1692 548 S 0.0 0.3 0:01.89 init
2 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/0
4 root 15 -5 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/0
5 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
6 root 15 -5 0 0 0 S 0.0 0.0 0:00.02 events/0
7 root 15 -5 0 0 0 S 0.0 0.0 0:00.01 khelper
41 root 15 -5 0 0 0 S 0.0 0.0 0:00.01 kblockd/0
44 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kacpid
45 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kacpi_notify
115 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kseriod
149 root 20 0 0 0 0 S 0.0 0.0 0:00.00 pdflush