eZZ,
You can only specify one default gateway for the system.
Use the command ‘netstat -rn’ to see where things are assigned.
Per your post ..
eth0 is 192.168.180.5 Mask 255.255.255.0 No gateway..
eth1 is 192.168.31.130 Mask 255.255.255.0 gateway 192.168.31.254
Traffic to Phones and Devices on 192.168.180.n will go out eth0. Traffic to Phones and devices on 192.168.31.n will leave via eth1. Traffic to any other network device will leave via 192.168.31.130 then route to the internet or other local networks through your router gateway at 192.168.31.254..