support multiple network access with a single NIC in VM

Nova assumes a single pool of public IPs, and allows the allocation and association of a public IP to a VM instance with a private IP. In reality, there might be multiple external networks that we'd like our VMs to be on. A typical example is that a data center might have connections from multiple ISPs, with different public IP blocks allocated. What is needed is to give a VM the flexibility of choosing which uplink to use for what kind of traffic. For example, send packets to a certain destination address block via a specifc ISP - for performance and/or billing reasons.

One way of handling this matter is to enable multi-NIC and place a VM onto different subnets. Indeed, this is a solution that is implemented in OpenStack. This however has a few drawbacks. In particular, having multiple NICs forces VM owners to manage routes inside of VMs, causing non-trivial management overhead and possible misconfiguration.

Another way, which I am proposing, is to take the routing management outside of VMs and into the nova network controller. This can leave the VMs with a single NIC and a simple default g/w, while enjoying the flexibility of multiple networks. The gist is that we define APIs at nova to allow project owners to assign multiple public IPs to a VM instance, and specify under what scenario to use which public IP (uplink/ISP.) This doesn't require any additional routing setup in the VM, potentially simplifies application development. Note that, this also can work well with multi-nic, because they are largely orthogonal.

The detailed change includes:
1) allow the specification of multiple public IP pools at nova level. Each pool may tie to a different interface available on the network controller (via vlans for example.)
2) allow an access control mechanism which rules whether a project is allowed to use a particular public IP pool.
3) allow the public IP allocation from different pools into a project.
4) allow the association of different public IPs (up to one from each pool) to a VM instance.
5) allow the specification of routing preference. For destination subnet X, NAT to public IP Y and use the proper uplink. For default, NAT to public Z. This should be done at API-level, not visible in the VM.
6) change nova-network: a) use iptables to DNAT into different public IPs based on the destination address; b) create a dedicated routing table for each VM private IP, such that the out-bound interface is selected before POSTROUTING DNAT rules are applied. I have a working prototype to show this works.