Configuring IPMP in Oracle Solaris 11

In this post, we are going to discuss how we will be configuring IPMP in Oracle Solaris 11. Oracle SPARC Virtual Machine supports link-based IP network multipathing which is known as IPMP with the help of virtual network devices.

How to Check vswitch configuration on CDOM 

# ldm ls -o net secondary
NAME
secondary
MAC
00:14:4f:f9:cf:da
VSW
NAME MACADDRESS NET-DEV DVID|PVID|VIDs
---- ---------- ------- --------------
secondary-vsw0 00:14:4f:f9:01:8f net6 1|1|--
DEVICE :switch@0 ID :0
LINKPROP :phys-state MTU :1500
INTER-VNET-LINK :on MODE :--
VSW-RELAY-MODE :local
How to Check NIC Configuration for the LDOM

Consider the LDOM server name is ldom01 just for demonstration purposes.

# ldm ls -o net ldom01
NAME
Ldom01
MAC
00:14:4f:fb:44:8a
NETWORK
NAME SERVICE MACADDRESS PVID|PVLAN|VIDs
---- ------- ---------- ---------------
vnet0 primary-vsw0@primary 00:14:4f:fb:c7:4a 1|--|--
DEVICE :network@0 ID :0
LINKPROP :phys-state MTU :1500
MAXBW :-- MODE :--
CUSTOM :disable
PRIORITY :-- COS :--
PROTECTION :--

Next, We will have to configure two virtual network devices vnet0 and vnet1 on Control Domain. Every domain is connected to a different virtual switch device on the service domain for using link-based IPMP.

Virtual Network devices get configured to obtain physical link state status. We already have vnet0 so we will modify existing devices via set-vnet command.

# ldm add-vnet linkprop=phys-state vnet1 secondary-vsw0 ldom01
# ldm set-vnet linkprop=phys-state vnet0 ldom01
# ldm ls -o net ldom01
NAME
ldom01
MAC
00:14:4f:fb:44:8a
NETWORK
NAME SERVICE MACADDRESS PVID|PVLAN|VIDs
---- ------- ---------- ---------------
vnet0 primary-vsw0@primary 00:14:4f:fb:c7:4a 1|--|--
DEVICE :network@0 ID :0
LINKPROP :phys-state MTU :1500
MAXBW :-- MODE :--
CUSTOM :disable
PRIORITY :-- COS :--
PROTECTION :--
NAME SERVICE MACADDRESS PVID|PVLAN|VIDs
---- ------- ---------- ---------------
vnet1 secondary-vsw0@secondary 00:14:4f:fb:4c:d1 1|--|--
DEVICE :network@1 ID :1
LINKPROP :phys-state MTU :1500
MAXBW :-- MODE :--
CUSTOM :disable
PRIORITY :-- COS :--
PROTECTION :--
When we complete all the mentioned configurations, then we will proceed with IPMP 
configuration in the Logical Domain.

We are trying to create an IPMP group and then we will add both virtual nic(vnics) to that IPMP group.

Because Logical Domain has an existing NIC vnet0 (net0), it needs to add the secondary NIC vnet1 (net1) in the IPMP group.

Oracle has a concept of vanity name. In our case, net0 and net1 are the Oracle Solaris 11 vanity names for vnet0 and vnet1.

Check the existing network configuration: On LDOM

root@ldom01:~# ipadm
NAME CLASS/TYPE STATE UNDER ADDR
lo0 loopback ok — —
lo0/v4 static ok — 127.0.0.1/8
lo0/v6 static ok — ::1/128
net0 ip ok — —
net0/v4 static ok — 10.175.176.47/23
root@ldom01:~#

Create additional nic net1 (vnet1).

root@ldom01:~# ipadm create-ip net1
root@ldom01:~# ipadm
NAME CLASS/TYPE STATE UNDER ADDR
lo0 loopback ok -- --
lo0/v4 static ok -- 127.0.0.1/8
lo0/v6 static ok -- ::1/128
net0 ip ok -- --
net0/v4 static ok -- 10.175.176.47/23
net1 ip down -- --
root@ldom01:~#

Already existing nic net0 has got an IP address configured of LDOM, IP configuration has to be deleted to allow us to configure it under the IPMP group.

Remember connectivity to the LDOM server will be going to lost, therefore, below configurations needs to be done through the LDOM console.

root@ldom01:~# ipadm delete-addr net0/v4
root@ldom01:~# ipadm
NAME CLASS/TYPE STATE UNDER ADDR
lo0 loopback ok -- --
lo0/v4 static ok -- 127.0.0.1/8
lo0/v6 static ok -- ::1/128
net0 ip down -- --
net1 ip down -- --
root@ldom01:~#

Create IPMP group “ipmp0” with nics net0 and net1 
grouped into it. Then assign the IP address to the
IPMP group.

root@ldom01:~# ipadm create-ipmp ipmp0
root@ldom01:~# ipadm add-ipmp -i net0 -i net1 ipmp0
root@ldom01:~# ipadm
NAME CLASS/TYPE STATE UNDER ADDR
ipmp0 ipmp down — —
lo0 loopback ok — —
lo0/v4 static ok — 127.0.0.1/8
lo0/v6 static ok — ::1/128
net0 ip ok ipmp0 —
net1 ip ok ipmp0 —
root@ldom01:~# ipadm create-addr -T static -a 10.175.176.47/23 ipmp0/v4
root@ldom01:~# ipadm
NAME CLASS/TYPE STATE UNDER ADDR
ipmp0 ipmp ok — —
ipmp0/v4 static ok — 10.175.176.47/23
lo0 loopback ok — —
lo0/v4 static ok — 127.0.0.1/8
lo0/v6 static ok — ::1/128
net0 ip ok ipmp0 —
net1 ip ok ipmp0 —
root@ldom01:~#

The IPMP configuration is complete. The LDOM server should be accessible on the network now.
To verify the configuration, it is possible to list the NICs and network configuration and compare the MAC address with that of vnetX interfaces provisioned in CDOM for this LDOM server profile.

To list the network configuration:

root@ldom01:~# ipadm
NAME CLASS/TYPE STATE UNDER ADDR
ipmp0 ipmp ok — —
ipmp0/v4 static ok — 10.175.176.47/23
lo0 loopback ok — —
lo0/v4 static ok — 127.0.0.1/8
lo0/v6 static ok — ::1/128
net0 ip ok ipmp0 —
net1 ip ok ipmp0 —
root@ldom01:~#

To list the NICs on the LDOM server:
root@ldom01:~# dladm show-phys
LINK MEDIA STATE SPEED DUPLEX DEVICE
net0 Ethernet up 10000 full vnet0
net1 Ethernet up 10000 full vnet1

To get the MAC details to check to match with the LDOM 
profile MAC address defined in CDOM:
root@ldom01:~# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
net0: flags=100001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,PHYSRUNNING> mtu 1500
index 2
inet 0.0.0.0 netmask fffffe00 broadcast 0.0.1.255
groupname ipmp0
ether 0:14:4f:fb:c7:4a
net1: flags=100001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,PHYSRUNNING> mtu 1500
index 3
inet 0.0.0.0 netmask ff000000
groupname ipmp0
ether 0:14:4f:fb:4c:d1
ipmp0: flags=108001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,IPMP,PHYSRUNNING> mtu
1500 index 4
inet 10.175.176.47 netmask fffffe00 broadcast 10.175.177.255
groupname ipmp0
lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1
inet6 ::1/128
net0: flags=120002000841<UP,RUNNING,MULTICAST,IPv6,PHYSRUNNING> mtu 1500 index 2
inet6 ::/0
groupname ipmp0
ether 0:14:4f:fb:c7:4a
net1: flags=120002000841<UP,RUNNING,MULTICAST,IPv6,PHYSRUNNING> mtu 1500 index 3
inet6 ::/0
groupname ipmp0
ether 0:14:4f:fb:4c:d1
ipmp0: flags=128002000840<RUNNING,MULTICAST,IPv6,IPMP,PHYSRUNNING> mtu 1500 index
4 inet6 ::/0
groupname ipmp0
root@ldom01:~#
Compare the MAC address of net0 and net1 with that of 
vnet0 and vnet1 respectively from CDOM.
root@cdom01:~# ldm ls -o net ldom01
NAME
Ldom01
MAC
00:14:4f:fb:44:8a
NETWORK
NAME SERVICE MACADDRESS PVID|PVLAN|VIDs
---- ------- ---------- ---------------
vnet0 primary-vsw0@primary 00:14:4f:fb:c7:4a 1|--|--
DEVICE :network@0 ID :0
LINKPROP :phys-state MTU :1500
MAXBW :-- MODE :--
CUSTOM :disable
PRIORITY :-- COS :--
PROTECTION :--
NAME SERVICE MACADDRESS PVID|PVLAN|VIDs
---- ------- ---------- ---------------
vnet1 secondary-vsw0@secondary 00:14:4f:fb:4c:d1 1|--|--
DEVICE :network@1 ID :1
LINKPROP :phys-state MTU :1500
MAXBW :-- MODE :--
CUSTOM :disable
PRIORITY :-- COS :--
PROTECTION :--
root@cdom01:~#

That concludes configuring IPMP in Oracle Solaris 11. This will be persistent across reboots. To make sure you can reboot the LDOM server and check configuration after reboot.