Configuring IPMP in Oracle Solaris 11

Configuring IPMP in Solaris 11Configuring IPMP in Oracle Solaris 11

In this post, we are going to discuss how we will be configuring IPMP in Oracle Solaris 11. Oracle SPARC Virtual Machine supports link-based IP network multipathing which is known as IPMP with the help of virtual network devices.

How to Check vswitch configuration on CDOM 

# ldm ls -o net secondary
NAME
secondary
MAC
00:14:4f:f9:cf:da
VSW
NAME MACADDRESS NET-DEV DVID|PVID|VIDs
---- ---------- ------- --------------
secondary-vsw0 00:14:4f:f9:01:8f net6 1|1|--
DEVICE :[email protected] ID :0
LINKPROP :phys-state MTU :1500
INTER-VNET-LINK :on MODE :--
VSW-RELAY-MODE :local
How to Check NIC Configuration for the LDOM

Consider the LDOM server name is ldom01 just for demonstration purposes.

# ldm ls -o net ldom01
NAME
Ldom01
MAC
00:14:4f:fb:44:8a
NETWORK
NAME SERVICE MACADDRESS PVID|PVLAN|VIDs
---- ------- ---------- ---------------
vnet0 [email protected] 00:14:4f:fb:c7:4a 1|--|--
DEVICE :[email protected] ID :0
LINKPROP :phys-state MTU :1500
MAXBW :-- MODE :--
CUSTOM :disable
PRIORITY :-- COS :--
PROTECTION :--
configuring-ipmp-in-oracle-solaris-11

Next, We will have to configure two virtual network devices vnet0 and vnet1 on Control Domain. Every domain is connected to a different virtual switch device on the service domain for using link-based IPMP.

Virtual Network devices get configured to obtain physical link state status. We already have vnet0 so we will modify existing devices via set-vnet command.

# ldm add-vnet linkprop=phys-state vnet1 secondary-vsw0 ldom01
# ldm set-vnet linkprop=phys-state vnet0 ldom01
# ldm ls -o net ldom01
NAME
ldom01
MAC
00:14:4f:fb:44:8a
NETWORK
NAME SERVICE MACADDRESS PVID|PVLAN|VIDs
---- ------- ---------- ---------------
vnet0 [email protected] 00:14:4f:fb:c7:4a 1|--|--
DEVICE :[email protected] ID :0
LINKPROP :phys-state MTU :1500
MAXBW :-- MODE :--
CUSTOM :disable
PRIORITY :-- COS :--
PROTECTION :--
NAME SERVICE MACADDRESS PVID|PVLAN|VIDs
---- ------- ---------- ---------------
vnet1 [email protected] 00:14:4f:fb:4c:d1 1|--|--
DEVICE :[email protected] ID :1
LINKPROP :phys-state MTU :1500
MAXBW :-- MODE :--
CUSTOM :disable
PRIORITY :-- COS :--
PROTECTION :--
When we complete all the mentioned configurations, then we will proceed with IPMP 
configuration in the Logical Domain.

We are trying to create an IPMP group and then we will add both virtual nic(vnics) to that IPMP group.

Because Logical Domain has an existing NIC vnet0 (net0), it needs to add the secondary NIC vnet1 (net1) in the IPMP group.

Oracle has a concept of vanity name. In our case, net0 and net1 are the Oracle Solaris 11 vanity names for vnet0 and vnet1.

Check the existing network configuration: On LDOM

[email protected]:~# ipadm
NAME CLASS/TYPE STATE UNDER ADDR
lo0 loopback ok — —
lo0/v4 static ok — 127.0.0.1/8
lo0/v6 static ok — ::1/128
net0 ip ok — —
net0/v4 static ok — 10.175.176.47/23
[email protected]:~#

Create additional nic net1 (vnet1).

[email protected]:~# ipadm create-ip net1
[email protected]:~# ipadm
NAME CLASS/TYPE STATE UNDER ADDR
lo0 loopback ok -- --
lo0/v4 static ok -- 127.0.0.1/8
lo0/v6 static ok -- ::1/128
net0 ip ok -- --
net0/v4 static ok -- 10.175.176.47/23
net1 ip down -- --
[email protected]:~#

Already existing nic net0 has got an IP address configured of LDOM, IP configuration has to be deleted to allow us to configure it under the IPMP group.

Remember connectivity to the LDOM server will be going to lost, therefore, below configurations needs to be done through the LDOM console.

[email protected]:~# ipadm delete-addr net0/v4
[email protected]:~# ipadm
NAME CLASS/TYPE STATE UNDER ADDR
lo0 loopback ok -- --
lo0/v4 static ok -- 127.0.0.1/8
lo0/v6 static ok -- ::1/128
net0 ip down -- --
net1 ip down -- --
[email protected]:~#

Create IPMP group “ipmp0” with nics net0 and net1 
grouped into it. Then assign the IP address to the
IPMP group.

[email protected]:~# ipadm create-ipmp ipmp0
[email protected]:~# ipadm add-ipmp -i net0 -i net1 ipmp0
[email protected]:~# ipadm
NAME CLASS/TYPE STATE UNDER ADDR
ipmp0 ipmp down — —
lo0 loopback ok — —
lo0/v4 static ok — 127.0.0.1/8
lo0/v6 static ok — ::1/128
net0 ip ok ipmp0 —
net1 ip ok ipmp0 —
[email protected]:~# ipadm create-addr -T static -a 10.175.176.47/23 ipmp0/v4
[email protected]:~# ipadm
NAME CLASS/TYPE STATE UNDER ADDR
ipmp0 ipmp ok — —
ipmp0/v4 static ok — 10.175.176.47/23
lo0 loopback ok — —
lo0/v4 static ok — 127.0.0.1/8
lo0/v6 static ok — ::1/128
net0 ip ok ipmp0 —
net1 ip ok ipmp0 —
[email protected]:~#

The IPMP configuration is complete. The LDOM server should be accessible on the network now.
To verify the configuration, it is possible to list the NICs and network configuration and compare the MAC address with that of vnetX interfaces provisioned in CDOM for this LDOM server profile.

To list the network configuration:

[email protected]:~# ipadm
NAME CLASS/TYPE STATE UNDER ADDR
ipmp0 ipmp ok — —
ipmp0/v4 static ok — 10.175.176.47/23
lo0 loopback ok — —
lo0/v4 static ok — 127.0.0.1/8
lo0/v6 static ok — ::1/128
net0 ip ok ipmp0 —
net1 ip ok ipmp0 —
[email protected]:~#

To list the NICs on the LDOM server:
[email protected]:~# dladm show-phys
LINK MEDIA STATE SPEED DUPLEX DEVICE
net0 Ethernet up 10000 full vnet0
net1 Ethernet up 10000 full vnet1

configuring-ipmp-in-oracle-solaris-11-1To get the MAC details to check to match with the LDOM 
profile MAC address defined in CDOM:
[email protected]:~# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
net0: flags=100001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,PHYSRUNNING> mtu 1500
index 2
inet 0.0.0.0 netmask fffffe00 broadcast 0.0.1.255
groupname ipmp0
ether 0:14:4f:fb:c7:4a
net1: flags=100001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,PHYSRUNNING> mtu 1500
index 3
inet 0.0.0.0 netmask ff000000
groupname ipmp0
ether 0:14:4f:fb:4c:d1
ipmp0: flags=108001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,IPMP,PHYSRUNNING> mtu
1500 index 4
inet 10.175.176.47 netmask fffffe00 broadcast 10.175.177.255
groupname ipmp0
lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1
inet6 ::1/128
net0: flags=120002000841<UP,RUNNING,MULTICAST,IPv6,PHYSRUNNING> mtu 1500 index 2
inet6 ::/0
groupname ipmp0
ether 0:14:4f:fb:c7:4a
net1: flags=120002000841<UP,RUNNING,MULTICAST,IPv6,PHYSRUNNING> mtu 1500 index 3
inet6 ::/0
groupname ipmp0
ether 0:14:4f:fb:4c:d1
ipmp0: flags=128002000840<RUNNING,MULTICAST,IPv6,IPMP,PHYSRUNNING> mtu 1500 index
4 inet6 ::/0
groupname ipmp0
[email protected]:~#
Compare the MAC address of net0 and net1 with that of 
vnet0 and vnet1 respectively from CDOM.
[email protected]:~# ldm ls -o net ldom01
NAME
Ldom01
MAC
00:14:4f:fb:44:8a
NETWORK
NAME SERVICE MACADDRESS PVID|PVLAN|VIDs
---- ------- ---------- ---------------
vnet0 [email protected] 00:14:4f:fb:c7:4a 1|--|--
DEVICE :[email protected] ID :0
LINKPROP :phys-state MTU :1500
MAXBW :-- MODE :--
CUSTOM :disable
PRIORITY :-- COS :--
PROTECTION :--
NAME SERVICE MACADDRESS PVID|PVLAN|VIDs
---- ------- ---------- ---------------
vnet1 [email protected] 00:14:4f:fb:4c:d1 1|--|--
DEVICE :[email protected] ID :1
LINKPROP :phys-state MTU :1500
MAXBW :-- MODE :--
CUSTOM :disable
PRIORITY :-- COS :--
PROTECTION :--
[email protected]:~#

That concludes configuring IPMP in Oracle Solaris 11. This will be persistent across reboots. To make sure you can reboot the LDOM server and check configuration after reboot.