diehn.net/blog

Intuitive to some, not to others.

If you want to ping across an Infiniband network, like we use ping to test other networks, you could use ibping. Pick a system to be the destination of the ping and then go find out it’s Port GUID and start a ibping server on it:

Set one system as the “server” thus:


autosup25:~ # ibstat
CA 'mthca0'
CA type: MT25204
Number of ports: 1
Firmware version: 1.2.0
Hardware version: a0
Node GUID: 0x00066a00980095d7
System image GUID: 0x00066a00980095d7
Port 1:
State: Active
Physical state: LinkUp
Rate: 10
Base lid: 3
LMC: 0
SM lid: 1
Capability mask: 0x02510a68
Port GUID:
autosup25:~ # ibping -v -S
ibwarn: [11498] ibping_serv: starting to serve...


So, the port guid on this machine is 0x00066a00980095d7. We’ll need that on the other end. So, now pick the server from which you want to start the ping – this feels backwards doesn’t it? – and do this:


autosup30:~ # ibping -G 0x00066a00a00095d7
Pong from autosup25.mi.fluent.com (Lid 3): time 0.157 ms
Pong from autosup25.mi.fluent.com (Lid 3): time 0.100 ms
Pong from autosup25.mi.fluent.com (Lid 3): time 0.091 ms
Pong from autosup25.mi.fluent.com (Lid 3): time 0.121 ms
Pong from autosup25.mi.fluent.com (Lid 3): time 0.090 ms
Pong from autosup25.mi.fluent.com (Lid 3): time 0.092 ms

--- autosup25.mi.fluent.com (Lid 3) ibping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5799 ms
rtt min/avg/max = 0.090/0.108/0.157 ms


Now, I just move from system to system pinging that server to make sure all my servers are able to talk over the Infiniband link. Vary this a bit and you could use it in Nagios or something to monitor the IB network in a compute cluster.

That’s it. All done. Except, don’t forget to go kill the ping server on your target.