Why did we choose InfiniBand




Throughput capacity inside the cloud is very important from the technological point of view. It’s a delicate moment, most suppliers of cloud services talk unwillingly speak to this point; do not want to disclose the structure of their network. At the same time for the user of this service this matter is principal: the quality of the product purchased from the provider depends on the throughput capacity. Large role in the capacity of infrastructure operation plays how the modes are connected between themselves, on which speeds and by which protocols. Services of the most enterprises can easily work with 1 Gbps (Gbit per second), but it should be taken into account that most providers use 10G,  and only the most advanced built their clouds of 40G or InfiniBand 56G.


In our cloud infrastructure SIM-Cloud and in our data centers we use equipment Mellanox InfiniBand 56 Gbps. At the current level it is more than enough to implement our task in visible future. However, in future, in the wake of rising cloud IaaS, naturally, we plan transferring to 100 Gbps. We chose InfiniBand protocol for several reasons:

  • universality;
  • hyper-convergence;
  • cost;
  • growth opportunity.


We put forward several arguments turning the scales in favor of choosing exactly InfiniBand by designing of cloud IaaS SIM-Cloud.


InfiniBand: History of origin

InfiniBand technology exists far long ago, just as much as Ethernet does. The difference was in its purpose only: InfiniBand has been developing for use in supercomputers. Within environment of highly productive systems the larger clusters should be combined with the bonding bus for fast data exchange. As the protocol was designed for introduction into the super-systems, accordingly, the tasks for which it was designed and the requirements for productivity, delays and architecture were defined by the cluster systems. However, as the Internet develops, InfiniBand developmental dynamics turned out to be not so rough, more likely because this standard is sufficiently niche.

Currently, against the background of cloud technologies growth, when clouds use has not been restricted by the highly-computation systems, but take the ground at the consumption level –InfiniBand got the second wind.

The protocol grew not only in its productivity, but also obtained mass expansion: previously yet in highly-productive environments the speed of data exchange was substantially lower - 4,8 Gbps, and currently 56 Gbps being a common standard.  The range of InfiniBand-equipment’s manufacturers, in particular, Mellanox, has already been selling with throughput capacity of 100 Gbps. There are all reasons to think that with such technology development rates InfiniBand-vendors can in the next 12 months produce even faster equipment.


Why did we choose InfiniBand exactly of Mellanox? The answer is quite simple: as the manufacturer of InfiniBand equipment Mellanox company contributes the most in the development of this protocol: introduce many new functions and improves the protocol itself. It is worth noting that Mellanox are so good that most manufacturers are either just branding Мellanox platform or seal Mellanox chips in their equipment.


InfiniBand Vs. Ethernet: similarities and differences

The principal advantages of InfiniBand protocol is a high throughput capacity and above all – low delay indexes. The standard and equipment itself allow transfer the packet 10 times faster than Ethernet. For high productive computers and up-to-date data transfer systems this plays a crucial role.


Why did we choose InfiniBand

No less important that this technology has been already improved within those environments where it was designed. This standard has “correct” plug-in-play architecture – everything is well automated: tree building, infrastructure, etc. That’s why the engineer has no need to make significant efforts in adjusting. Moreover, in the protocol itself fail-safety stock has already been put.

Of course, all these opportunities are available in Ethernet as well, but unlike InfiniBand they are implemented by the entire pool of technologies, each of which requires a separate adjustment and then – obligatory conformance testing. I.e., when working with Ethernet engineer shall take into account more quantity of infrastructure parts.

 Why did we choose InfiniBand

Mellanox ranges InfiniBand equipment as hyper-convergent: the entire equipment may be used both for InfiniBand-network and for Ethernet-network (in the latter case Ethernet is implemented over InfiniBand-network). This means the unavailability of the necessity to place different distribution boards for different networks: it’s enough to install one device detaching thereon two port groups – SAN and Ethernet 40 Gbps. Consequently, maximum number of roles is coincided in one device – the equipment is unified. Together, it should be noticed that as a technology InfiniBand is enough simple, there are no much functions available in Ethernet complicating control.


InfiniBand in applied tasks

Currently InfiniBand protocol is actively used at the level of SAN-network in cloud environments. At the moment we are planning introduction of the second architecture of our cloud SIM-Cloud, wherein we will fully pass to hyper-convergent infrastructure – InfiniBand will be at the level of SAN-network, and at the level of WAN-network. In comparison with the distribution boards the speed of such equipment will be higher with lower delays (being very topical for highly-loaded applications). Hence, when using Ethernet-protocol delays makes above 20-40 ms, in InfiniBand delay makes up 1-2 ms. The advantage is obvious.

 Why did we choose InfiniBand

Another very important Infiniband peculiarity, “grown” from its “supercomputer past”: the opportunity of building non-blocking network (with the typology of a fat tree– FatTree). I.e., such network where by simultaneous data transfer with 50% of all network nodes by another 50% delay and the speed of data transfer will remain at the same level when by communication of one node with the another one.

 Why did we choose InfiniBand

Currently all-flash systems are available at the market building on fast flash-disks. Naturally that for such system the fast environment for data transfer is required. Otherwise, by lack of data transfer speed between storages, the consumers of the service will not get the expected system operation quality, and then the whole sense of fast flash-memory (evidently that race car Ferrari is not able to show itself at its finest on the country road in autumn weather!) is lost. However, the opinion exists that 50% of IT-tasks still do not require so high speeds. Flash drives are appropriate where large data bases exist and high productivity for fast data selecting is required, - where the administrators most often complain, “More powerful processors….more powerful disks are required”. Besides, solutions of DaaS, VDI types require more speed of the highly-productive storage – for example, when all virtual machines call the storage leading to so called boot storm and storage “freezing”. But for other tasks, less complicated and more widespread (such as: criminal servers, mail servers, etc.), the speed of SSD and SATA disks is enough. That’s we think that there is no need to pay for unnecessary speed.


Our InfiniBand Use Case

We faced the task to build enough zoomed storage. There were two alternatives of its implementation: either we build classical Fiber Channel storage + regular storages, or we pay our attention to modern SDS (Software Define Storage). The second alternative turned to us, as to courageous innovators preferring proactively increase the effectiveness of our solutions, more attractive. By more detailed studying the question on building the infrastructure with SDS it is turned out that in this case the throughput capacity is not so much important (in average 15 Gbps par channel is enough) as very low delays in packets transfer. Ethernet-distribution boards with the same low delays as with InfiniBand are to massive and expensive devices not exactly suitable to our task (we think that this equipment is more convenient for using it in VAN-network, but not for storage-network). Considering offers of different vendors we finally chose equipment InfiniBand of Mellanox for testing.

Results of the trial-period completely satisfied our expectations. A pleasant breakthrough during the testing became the fact that there were no “clean” marketing in Mellanox presentations – all equipment characteristics fully complied with the declared functions. Accordingly, after successful testing completion the equipment was installed in our IT-infrastructure.


Share this: