1The Symmetric Communication Interface (SCIF (pronounced as skiff)) is a low 2level communications API across PCIe currently implemented for MIC. Currently 3SCIF provides inter-node communication within a single host platform, where a 4node is a MIC Coprocessor or Xeon based host. SCIF abstracts the details of 5communicating over the PCIe bus while providing an API that is symmetric 6across all the nodes in the PCIe network. An important design objective for SCIF 7is to deliver the maximum possible performance given the communication 8abilities of the hardware. SCIF has been used to implement an offload compiler 9runtime and OFED support for MPI implementations for MIC coprocessors. 10 11==== SCIF API Components ==== 12The SCIF API has the following parts: 131. Connection establishment using a client server model 142. Byte stream messaging intended for short messages 153. Node enumeration to determine online nodes 164. Poll semantics for detection of incoming connections and messages 175. Memory registration to pin down pages 186. Remote memory mapping for low latency CPU accesses via mmap 197. Remote DMA (RDMA) for high bandwidth DMA transfers 208. Fence APIs for RDMA synchronization 21 22SCIF exposes the notion of a connection which can be used by peer processes on 23nodes in a SCIF PCIe "network" to share memory "windows" and to communicate. A 24process in a SCIF node initiates a SCIF connection to a peer process on a 25different node via a SCIF "endpoint". SCIF endpoints support messaging APIs 26which are similar to connection oriented socket APIs. Connected SCIF endpoints 27can also register local memory which is followed by data transfer using either 28DMA, CPU copies or remote memory mapping via mmap. SCIF supports both user and 29kernel mode clients which are functionally equivalent. 30 31==== SCIF Performance for MIC ==== 32DMA bandwidth comparison between the TCP (over ethernet over PCIe) stack versus 33SCIF shows the performance advantages of SCIF for HPC applications and runtimes. 34 35 Comparison of TCP and SCIF based BW 36 37 Throughput (GB/sec) 38 8 + PCIe Bandwidth ****** 39 + TCP ###### 40 7 + ************************************** SCIF %%%%%% 41 | %%%%%%%%%%%%%%%%%%% 42 6 + %%%% 43 | %% 44 | %%% 45 5 + %% 46 | %% 47 4 + %% 48 | %% 49 3 + %% 50 | % 51 2 + %% 52 | %% 53 | % 54 1 + 55 + ###################################### 56 0 +++---+++--+--+-+--+--+-++-+--+-++-+--+-++-+- 57 1 10 100 1000 10000 100000 58 Transfer Size (KBytes) 59 60SCIF allows memory sharing via mmap(..) between processes on different PCIe 61nodes and thus provides bare-metal PCIe latency. The round trip SCIF mmap 62latency from the host to an x100 MIC for an 8 byte message is 0.44 usecs. 63 64SCIF has a user space library which is a thin IOCTL wrapper providing a user 65space API similar to the kernel API in scif.h. The SCIF user space library 66is distributed @ https://software.intel.com/en-us/mic-developer 67 68Here is some pseudo code for an example of how two applications on two PCIe 69nodes would typically use the SCIF API: 70 71Process A (on node A) Process B (on node B) 72 73/* get online node information */ 74scif_get_node_ids(..) scif_get_node_ids(..) 75scif_open(..) scif_open(..) 76scif_bind(..) scif_bind(..) 77scif_listen(..) 78scif_accept(..) scif_connect(..) 79/* SCIF connection established */ 80 81/* Send and receive short messages */ 82scif_send(..)/scif_recv(..) scif_send(..)/scif_recv(..) 83 84/* Register memory */ 85scif_register(..) scif_register(..) 86 87/* RDMA */ 88scif_readfrom(..)/scif_writeto(..) scif_readfrom(..)/scif_writeto(..) 89 90/* Fence DMAs */ 91scif_fence_signal(..) scif_fence_signal(..) 92 93mmap(..) mmap(..) 94 95/* Access remote registered memory */ 96 97/* Close the endpoints */ 98scif_close(..) scif_close(..) 99