IOSIG: Declarative I/O-Stream Properties Using Pragmas

A simple block device (HDD, SSD, etc.) has a Mean Time To Failure (MTTF) addressing the failure frequency of the device (e.g., HDD operational failures [12, 13], SSD chip failures [3, 32], etc.), and a Bit Error Rate (BER) [12]. Bit errors in sectors can be corrected by the error-correcting code (ECC) of the file system if they do not exceed the maximum correctable errors per sector. Otherwise, they are uncorrectable. The Uncorrectable Bit Errors Rate (UBER) is computed as follows [33]: where \(N\) is the sector size and \(E\) is the maximum number of correctable errors per sector. For a complex block device consisting of multiple block devices (e.g., disks) MTTDL is defined by constructing Markov models [17] according to the internal (graph-)structure of the complex block device (e.g., Ceph network of storage nodes, Lustre, RAID‑0, RAID‑1, etc.) and using MTTF, BER, and MTTDL of the sub-devices in the Markov model. For instance, the MTTDL value for MDS (Maximum Distance Separable) erasure codes tolerating \(m\) fails out of \(m+n\) drives (e.g., XOR [22, 35], Reed-Solomon [11], etc.), considering the MTTF, UBER, and Mean Time To Repair (MTTR) of the disks (sub-devices) is computed as follows [23]: where \(d=m+n\) (total number of disks), \(\lambda=1/\mathit{MTTF}\), \(\mu=1/\mathit{MTTR}\), and \(h=\mathit{size\cdot UBER}\). Replication with \(k\) replicas could be imagined as a special case of the \(n+m\) scheme with \(n=1\) and \(m=k-1\). Eq. 2 can be used to define MTTDL of different reliability schemes (erasure codes, replication) performed by network file systems such as Ceph, NFS, and Lustre on a network of storage nodes as well as locally operated reliability on multiple local disks such as RAID and ZFS [6]. For MTTDL values of non-MDS codes refer to [20, 23]. A file on a simple block device (single disk) with no redundancy/parity/ECC is not considered as reliable. For a file on a complex block device (or its partitions), MTTDL is computed using MTTF and BER of its sub-devices according to Eq. 2. Another way of representing reliability is the probability of losing data during a given time (e.g., job’s lifetime). This is directly defined using the MTTDL of the file and by the cumulative distribution function (CDF) [34, Sect. 1.3.6.2. Related Distributions].

Fig. 4 shows parameters of block devices, mount points, and file systems. Some parameters can be determined automatically and some of them require information from the user (configuration file). We determine the parameters using configuration file (\(\mathit{MTTF}\), \(\mathit{BER}\), \(\mathit{volatile}\), \(\mathit{global}\)/\(\mathit{local}\), \(\mathit{label}\) (for mount points), \(\mathit{max\_filesize}\) (file system), \(\mathit{archive}\), \(\mathit{tamperproof}\)), benchmarking [4, 25, 28] (\(\mathit{latency}\), \(\mathit{random}\)/\(\mathit{sequential}\) \(\mathit{read}\), and \(\mathit{write}\) bandwidths), system inspection (\(\mathit{mount\_point}\), \(\mathit{device\_path}\), \(\mathit{capacity}\), \(\mathit{block\_size}\), \(\mathit{format}\)), and computation (\(\mathit{MTTDL}\) and \(\mathit{availability}\)). However, the user can still overwrite the automatically determined parameters using the configuration file.

To enforce the use of the optimal storage location selected according to Sect. 5 for compilable programs, we inject a piece of code into the application at the place of the defined pragmas at compile-time. Fig. 5 shows the simplified code that is injected into the application at the location of the pragma definition. If the target file does not exist already, a symbolic link [1] pointing to the optimal destination (see Sect. 5) is created with the original filename of the ‘open’ call of the file descriptor. As a result, the IO stream of the file descriptor is automatically redirected to the optimal destination. Finally, the close(fd) call actually closes the file at the optimal destination.

There are several advantages of using pragmas to describe IO and define parameters. The source code remains compilable and executable without the IOSIG plugin. When the plugin is available, the same application can benefit from optimized IO without additional changes. On the other hand, adding some pragmas to an existing application at its expensive IO initializations (calls of open(…), fopen(…), etc.) is a low barrier an application developer or expert user could easily pass.

IOSIG is implemented as a GCC pass plugin. GCC compiles and optimizes a source code by calling a sequence of passes [19, Passes and Files of the Compiler] each responsible for a specific task. First, the parsing pass [19, Parsing pass] transforms a specific source language (C, C++, Fortran, Ada, Go, etc.) to a generic tree representation. Then, the generic representation is transformed into the GIMPLE [31] representation (a representation that can be optimized) [19, Gimplification pass]. Next, several optimization passes (IPA, high-level, low-level) are performed. Finally, the code is translated into the machine language. To interfere with the compilation process (i.e., parse the custom pragma notions, inject redirecting codes), we implemented some extra GCC passes and registered them at the proper positions in the sequence of GCC passes (using the GCC pass manager [19, Pass manager]).

The optimal destination is determined by the IOSIG plugin after lexing the pragma and fetching the defined parameters for the file descriptor. To lex pragmas we created a pragma handler (handling the \(\mathit{sig}\) and \(\mathit{sigdef}\) pragma notions) that is registered as the \(\mathit{PLUGIN\!\_PRAGMAS}\) callback of GCC [19, Registering custom attributes or pragmas]. Our pragma handler uses the GCC lexer to parse the defined pragmas, fetch the parameters, and to generate errors/warnings if necessary. To make the necessary code changes, we implemented a GCC pass that is called by GCC after the gimplification pass and before the optimization passes. Our pass constructs the GIMPLE representation of the codes to be injected. As Fig. 5 shows, an \(\mathit{if}\) statement checking the existence of a file, and a \(\mathit{symlink}\) call within the \(\mathit{if}\) statement is added. The first argument of the \(\mathit{symlink}\) call is the optimal destination, which is determined considering the parsed pragma parameters. The second argument is fetched directly from the \(\mathit{open(filename,...)}\) call in the source code. The generated GIMPLE code is inserted into the source code’s GIMPLE representation at the pragma’s location. If the \(\mathit{symlink}\) function is not declared by the source code, we declare it in GIMPLE so we can use it.

If the source code or the knowledge to read the code of an application (to add the pragma definitions to it) is not available, we redirect the IO streams of the executable binary by other means. This approach requires the user to know the IO behavior of the application, i.e., the IO destinations of the application and the parameters that describe their desired IO behavior.

We provide an executable of IOSIG that receives a path (non-optimized IO destination of the application) along with different parameters as input (IO behavior description) and creates a symlink at the given path pointing to the optimal destination on the optimal mount point. This preparatory IOSIG executable can be called multiple times to create symlinks for each IO destination of the application. Then, the application is executed as usual and performs its IO without knowing about the created symlinks at its destinations. The symlinks redirect the IO transparent to the application. After execution, a finalizing executable transfers the actual data from the optimal to the desired locations.

While the source code of the application is not needed for this approach, it requires knowledge about the exact IO destinations of the application and the respective IO behaviors to provide the IOSIG parameters. Contrarily, in the case of the pragmas, the probably well-informed developer packs the parameters describing the intended IO into the application’s source code.

We choose the optimal storage device for a particular IO demand based on our cost model (Sect. 5). Fig. 6 compares the model of Eq. 3 to the actual costs of writing data chunks for several chunk sizes and different storage devices of Table 2. We use the command-line utility dd [28] to repeatedly generate IO with different chunk sizes (size-per-IO). With doubling chunk sizes between \(512\,\text{B}\) to \(1\,\text{GB}\), Fig. 6a and b illustrate a good match between our model and the actual experiments in all devices.

Fig. 6a shows for both, the model and the experiments, that the shared NFS delivers higher throughput than the HDD for chunks smaller than \(\approx 100\,\text{MB}\), but for chunks larger than that, the HDD delivers better performance. While the HDD suffers from a higher latency than the shared NFS, it can deliver a higher maximum bandwidth (Table 2). Hence, for larger data chunks, the higher bandwidth outweighs the HDD’s higher latency compared to the shared NFS. Our plugin chooses the optimal device in such a scenario according to the given size-per-IO without requiring the user/developer to know about the details of the underlying IO infrastructure (see Sect. 7.3 for a practical use-case).

With linear increments of the chunk size between 2 to \(256\,\text{KB}\), Fig. 6c presents the model and experiments of the shared NFS exposing effects of block size on the throughput. We see harmonical variations of the throughput reaching peaks and troughs every \(22\,\text{KB}\), the internal blocksize of the device. Such blocksize can be determined by either calling some simple OS commands (e.g., creating an empty file and checking out the consumed storage space), or by performing some benchmarks for more complex cases (e.g., network file systems). We determine the closest block size match for complex block devices automatically by benchmarking and our plugin takes them into account when it chooses the optimal IO destination for a given size-per-IO. Effects of block and page size of \(4\,\text{KB}\) can be seen in Fig. 6d–fthat compare model and experiments with chunk sizes between \(512\,\text{B}\) to \(32\,\text{KB}\) on several devices. Altogether, Fig. 6 confirms our model to predict the IO throughput well enough for finding the optimal destination for an application’s IO in general.

To study our plugin with different access patterns, we developed a sample C/C++ program for image manipulation. We use the simple PGM [36] format that stores, after a short header for width and height of the image, byte-wise linear grayscale values (0–255) for each pixel without complex encoding or compression. We used this format merely to generate IO operations with different access patterns. The data could also be imagined as large matrices filled with values. We perform IO operations synchronized, i.e., the data is flushed to disk for each chunk. This should generate more stable, general, and objective results and reduce potential side effects of different operating systems (cache, memory, data size, etc.). Although, as we determine storage devices’ performance characteristics automatically using benchmarks (Sect. 7.1), our plugin also handles more specific scenarios involving different cache effects of the OS and various system properties. The user can complement or override the values in the configuration file.

We use three transformations on PGM images, illustrated in Fig. 8, in our test application to generate different access patterns. Posterization (Fig. 8b) writes the results chunkwise into the output file (chunkwise sequential access). Dividing a picture (Fig. 8c) requires strided IO accesses with seeking regularly to specific distances: every \(2k\)-th data chunk (\(k\in\mathbb{N}\geq 0\)) is written on the upper half of the file (seek to the offset \(k\)), and every \((2k+1)\)-th data chunk is written on the lower half of the file (seek to the offset \(\textit{totalsize}/2+k\)). Image shuffle (Fig. 8d) shuffles the chunks of the input image (chunkwise shuffling) by inserting them at (pseudo) random places into the output file (seek to uniformly distributed offsets) to produce random-access IO. After computing each chunk, the chunk is flushed to disk.

Fig. 7 shows a template of how our application uses pragma notions. We perform our image transformations and test the effects of different pragma annotations.

Posterization: Chunkwise Sequential Access. Fig. 11a inspects the mentioned tradeoff between the local HDD and the shared NFS devices for an application with and without the pragma notions. The application loads a \(6\,\text{GB}\) input image, posterizes it (Fig. 8b), and writes the output file chunkwise (sequential chunks) in \(16\,\text{MB}\) to \(2\,\text{GB}\) chunks. The corresponding pragma notion inserted for \(16\,\text{MB}\) chunks is:

The totalsize pragma excludes the local SSD and NVRAM as storage devices due to capacity reasons, and Ramdisk is not considered due to the persistent pragma. The only devices capable of storing the output are the shared NFS and the local HDD. For chunks smaller than \(\approx 100\,\text{MB}\), the optimal device would be the shared NFS, and the local HDD otherwise (see also Fig. 6a). We compare the overall runtime for different chunk sizes with and without pragma annotations when the application intends to use the local HDD vs. the shared NFS (Fig. 11a). With our pragma definitions, the fastest storage device is chosen for each particular chunk size automatically, independent of the initially targeted storage device of the application.

Divided Pixels: Strided Access. We evaluate how pragma parameters redirect the IO when performing regular seeks and writes into the upper and lower partitions of the output image. The size of the input image is \(1\,\text{GB}\), and the IO accesses are performed chunkwise (\(\textit{size-per-IO}=\textit{chunk}\), write and sync each chunk) in the range of \(32\,\text{KB}\) to \(32\,\text{MB}\). The destination of the output path is on the local HDD, which undoubtedly delivers poor performance. We aim to show how destinations are chosen for different given pragma notions, and how those improve the application’s lifetime. The inserted pragma notions ps1 to ps4 are listed in Fig. 10.

For \(1\,\text{GB}\) storage size, all our devices could be used. As the accesses involve seeking, we use the random property. The parameter set ps1 can only be satisfied by the shared NFS, due to the global parameter. For ps2, global is not required, but the shared NFS could still be chosen as the optimal device. Instead, ps2 requires a minimum MTTDL of 20 years for \(1\,\text{GB}\). Our RAID 1 system with 2\(\times\)SSDs achieves more than 100 years according to our reliability model of Sect. 4.1 (see Sect. 7.4 for details). Without a reliability demand, ps3 needs the data to be stored on a persistent device (persist), which excludes the Ramdisk, but leaves us with NVRAM, which is the fastest persistent device. Finally, if reliability and persistency are not required (ps4), then the Ramdisk will be chosen here.

Fig. 11b confirms the outlined device choices. The application’s lifetime is improved by applying different parameter sets in the pragma notion. Run on HDD without pragmas and in chunks of \(512\,\text{KB}\) takes more than 5 minutes. The ps1 pragma (using NFS) improves the lifetime massively (\(14379\,\text{ms}\)), ps2 (local RAID) improves it further (\(2519\,\text{ms}\)), ps3 delivers the results within \(1448\,\text{ms}\), and ps4 achieves the best performance (\(818\,\text{ms}\)) by using the Ramdisk.

Shuffled Image: Random Access. For random access involving uniformly distributed seeks of random lengths, Fig. 11c shows similar results compared to the strided IO access for the sets of parameters in Fig. 10. For small chunks, the application faces dramatic performance degradations, especially with the local HDD (as expected). Altogether, we see that the pragma notions successfully redirect IO to the appropriate device considering the IO demands and device characteristics.

The shown experiments used totalsize of \(1\,\text{GB}\) and \(6\,\text{GB}\) respectively. We also ran experiments with larger sizes and longer application runtime gaining analog results compared to the presented plots. As one can derive from Fig. 11a–c, the runtimes for each measurement point ranges from at least 6 seconds up to more than 16 minutes already for the smaller sizes.

In a Monte Carlo [27] simulation we emulate faulty storage devices to confirm that the MTTDL model of Sect. 4.1 appropriately estimates storage reliability. To simulate a faulty environment, each device faces failures according to its given MTTF and produces bit errors with the given BER. We assume an exponential failure distribution as suggested by [15, 30]. Fig. 12 compares the modeled MTTDL of different RAID systems with the simulated values. An MTTF of 10 years per device is assumed [26]. The bit errors with the rate (BER) of \(10^{-6}\) (one bit-error per \(128\,\text{KB}\)) are corrected by ECC, which fixes up to 2 bit-errors per 512-bit sectors [33]. This results in a UBER value of \(\approx 10^{-14}\) (one uncorrectable bit-error per \(10\,\text{TB}\)) that is often quoted by the HDD and NAND-based storage manufacturers [33]. The recovery is assumed to take one day [20].

The simulations and model of Sect. 4.1 match properly (Fig. 12). As expected, larger data have lower MTTDL (more bit errors). For instance, \(1\,\text{TB}\) has an MTTDL of 15 months on RAID5 and 12 years on RAID1. The MTTDL and availability (computed using MTTDL and the job’s lifetime) parameters of our plugin specify minimum values of them for a file.

We used existing models to describe the MTTDL of storage devices to demonstrate that IOSIG is beneficial in this context as well, and guiding parameters are not limited to access patterns.