Energy efficiency initiatives have driven down energy consumption significantly over the past decades. Today's homes probably consume as much energy for lighting as that required for two or three old 100 W light bulbs. But who would have thought that, using the latest generation of hard disk drives, a petabyte of storage requiring less energy than five of those old light bulbs could be achieved?

With the demand for always-on, online storage capacity for databases seemingly showing no signs of abating, it is vital to develop storage systems that can keep up with this growing flood of data while simultaneously fulfilling certain criteria. Cost per capacity ($/TB) is usually the most important of these, due to the immense quantities of data involved. However, energy consumption is another aspect to consider as this impacts the long-term operational costs. This energy should also be consumed efficiently, thereby reducing the need for cooling that also incurs costs.

Physical dimensions of the end solution also need to be considered. Increasing the number of disks requires a housing with increased volume. Ideally, the server housing should easily be accommodated by a standard 19" rack system, fitting into existing infrastructure of 1000 mm long racks. Performance is obviously another factor but, if the key goals are high capacity at low power consumption, it is possible to tolerate toward lower IOPS or throughput figures.

In an investigation undertaken by the research team at Toshiba Electronics Europe GmbH, a project was undertaken to see if it was possible to build 1 PB of data storage into a system consuming less than 500 W of power.

CHOICE OF STORAGE
The requirement for mass capacity is achieved most cost-effectively with the use of HDDs, the top capacity models of which have similar $/TB ratios for 12 TB, 14 TB and 16 TB models. However, in order to ensure that the final system would fit into a standard 19" rack, it clearly made sense to select the largest 16 TB capacity drives to keep the physical volume required to an absolute minimum. This choice also aligns well with the power consumption goal, since the power dissipation per unit capacity has successively dropped with the introduction of new HDD models (see Table 1).

This is due not only to the new technology implemented, but also thanks the move to helium-filled drives (see Figure 1).

The 16 TB models of Toshiba's MG08 series are available with both SAS and SATA interfaces. The SAS interface provides two 12 GB/s channels that are ideally suited to systems where high availability and throughput are a priority. However, there is a power consumption cost associated with this choice since SAS drives consume around one to two watts more than their SATA counterparts. Since the goal was to reduce power consumption, the SATA interface model MG08ACA16TE was the chosen candidate for this project.

The individual specifications for this particular drive, in terms of power dissipation, are shown in Table 2.

SELECTING AN ENCLOSURE
With the storage defined, the next step was to select a suitable enclosure. Top-loader models are convenient and available as a JBOD in four unit high 19" rack format. A 60-bay model from AIC, the AIC-J4060-02, was selected for this project. The single expander version was chosen, saving on cost and power dissipation, matching with the specification of the one-channel SATA interface. Once filled with 16 TB HDDs, the solution has a gross storage of 960 TB, almost one petabyte. The JBOD is then connected to the host bus adapter (HBA) or RAID controller of the server via one mini SAS-HD cable. With a length of just 810 mm, this JBOD fits into any existing rack.

BASELINE TESTING
An initial power consumption measurement was made without the HDDs via the 220 V inputs to the twin redundant power supply. With no HDDs inserted, but both the JBOD and SAS link up, an initial measurement of 80 W was made. The next step was to measure power consumption with a single drive under different workload conditions.

Write workloads were chosen that simulated archiving, video recording and backup using 64 kB sequential blocks. Using the same block size, sequential reads were also undertaken, equivalent to a backup recovery and media streaming workload. To provide a further data point, 4 kB random read/writes were also performed, corresponding to the agile "hot-data" workload of databases. Obviously, these do not fully correlate with the typical workload for this type of system but allowed the collection of reference data for comparison purposes.

In addition to these borderline cases a test with an approximate real workload was carried out. A mix of different block sizes was read and written randomly (4kB: 20%, 64kB: 50%, 256kB: 20%, 2MB: 10%). In order to achieve the maximum possible performance, all synthetic loads were executed with a queue depth (QD) of 16. In addition to these tests, a standard copy process was started on a logical drive under Windows and the power dissipation measured.

The results for the individual drive use case consistently shows a lower power consumption than that given in the data sheet for the selected drive (see Table 3). Another point to note is that, in opposition to the data sheet, sequential loads result in higher power consumption than random access loads. This can be traced back to the power needs of the JBOD, since the SAS expanders require more power at high bandwidths in sequential operation.

Testing various configurations
With all the slots of the JBOD filled, the maximum power consumption when the system was idling lay at a respectable 420 W. This is slightly higher than expected (80 W + 60 x 4 W = 320 W) and can be traced back to the fact that the controller occasionally addresses the HDDs even in idle mode. On the other hand, the peak start-up power measured lay at just 720 W, significantly lower than the sum of the JBOD plus the spin-up data sheet values for the HDDs (80 W + 60 x 16.85 W = ~1100 W). This can be traced back to the staggered approach to spin-up the system employs, applying power to the HDDs one after the other.

The system was re-tested using the same workloads used for single drive operation. The highest power consumption of 500 W measured occurred during sequential reads of 64kB blocks, while the lowest of 445 W was for both sequential 64 kB and random 4 kB writes (see Figure 2).

Two further configurations were also investigated. The first combined the 60 disks into a local RAID10 with 5 sub-arrays to create 480 TB net storage. This was then formatted as two 240 TB logical drives under Windows Server 2016. Here, sequential accesses required less power, while random accesses essentially matched that measured in JBOD mode.

Implementing a software-defined, zettabyte file system (ZFS) using JovianDSS from Open-E also resulted in improvements in power consumption for read tests, but slightly higher measurements when writing. In this configuration two 800 GB enterprise SSDs were also added as a read cache and a write log buffer, with the resulting 240 TB logical drives made available over iSCSI.

CONCLUSIONS
Toshiba Electronics Europe GmbH estimates the total capacity of enterprise capacity (Nearline) HDDs shipped in 2019 at around 500 exabytes (500,000 petabytes). If all these HDDs were operated as 16TB models in 60-bay JBODs, this would result in a continuous power consumption of 225MW (equivalent to an average coal-fired power plant). However, since the majority of HDDs delivered in 2019 had even lower capacities, it can be assumed that the power consumption was even higher and it is clear that there is significant room for improvement to reduce the industry's W/TB power consumption figures.

The investigations and testing undertaken by Toshiba show that, thanks to the power efficiency of the latest generation of high-capacity, helium-filled disks, petabyte storage that typically demands less then 500 W of power is indeed achievable. This is a significant milestone for data centres working to grow capacity while keeping both capital expenditure and operating costs down. Additionally, this can be achieved in a range of configurations, from pure JBOD, through RAID, to software-defined, and in a standard dimension 19" rack format with a commonly available enclosure.

More info: www.toshiba-storage.com