* (1) had a ceph MDS hiccup with client failing to release caps error, killed the slurm job as it was taking too long
* (2) 32n1t ssd put too high loads on the SSD pool, perhaps due to having too few storage nodes there vs the clients, and also the big mismatch in network speed (100G on client vs 25G on storage), and crashed 2 storage nodes. Did not have time to run the 32n1t nlsas
Note: the metadata performance is mostly about the NVMe metadata pool, and does not really reflect the differences between NLSAS and SSD.
Note: the metadata performance is mostly about the NVMe metadata pool, and does not really reflect the differences between NLSAS and SSD. Sometimes, the metadata performance in the SSD pool benchmarks can actually be lower than that of the NLSAS benchmarks because the metadata OSDs are on the same servers as the SSD OSDs, therefore the loads from the SSD pool benchmarks would affect metadata performance more.
## System loads
...
...
@@ -165,6 +165,8 @@ During testing, we observed and also monitored the system loads. Here are some h
IO easy writes put a lot of load on our NLSAS OSDs, which created a storm of slow requests. At worst, they affected every single NLSAS OSD, and piled up like this: `90817 slow requests are blocked > 32 sec`. However, they cleared up as soon as the test neared its end, and did not cause any harmful effect.
Although the cluster and NLSAS pool coped fine, due to this, we do not recommend running big jobs directly on the NLSAS pool, which should be reserved for long term storage only. Compute jobs should be on faster scratch storage e.g SSD or NVMe.
### MDS requests
Our MDS nodes got hit really hard during the metadata tests. The 10n16t benchmarks put the biggest load we had ever seen on them, e.g:
