HPC-IO500-2019-07-08.md

@@ -153,4 +153,29 @@ Why these test?
 | mdtest_hard_del kiops   | n/a (1)    | 2.673     | 4.160      | 4.310    | 4.577        | 4.491      | n/a    |
 
 * (1) had a ceph MDS hiccup with client failing to release caps error, killed the slurm job as it was taking too long
-* (2) 32n1t ssd put too high loads on the SSD pool, perhaps due to having too few storage nodes there vs the clients, and also the big mismatch in network speed (100G on client vs 25G on storage), and crashed 2 storage nodes. Did not have time to run the 32n1t nlsas
\ No newline at end of file
+* (2) 32n1t ssd put too high loads on the SSD pool, perhaps due to having too few storage nodes there vs the clients, and also the big mismatch in network speed (100G on client vs 25G on storage), and crashed 2 storage nodes. Did not have time to run the 32n1t nlsas
+
+Note: the metadata performance is mostly about the NVMe metadata pool, and does not really reflect the differences between NLSAS and SSD.
+
+## System loads 
+
+During testing, we observed and also monitored the system loads. Here are some highlights:
+
+### Slow request storm
+
+IO easy writes put a lot of load on our NLSAS OSDs, which created a storm of slow requests. At worst, they affected every single NLSAS OSD, and piled up like this: `90817 slow requests are blocked > 32 sec`. However, they cleared up as soon as the test neared its end, and did not cause any harmful effect. 
+
+### MDS requests
+
+Our MDS nodes got hit really hard during the metadata tests. The 10n16t benchmarks put the biggest load we had ever seen on them, e.g:
+
+```
++------+--------+----------------+---------------+-------+-------+
+| Rank | State  |      MDS       |    Activity   |  dns  |  inos |
++------+--------+----------------+---------------+-------+-------+
+|  0   | active | mds3-ceph2-qh2 | Reqs: 19.8k/s | 2542k | 2519k |
+|  1   | active | mds2-ceph2-qh2 | Reqs: 14.4k/s | 6751k | 6751k |
++------+--------+----------------+---------------+-------+-------+
+```
+
+