Optimisation of pipeline
Run the full pipeline with a small subset of sample files and use /usr/bin/time -v to get timings and memory usage.
Go through each task individually and see if there are any means to improve run time and memory usage.
Run the full pipeline with a small subset of sample files and use /usr/bin/time -v to get timings and memory usage.
Go through each task individually and see if there are any means to improve run time and memory usage.