2023
Situation & Context
Critical regulatory process involves multi-stage complex computation steps across over 5 teams and a dozen systems. Cutting process time is crucial to ensure Insurance Company hits regulatory disclosure deadlines.
Diagnosis
A particular set of job processes took over 40 hours of processing time to complete. From IT hardware, software configurations, and programming logic - we needed to find a way to process the business logic in a drastically faster way.
Solution & Task
Optimize the runs by any means necessary, at all layers of the stack: business logic, application tuning, OS tuning, hardware setup, IO block alignment using GPFS or local flash drives, paging, CPU c-states, etc…
Action
Created a testing benchmark of a particular period of data with performance metrics captured for RAM, disk, CPU, paging and wall time processing for various stages of each job execution. We identified key areas that could be improved. Once we identified the bottleneck (we were CPU bound), we worked to parallelize through multithreading, performed OS tuning to optimize CPU C-states and turned off compression.
Result
Collectively, the efforts reduced process time from 40h to <5h allowing business to be able to complete jobs in under a single business day - rather than the 4-5 it originally took. Secondly, this decrease in processing time also significantly reduced the business risk of job failure. Originally, if there was a failure in a particular stage - the business team would “lose a day” attempting a fix. Now the business team only lost a couple of hours.
This cover has been designed using images from Flaticon.com