Itanium server
So we got a really nice Itanium server donated from HP. I have access to 8 CPU cores and 32GB of memory so it's great for running the process I'm currently setting up. My boss wanted me to write a little info about what I'm using it for and I thought I might as well post it here.
Our nightly process is being developed on the Itanium server. To work with our environment I installed the IA64 .NET runtime with the IA64 remote debugger. Compiled applications can run without modification on the server with no work, but I have written code designed to utilize the extended resources of the server.
With access to 8 of the 16 cores available I tried writing the program to break into threads. The process’ goal can be divided into two sections. The first is to download student records from the school’s Oracle database(Aries), compare them to a copy of yesterday’s data to find which records are new, updated or deleted, then submit changes to our CRM system. Part two is pulling the data out of CRM which is now the merged between updates from Aries and updates made inside of CRM. Once the data is pulled out it is sent to various other systems: Account Creation, WebCT, Print Balance Manager, ect.
With roughly 800,000 student records I was originally going to process the records one at a time, but with the multiple cores I break the 800,000 records into chunks and process each chunk in an individual thread. I have different parameters for the number of records per chunk and the number of chunks processed at a time. The biggest help here is access to all the memory. A copy of the entire Aries database was just over 5 GB on the file system, but we don’t need to load it all. I load about 1 GB of data into memory. When this is broken into chunks at the end of the process the chunk data is serialized and saved to the file system so it can be compared to the next day’s data. The chunks are indexed by their associated CSU ID so the next day when a record is being processed it caches the previous day’s chunk in memory. For a time, essentially two times the memory is loaded if all chunks were being processed at that time, which they are because the server can easily accommodate this.
Compared to the server I was developing the process on previously with 2 CPU cores and 3 GB of memory the speed of the program has increased tremendously.
I'm not sure if my methods are the greatest but I sure am learning a lot as I go through this! In fact, this is why I named the blog empirical programming... because in all honesty I can get things done, but I have no idea if it is the right or wrong way.
