Check consistency and other opportunities for parallerization

We run the program with 8 threads and the original program without thread on the whole rule set and a whole trace file. For each packet, we checked the rule number that matches the packet. We compared the results from both versions of programs, we found they are exactly the same. We think our program with multiple threads is consistent with the original program.

Yesterday, we discussed with dheeraj about other opportunities for paralleling the code. We believe there are other functions and for loops that can be parallelized. We will look further into this.

check consistency

After the meeting today, we checked consistency between code with multiple threads and without thread. We printed out the matched rule number for each packet. After an initial checking for a portion of packets, both multiple thread version and serial execution give same matched rule number for each packet. We are running the code on the whole trace file to further verify this.

Experiments with 2 and 8 threads

We looked carefully into the variables and modified the code for parallelizing the recursive function. We figured out the problem we had yesterday on how to set number of threads. The code is working for the following experiments we have carried out.

Experiment 1: We made a smaller ruleset (containing about 100 rules rather than original 1802 to make the program faster) and run our program on a huge trace file (trace3a, about 2G). The number of threads is 8.

Experiment 2: We run the code on the same trace file and same rule set as in experiment 1, the number of threads is 2.

Experiment 3: we tested the same trace file and the same rule set without threads.

We need to look further into the results we got to make sure that the new code is semantically equivalent to the original code.

one step closer

We managed to get the code with parallelization to work using a small ruleset (the regular one times out/uses too much memory on the machine that it is running on).

There are a few current caveats that we have to work out:
1) We have not been able to get the command that sets the number of threads to work yet, right now it just uses the default of 2.
2) We will need to test to make sure that the new code is semantically equivalent to the original code
3) We will need to test the multi-threaded version to see how it performs
4) We may need to make changes that limit the number of threads that are created by the program

more errors

I haven't really been able to troubleshoot the seg fault error that well due to another error that we are getting. After a while the terminal just prints out Killed and we loose the connection to the new computer. I think the session is timing out, we will have to email Mike about this and see if he can help us fix it.

We decided to look at two different approaches to the seg error we had. Bryan is going to debug and find the reason why the error occurs. I'm looking at how to transform the recursive function to one without recursion. I'll first try using stack, which seems practical with multiple threads.

first try of recursion parallelization

Mike helped us to have a new machine to test the program with OpenMP. A simple try gave us a segmentation fault error message. We found some articles about this and will try other ways to parallelize the recursion.

Recursion

We applied a simple way to do recursion using OpenMP. We tried it on an example and it worked well. We will try it on rulecache.c. We usually remotely log onto a machine to run our programs, but that machine does not have gcc 4.2 installed and thus can not support OpenMP. We have contacted Mike who is responsible for the lab and he will help us on this.

OpenMP and recursion

We found the bottleneck function is a recursive function. We are reading more to understand how OpenMP works with recursion.

rulecache.c

We have found the second pass of function assignrhl() in rulecache.c, which tries to expand existing RHLs, takes longer time to run. After a careful examination, I found it is function check_conflict() in this section of code that actually runs slowly. We will try to parallelize this function.