Strace buggy php script running as cgi – Strace daemon

Very often I need to strace a script which is not “constantly” running, and must be “catch” during it’s execution.

Today I had to debug some bad behaving php script, but because php was running as CGI (mod_suphp) it was hard a task.

I had 3 choices in order to successfully attach the strace to the php process:

METHOD 1: Attach to Apache during startup and then strace the php execution

This is a good choice and I’ve done it multiple times before. The process is pretty simple – when you strace to Apache at the beginning (strace /etc/init.d/httpd start)and if you are using strace -f option,  your strace will automatically attach to the forked by Apache php CGI process.

 

But there are some BAD requirements, which were not good for my current case:

• There must be no hits to the web server except mine. This is necessary in order to be sure I’m stracing the right hit/script/process;
• In order to satisfy the upper requirement, we need to stop all external traffic and make sure that we are the one and only hitting this;
• For a production environment that’s a bit shitty way to go … so I just IGNORED it

 

METHOD 2: Change the way PHP is running and make it execute as FastCGI

The idea is very good and worked multiple times for me, BUT there are still some “BUTS”  and they are:

•  Again as stated in “1” you must assure, that you are the only visitor while testing/troubleshooting. This brings downtime for the site, which not always is acceptable;
• Sometimes (not so rare) the site may NOT work after switching to FastCGI ; That’s pretty annoying, especially if your client noticed the malfunctioning before you :)

METHOD 3: Use custom deamon like bash script for stracing the process

My final struggle ended with creating simple bash script, which does the following:

– Starts infinite while loop, waiting for process matching predefined criteria

– When the process shows, we get it’s PID and attach strace to it

– The process is defined as grep style regex, which the script tries to find in the ps aux list

Of course there are some drawbacks too using this method:

• If the site is intensively visited you may not hit the right pid – it may be someone else’s hit; In my case the script was one that people access very rare;
• Stracing won’t be FULL ! Given the fact we are in a while loop (in the script), there’s some time needed while the loop catches process execution and attach strace to it. During this “black hole” of detecting and attaching, things WILL happen in the script, which ARE NOT going to be logged by strace; In my case that wasn’t a problem cause I didn’t care about the beginning of the script.

And here is the script itself:

 

#!/bin/bash

STRACE_OPTIONS=""
PID_REGEX=""

function usage {
        cat << EOF 
        Usage: 
                ${0##*/}  [-s strace_options] [-p  pid_regex] 

        The program will start a while loop and wait for a process, which matches the pid_regex 
        criteria (by executing ps aux | grep -E "pid_regex"); 
        When we have some processid found, we will strace it by using the strace_options supplied.

        Example:
                $0 -s "-e write -s 2000" -p "my_bad_suphp_script"
EOF
        exit 1;
}

# ============ GET OPTIONS ==============
while getopts "s:p:" option
do
        case $option in
                s) STRACE_OPTIONS="$OPTARG";;
                p) PID_REGEX="$OPTARG";;
                *) usage ;;
        esac
done
option_regex="[a-z]|[A-Z]|[0-9]"
# Check for empty options
echo $STRACE_OPTIONS | grep -E "$option_regex" &> /dev/null;
if [[ $? -ne 0 ]]; then 
        usage
fi
# Check for empty options
echo $PID_REGEX | grep -E "$option_regex" &> /dev/null;
if [[ $? -ne 0 ]]; then 
        usage
fi
# +====================================+ #

echo "Strace options: $STRACE_OPTIONS";
echo "PID_REGEX: $PID_REGEX";

while [[ "$pid" -eq "" ]]; do
        pid=$(ps aux | grep -E "$PID_REGEX" | awk {'print $2'});
        if [[ $pid -ne "" ]];then
                strace $STRACE_OPTIONS -f -p $(ps aux | grep -E "$PID_REGEX" | awk {'print $2'})
                break;
        fi 
done

 

 

In my case the steps of using the script were the following:

 

Start the strace “daemon”

# "-e open" tells strace to print only "open" syscalls
# "-s 2000" tells what size to print

root@web_server#bash strace_suphp.sh -s" -e open -s 2000 "-p"my_test_php_script.php"

 

Issue the hit:

curl http://example.com/my_test_php_script.php