Centos 6 – Curl , SSL and Illegal Instruction error

Published on Author gryzli8 Comments

The “Illeagal Instruction” problem

If you recently started to see errors like “illegal instruction” while trying to execute different web accessing scripts/binaries (like curl httpS://example.com), then most probably your Centos 6  NSS package was just updated.

 

The impact of this bug is enormous, because every single HTTP library which uses nss,openssl for maintaining HTTPS connections would crash.

This includes all kinds of: wget, curl, libcurl, php + curl, perl + curl, perl + lwp … and so on  (endless list)

 

This bug causes the script to immediately crash, and if you strace it you will see that it ends with “SIGILL“, which is a SIGNAL for using illegal cpu instruction.

 

The problem is already reported in Redhat Bugzilla, and here is the thread:

https://bugzilla.redhat.com//show_bug.cgi?id=1249426

 

The problem is mostly seen when your VPS is running under XEN hypervisors.

 

 

What is the temporary fix for this ?

As suggested in the redhat bugreport also, the fix is to add the following environment variable:

export NSS_DISABLE_HW_GCM=1

 

Fix script executing and init scripts

So if you run curl, or some other script hitting the bug, you can change your command to:

NSS_DISABLE_HW_GCM=1 curl https://example.com

or even edit your profile, and init scripts:

vim /etc/init.d/apache

--> put "export NSS_DISABLE_HW_GCM=1" at the beginning

vim /etc/init.d/some_other_software

--> put "export NSS_DISABLE_HW_GCM=1" at the beginning

Add it to profile.d

echo "export NSS_DISABLE_HW_GCM=1" > /etc/profile.d/fix_nss_bug.sh

 

Fix the Cron Jobs

In order to add the environment variable to all newly spawned cron jobs, follow the steps below:

 

Add the following to /etc/pam.d/crond

vim /etc/pam.d/crond

 

vim /etc/security/pam_env.conf

 

UPDATE (18.06.2016)

Fix PHP and  PHP-FPM

If your php-fpm or php CGI, doesn’t work after fixing the FPM init script or Apache/Nginx init scripts there is another dirty fix for this.

I was getting the following lines on one of my servers with php-fpm:

This could be “dirty fixed” with the following steps:

1) Create some PHP file with following content

– Keep in mind there is NO “?>” closing tag.

 

2) Now you need to prepend the file above to  all of your php scripts

Happily, PHP already has such functionality, so you only need to add the following line in your global php.ini file.

 

3) Finally restart your php-fpm or your web server in order the change to take effect

 

Fix cPanel Softaculous, RVSiteBuilder  or another plugin/cPanel misbehavior

If you happen to use cPanel as a hosting control panel, then you might have problems with some 3rd party plugins (or 1st party), which are affected by this bug.

The problem is that cPanel uses it’s own web server instance, for running the cPanel/WHM Web GUI.

In order to fix their internal PHP, you must do the same as in the “Fix PHP and PHP-FPM” step, with the only difference, that you must include the file into their internal php.ini:

/usr/local/cpanel/3rdparty/etc/php.ini

 

When I first reported this problem to them, they didn’t have any representative info about this, but now they have created a dedicated post, with detailed information, how to fix yourself on a cPanel based server:

https://forums.cpanel.net/threads/xen-hw_aes-detection-issues-yum-update-illegal-instruction.551681/

8 Responses to Centos 6 – Curl , SSL and Illegal Instruction error

  1. Thanks for the Info. Ran into this while doing test-work in AWS. Thought it was because t1.micro instances have too little memory (stopping a t1.micro PV instance and restarting as an m1.small or m3.medium also seems to make the problem go away). Nice to see can continue to use t1.micros for lower-cost taskings.

  2. Hi,

    That sounds very interesting.

    I suppose that Amazon are using Xen based hypervisors for the t1.micro instances ,and KVM (or something different) for the small, medium instances :)

  3. I can confirm that in our XenServer environment, we’ve seen the problem only on paravirtualized hosts–not on fully-virtualized hosts. So if you have an option to use HVM virtualization, it may “solve” things as far as nss-based applications are concerned.

  4. Thanks a TON. I spent the better part of a day trying to figure this out after yum updating NSS. It immediately broke my curl calls to all AWS EC2 servers. Even after yum downgrade of all NSS stuff. Scripts calling AWS with cUrl would work fine when called from the shell, but when passed to fast-cgi PHP-FPM in Nginx using a web browser the exact same scripts would cause the SIGILL error mentioned above and terminate the process. This was a tough one to troubleshoot — only affected the AWS remotes… something with SSL (?) Again, thanks! I’m up and running again thanks to you.

Leave a Reply

Your email address will not be published. Required fields are marked *

Captcha * Time limit is exhausted. Please reload CAPTCHA.