File downloads timing out and arriving corrupted

[digitalthinker]

Hi Michael,

I would like to thank you for a wonderful and simple PayPal extension. I've had quite a headache trying to take the big shopping carts from the Joomla community and fit them into a straightforward solution. Nice PayPal Downloads fit my needs perfectly so thank you for addressing the digital downloads so gracefully.

I've enjoyed using the Nice PayPal Downloads extension for a few days now and I've finally run into a little snag.

I've noticed that some of my customers have been receiving a corrupted file as a download. After some testing, I've discovered that the php timeout setting on my shared 1and1.com server was the culprit. After changing the max_execution_time in the php.ini and placing it in my root site's folder, the 15 minute default time was overridden and now the downloads time out after about 20 to 21 minutes.

I have purposely limited the download speeds in the settings of the extension to test the download timeouts (fast connection in my workplace doesn't let me reach the timeouts).

I came across another post on the forum (can't seem to find the link since) that suggested that putting the maximum concurrent download connections to 0 may help. Changing this value doesn't appear to affect the download other than not showing the download size and the progress.

My file size is limited to 400mb, and my file download is itself 289 mb. I uploaded the file by placing it in the correct folder via ftp.

My maximum speed is maxed out at 500k, and currently sitting at 50k for testing purposes so I can reach the file timeouts.

Do you have any suggestions where I can start my investigation and find out if my joomla setup, my extension settings, or my hosting service need tweaking?

Cheers,


Vlado

[Michael]

Hi Valdo,

I am happy to hear that you have found the Nice PayPal Downloads Extension so pleasant to use. Let's see if we can't get this pesky issue resolved, so you can continue your enjoyment.

Try this, open one of the corrupted download files in a simple text editor, like the kind you'd use for web work. I often use Notepad++ for this. Once opened, at the top or bottom of the file, you will probably see an error message. This error message should give you a thread to follow in respect to what is causing the corruption.

What happens behind the curtain, is that the download file is built in a virtual browser, so to speak, and then pushed to the visitor's browser for download. When an error occurs, the error message is pushed to the virtual browser and becomes part of the file. Obviously the error message does not adhere to the file syntax, so it causes the file to be corrupt.

I would think if the download is timing out, then a time out error message is being pushed along with the file and corrupting it. What is strange is that, in the extension's download script, I purposely disable PHP's time out feature. So, in theory the download should run 'til the end of time, without timing out.

Also, when setting the max concurrent connections to 0, you bypass the max concurrent connections feature. Some server configurations do not handle the concurrent connections feature, used to facilitate download accelerators, very well. As a result, download files can corrupt. So, by bypassing the feature, the file is served as one big file instead of small chunks. The bypass is more in line with the way a download would work if you clicked a link that pointed directly at the file.

~ All the best

[digitalthinker]

Thanks for the speedy reply and the behind the scenes explanation, Michael!

I installed NotePad++ and I have looked at the top and bottom of the files that are reporting as corrupted. Here is an instance of what I see throughout one of the files:

Code: Select all

<br />
<b>Warning</b>:  fread() [<a href='function.fread'>function.fread</a>]: Length parameter must be greater than 0 in <b>/homepages/14/d106605555/htdocs/http_footloosegames/live/components/com_nicepaypaldownloads/controllers/Download.php</b> on line <b>230</b><br />
<br />


The thing is that this file could potentially be the one that I downloaded prior to setting up the PHP timeout past 50,000 seconds. I tested with two files, one in Firefox, one in Chrome and neither of them has any clear error. All I see is binary data and a lot of NUL, SUB, ETX and other directives which must be part of the binary file itself.

Is there possibility that my shared hosting setup is causing something to break? What sort of dependencies does Nice PayPal Downloads have besides PHP?

Cheers mate,


Vlado

[digitalthinker]

I did more testing this morning on my Mac at home with hopes that I'll see something different.

I enabled concurrent connections this time around, and had a few downloads going at once so that I could watch them time out at a slower speed. They timed out again after about 20 minutes.

I opened the files in TextEdit and all I see is binary data. I'm not seeing any browser-specific code like I found in one of the older files in the previous post.

Any ideas where I should be investigating?

[digitalthinker]

I did some snooping on the web to see if I could find a discussion of a similar problem and with my very same host.

I found this one on stack overflow: http://stackoverflow.com/questions/2084067/php-script-stops-running-arbitrarily-with-no-errors

His problem described is exactly what I'm seeing, other than the fact that my downloads appear to time out with more consistent timing.

Take a look at the third and fourth-last comments at the bottom of the page by James and Rob. I'm not a web expert by any means so the technical explanation is something that went a bit above my understanding, but I'm wondering if their suggestions there are useful to resolving my timeout issue? Rob claims that his method of approach fixed his timeouts, although I'm pretty sure that I've read another thread on this forum before that would suggest that Nice Pyapal Downloads is already implementing something like this.

Hope this is useful in tracking down the cause

Cheers,


Vlado

[Michael]

HI,

Well, your online research is very helpful. To give you some insight on the extension, when the download process is initiated there is a PHP setting that gets called:

set_time_limit(0);

This should effectively stop PHP from timing out the script. In theory, PHP should continue to run the script without a time limit.

Now, you have stated that you are already diddling with time limit settings and the process is still timing out. The curious thing is that there is no error message. If it were PHP shutting the process off, there should be an error message.

The second answer from the bottom (by Hendy Irawan) on the stackoverflow.com site (http://stackoverflow.com/questions/2084 ... -no-errors) is the most compelling answer. If you are on a shared hosting site, your host may very well have a program that terminates zombie processes after a set period of time. A zombie process is a term that means any computer program that is mindlessly running in the background using up system resources and not accomplishing anything. So, the host may have a zombie process termination program that is mistaking your download script as a zombie process.

I highly recommend that you check with your host's tech support. They will be able to verify this and/or provide additional information about the server. This additional information may provide the key to understanding what is happening here. Put it to them simply. Let them know you are running a download script in PHP. It is a proven mature script and runs successfully on other servers. But when you try to download large files over long periods of time on this server, the process always gets terminated early. Ask them if they why they think this might be happening and ask them if they think it might be due to any zombie process termination program they might be running.

~ Best regards

[Michael]

In addition to my above post, I wanted to add that the files are arriving corrupt because they never finish downloading.

~ All the best

[digitalthinker]

Very informative response, thank you for walking me through this.

I had a chance to speak with the folks over at Canadian Cloud Computing who we share an office facility with, and they are convinced that shared hosting is the problem. Besides the resource limitations, they described that most hosts will have a set of services that will hunt down rogue processes in order to keep their uptime high. So in essence, what you're describing as a zombie process seems to be exactly what my server is seeing these downloads as.

Even more interestingly, I just took a look at my php info that joomla displays in the system info tab and what I'm seeing is not what I saw yesterday. After placing the php.ini file a couple of days ago, I was able to get joomla to see the new max_execution_time value. I checked again today and it's been reset, and the php.ini file path no longer shows under the "Scan this dir for additional .ini files" label. Oddly enough, the php.ini file is still sitting where it should be. It's quite likely that my host didn't like the timeout of 500,000 sitting in there and did something to block it.

My best conclusion is that my hosting is not adequate for what I need to do with it. I will be looking into a virtual private server and migrating to it in the next few days. I will let you know if these problems persist or if a proper hosting solution is to blame

[Michael]

Hi,

Thanks for the update. It does sound like your host has some automated server management processes running. As you mentioned a VPS may be more suitable for your needs. I would recommend contacting your host first, before migrating the site. Sometimes, support teams are able to make exceptions or alterations to suit the needs of particular clients. You may find that a quick support ticket submission will resolve the issue and save you the time of having to migrate the site. On the other hand, running on a VPS gives you a lot more control, and in my opinion preferred over shared hosting solutions.

~ All the best

[digitalthinker]

I just struck a sweet deal with Canadian Cloud Computing guys so the VPS will be ready tomorrow. The CCC guys are down the hall from our studio so the face to face support will go a long way.

We're also expecting a minimum of 10k visits and 1k downloads a day for a period of 3 weeks at the end of March so I'll make sure we're well prepared with the VPS. Having seen the tech specs of my existing shared hosting, I don't think we'll be able to meet that demand without migrating.

I'll be playing with this over the next couple of days and I'll let you know how it goes. I'm really excited to get the plugin up and running

[Michael]

That's great, let me know how it goes.

~ All the best

[digitalthinker]

I migrated the website to the new host and everything appears to be working ok so far. The downloads continue to time out, now after a shorter period of time of about 5-7 minutes. Further research on the web suggests that php, fastcgi and IIS are the things that control timeout values that could be causing problems.

Now, I'm on a linux server so the way I understand is that IIS is not present as it's a windows thing (correct me if I'm wrong). My php values are set up correctly, so that only leaves fastcgi timeouts.

i've read searched around the web but I'm unable to find clear instructions on how to change the fastcgi timeout values. Do you have any recommendations on how to do this or where to look?

Cheers,

Vlado

[Michael]

Hi,

You are correct about IIS. IIS is 100% Windows. As for FastCGI, I am not familiar with it. Thinking about the time outs being on a Linux box, perhaps it is an Apache setting that is causing them. What does your web host's tech support think?

~ Best regards

[digitalthinker]

I looked at the apache log and noticed that the download processes were being terminated. Here is one of the errors:

Code: Select all

error.log:[Sat Mar 19 02:00:18 2011] [error] child process 29851 still did not exit, sending a SIGKILL

At this point I'm not sure where I need to make changes and what needs to be tweaked. I'm still waiting on a response from the host's technicians on the error.

[Michael]

Nice catch!

How long is the download running before it is killed? I am able to run a download for over an hour on my VPS without difficulty. Your host may be able to alter the default Apache setting for you.

~ Best regards