php FastCGI 503 Service Unavaible

[Reinis Rozitis]

Hello,
Is there a way to avoid the 503 Service Unavaible errors?

I'm using lighttpd with spawned like 50 - 60 childs of fastcgi. The good
part is server load is about 60% lower than using mod_php with apache (at
least it can be controled). But as the php scripts are very cpu/memory
intensive in peak hours when the requests per second are over 60-70 there
are times the error page shows up which is not very friendly.

So is there a solution in case lighttpd cant get response in first try (or
it is broken) try to make a second (or even third) request to the fastcgi
process?

rr

[Reinis Rozitis]

There are 2 types of errors.. The first as far as I understand is ok, as it
seems to handle closed connections.

2004-10-19 00:17:00: (network_linux_sendfile.c.77) write failed: Connection
reset by peer 21
2004-10-19 00:17:00: (connections.c.526) connection closed: write failed on
fd 21
2004-10-19 00:17:00: (fcgi.c.1320) emergency exit: fastcgi: connection-fd:
21 fcgi-fd: 27


This one seems the case in which the 503 error is returned. Would be nice if
lighttpd tried to run the script again.

2004-10-19 00:38:19: (fcgi.c.1004) unexpected end-of-file (perhaps the
fastcgi process died): 123
2004-10-19 00:30:43: (fcgi.c.1004) unexpected end-of-file (perhaps the
fastcgi process died): 233
2004-10-19 00:30:49: (fcgi.c.1004) unexpected end-of-file (perhaps the
fastcgi process died): 221

Forgot to mention the connection is through socket:

fastcgi.server = ( ".php" =>
( "fphp" =>
(
"socket" => "/tmp/php.sock"
)
)
)

----- Original Message -----
From: "Jan Kneschke" <***@kneschke.de>
To: "Reinis Rozitis" <***@roze.lv>
Cc: <***@lists.kneschke.de>
Sent: Wednesday, October 20, 2004 11:25 PM
Subject: Re: php FastCGI 503 Service Unavaible

Post by Reinis Rozitis
Hello,
Is there a way to avoid the 503 Service Unavaible errors?
I'm using lighttpd with spawned like 50 - 60 childs of fastcgi. The good
part is server load is about 60% lower than using mod_php with apache (at
least it can be controled). But as the php scripts are very cpu/memory
intensive in peak hours when the requests per second are over 60-70 there
are times the error page shows up which is not very friendly.
So is there a solution in case lighttpd cant get response in first try (or
it is broken) try to make a second (or even third) request to the fastcgi
process?

Please check the error-log what the reason for the 503 is in you case.

Post by Reinis Rozitis
rr

Jan
--
Jan Kneschke http://jan.kneschke.de/
Perhaps you want to say 'thank you, jan': http://jk.123.org/wishlist/

[Jan Kneschke]

Post by Reinis Rozitis
There are 2 types of errors.. The first as far as I understand is ok, as it
seems to handle closed connections.
Connection reset by peer 21
2004-10-19 00:17:00: (connections.c.526) connection closed: write failed on
fd 21
21 fcgi-fd: 27
This one seems the case in which the 503 error is returned. Would be nice
if lighttpd tried to run the script again.
2004-10-19 00:38:19: (fcgi.c.1004) unexpected end-of-file (perhaps the
fastcgi process died): 123
2004-10-19 00:30:43: (fcgi.c.1004) unexpected end-of-file (perhaps the
fastcgi process died): 233
2004-10-19 00:30:49: (fcgi.c.1004) unexpected end-of-file (perhaps the
fastcgi process died): 221

This might be the dieing PHP process which kills itself after 1000 or
10000 requests.

Can you check that about 1000 or 10000 requests have been sent to
php-server before those appeared ?

If yes, i could provide a way to handle those.
Should not cause a problem.

Jan

[Reinis Rozitis]

Can you check that about 1000 or 10000 requests > have been sent to
php-server before those appeared ?

Could be, but its kinda hard to check to be 100% sure if the fastcgi process
has been killed because of the request limit (now I have set 5000).
There are like 1 mil hits per day (just php pages) and with most of the
debug/log info turned off (otherwise the log grows way too fast).

If yes, i could provide a way to handle those.

I would be very glad if you could include additional fcgi process
handling/balancing.
This would be a nice reason to make something true from your wishlist :)

rr

[Jan Kneschke]

Post by Reinis Rozitis

Can you check that about 1000 or 10000 requests > have been sent to
php-server before those appeared ?

Could be, but its kinda hard to check to be 100% sure if the fastcgi process
has been killed because of the request limit (now I have set 5000).
There are like 1 mil hits per day (just php pages) and with most of the
debug/log info turned off (otherwise the log grows way too fast).

Just tell me how many requests have been served by php and how many
resulted in a 503 condition ?

Post by Reinis Rozitis

If yes, i could provide a way to handle those.

I would be very glad if you could include additional fcgi process
handling/balancing.

Give me some pointers.

Post by Reinis Rozitis
This would be a nice reason to make something true from your wishlist :)
rr

Jan

[Reinis Rozitis]

Post by Jan Kneschke
Just tell me how many requests have been served by php and how many
resulted in a 503 condition ?

Ok in time span from 2004-10-19 00:17:00 till 2004-10-19 17:34:20
there were ~400 000 requests and ~6379 resulted in 503 error.
It kinda looks right if we count that every fcgi serves 5000 requests and
there are 50-60 childs.

Of course they show up when the load is high so the in the middle of day
users see them quite often.

Post by Jan Kneschke
Give me some pointers.

The solution to this in case of broken/unsuccesfull response from fcgi
server maybe to try make a request to another fcgi process (though this
could degrade speed).

If we are correct and this happens only when php fcgi child dies, maybe it
could be avoided to set the request count just enourmos or somehow just make
the fastcgi server childs permanent?

rr

[Jan Kneschke]

Post by Reinis Rozitis

Post by Jan Kneschke
Just tell me how many requests have been served by php and how many
resulted in a 503 condition ?

Ok in time span from 2004-10-19 00:17:00 till 2004-10-19 17:34:20
there were ~400 000 requests and ~6379 resulted in 503 error.
It kinda looks right if we count that every fcgi serves 5000 requests and
there are 50-60 childs.
Of course they show up when the load is high so the in the middle of day
users see them quite often.

Post by Jan Kneschke
Give me some pointers.

The solution to this in case of broken/unsuccesfull response from fcgi
server maybe to try make a request to another fcgi process (though this
could degrade speed).

Yes, something like this.

Post by Reinis Rozitis
If we are correct and this happens only when php fcgi child dies, maybe it
could be avoided to set the request count just enourmos or somehow just make
the fastcgi server childs permanent?

Permanent is not a good idea as PHP might leak memory. Keeping them
dieing from time to time is a good thing. But please check if the number
of 503 drop if you increase the PHP_FCGI_MAX_REQUESTS setting.

Post by Reinis Rozitis
rr

Jan

[Reinis Rozitis]

Permanent is not a good idea as PHP might leak > memory. Keeping them
dieing from time to time is a good thing. But >please check if the number
of 503 drop if you increase the >PHP_FCGI_MAX_REQUESTS setting.

Seems that there are also other kind of problems..
Increased PHP_FCGI_MAX_REQUESTS to 200000 (which imho couldnt be depleted)
but some of the test users still get sometimes the 503 error page (though
less than before).

On other hand it could be on the php side (bugs in fastcgi api). Still as
the server load is significant lower (~5 times) than running with apache,
maybe you could provide patch (if not including in the official release of
lighttpd) that made the fastcgi module to make check the response from fcgi
server and in case of incorrect result make a second one.

rr

[Jan Kneschke]

Post by Reinis Rozitis
Hello,
Is there a way to avoid the 503 Service Unavaible errors?
I'm using lighttpd with spawned like 50 - 60 childs of fastcgi. The good
part is server load is about 60% lower than using mod_php with apache (at
least it can be controled). But as the php scripts are very cpu/memory
intensive in peak hours when the requests per second are over 60-70 there
are times the error page shows up which is not very friendly.
So is there a solution in case lighttpd cant get response in first try (or
it is broken) try to make a second (or even third) request to the fastcgi
process?

Please check the error-log what the reason for the 503 is in you case.

Post by Reinis Rozitis
rr

Jan