Clone from staging.git.l.o is very slow when using dumb protocol

After opening the admin settings page the above error don't appear anymore...
Further testing show that the 'smart' protocol works while the 'dumb' one never finishes (like an infinite loop) with the following logs picked up from the server https://pastebin.linaro.org/2147/

After some testing it turns out that the clone is actually very slow, here's some comparison of the clone with smart and dumb http protocol: https://pastebin.linaro.org/2149/

The suggested fix is to serve those files with apache using mod_xsendfile... linaro-license-protection should be doing something similar

I'm taking a look at the xsendfile module in Apache and will test locally how it can work.

The attached patch rewrites the dumb HTTP protocol we use in RhodeCode with an X-Sendfile directive.
In this way files are served by Apache instead of being served by RhodeCode.

It is necessary though to install libapache2-mod-xsendfile and to enable it in the Apache configuration.

Clone operations time is slightly improved, to decrease it even further we might consider using some sort of caching.
It is also possible to set ETag and Expires headers directly within RhodeCode.

Code has been merged and deployed on staging instance. Config repository has been updated with correct Apache directives.

Dumping some thoughts for us to test already mentioned on IRC...
I know apache isn't the bottleneck but if we lift some weight from it, we might get some improvements, so I think it's worth giving the options below a shot:

1. git-http-backend: it can be configured with apache without interfering with Rhodecode. Found this article http://p0l0.binware.org/index.php/2011/08/26/git-over-http-git-http-backend/
Of course it should be configured with accelerated static for apache 2.x options, which can be found in official git-http-backend documentation here https://www.kernel.org/pub/software/scm/git/docs/git-http-backend.html

2. Play around with mpm_worker setup and try mpm_prefork as well. Like reducing the start server and spare threads might free up some memory and cpu power for us to make things go faster. And prefork has proven better in some concurrent testing cases for me a while ago. Again, we should think about the options and test various settings.

Also, I'm bumping the priority on this one because we mustn't go to production if we don't at least something out.

On Tue, Apr 16, 2013 at 4:32 PM, Stevan Radaković
<email address hidden> wrote:
> Dumping some thoughts for us to test already mentioned on IRC...
> I know apache isn't the bottleneck but if we lift some weight from it, we might get some improvements, so I think it's worth giving the options below a shot:
>
> 1. git-http-backend: it can be configured with apache without interfering with Rhodecode. Found this article http://p0l0.binware.org/index.php/2011/08/26/git-over-http-git-http-backend/
> Of course it should be configured with accelerated static for apache 2.x options, which can be found in official git-http-backend documentation here https://www.kernel.org/pub/software/scm/git/docs/git-http-backend.html

That is exactly what we are doing on staging.android.git.l.o.

The problem I see with that approach, at least with RhodeCode in mind,
is that RhodeCode becomes sort of useless.
We would need to bypass it if we want to do like on staging android,
or we sort of end up doing the same thing that RhodeCode was doing
before with the smart protocol: calling and external process, running
git and passing the output to the original request. That might become
again the bottleneck.

Bypassing RhodeCode means also we will lose all the access control features.

> 2. Play around with mpm_worker setup and try mpm_prefork as well. Like
> reducing the start server and spare threads might free up some memory
> and cpu power for us to make things go faster. And prefork has proven
> better in some concurrent testing cases for me a while ago. Again, we
> should think about the options and test various settings.

I'm trying locally with that, cloning 8 repositories at the same time.
At the moment I do not see much difference on Apache side.
The big CPU usage always happens from RhodeCode.

> Also, I'm bumping the priority on this one because we mustn't go to
> production if we don't at least something out.

We will definitely not go in production at this stage. We will
probably lift git.l.o as is now, and move it to another server.
We'll still work on this during this week, sync up with Alan tomorrow,
and then probably with Philip in the next days.

--
Milo Casagrande | Automation Engineer
Linaro.org <www.linaro.org> │ Open source software for ARM SoCs

Did some work this morning, and found out that not all the git requests were served through X-Sendfile, only the first one, so the majority of them were still served by RhodeCode directly.

Will update the production server today with the latest changes.

Changes have been deployed on staging, along with some others, but clone times are still very high when the server is under a high load (multiple clone operations at the same time).

After some work, we finally came to the culprit of the problem: user access requests were not cached by RhodeCode and with the dumb HTTP protocol, each requests had to be granted access for the same user with the same IP on the same repository.

Introducing a small caching mechanism solved the problem. Changes have already been applied on the staging instance and tests are being performed.