Embrace Constraints To Evolve

Hosted forges a bit slow at the moment

At the moment your hosted Indefero forge can be a bit slow. Not terribly slow, but slower than what you can expect from the offer. This is the daily dump of each forge which starts to load the system too much. The good news is that for the past 3 days I have been working on getting that under control.

Every night (and it is now stretching itself during the day), the system does that for each active forge:

dump the content of the database in a JSON format;
make a full dump of each Subversion project;
evaluate the size of your forge database;
evaluate the size of each of your projects and attachments.

This is very CPU and disk intensive. The good point is that the servers running Indefero are powerful, but even very powerful systems cannot support a design which cannot scale.

What as already been done to mitigate this problem:

when you access your forge stats page, the size of your forge is not computed anymore in real time but in background by the backup process.

What is next on the list:

automated backup of your repositories at post commit;
control of the size of your forge on the backup server not to load the master;
on-demand JSON dump of your forge as it takes less than 300 ms per JSON file (you have two of them) to create them;
evaluation of the database size, only if the forge had some activity in the last 24h.

It is a hard process to get that done, it requires a lot of thinking to make it safe. I will still rsync twice a day to ensure a backup at the end of the day for both of you working from Europe/Asia and South/North-America.

Yes, scaling issues means that Indefero is growing both as hosted solution and free software. I am especially pleased to see that the average team size using Indefero is growing too. The new version targeting ease of use for teams of 5 to 15 working on 5 to 10 related projects will be spot on the current needs.

Loïc d'Anterroches,
May 18, 2010