Just a short notice. Our provider is not really happy with the stability of the routing. Basically, they try to get better BGP performance (I have no idea what it really means) and this requires some hardware changes and some experimentations. They are pretty good at doing that within minimum impact, but still, you may notice some downtime here and there. The downtime may not be full as this can come from partial announce of the routes.

Also, they are working on the UPS system because of minor warnings, the UPSes are complaining about fan problems even so the fans are running normally. The maintenance team of the UPS provider is on site to take care of the problem.

I really appreciate how open OVH is when dealing with their internal issues (their bug tracker is simply open to everyone), this is why Indefero is hosted with them.

Update: Everything is under control, no rotuing issues anymore and the UPS are under stricter control for a while.

It may be obvious, but I have not developed Indefero the best possible way. That is, I forget to correctly tap into the vast amount of information I have from the hosted forges.

Up to now, I am collecting only a minimal amount of information from the hosted forges. These statistics are more business related, that is, I need them to correctly run the hosting. These includes when was a forge active the last time (for backup), the storage used (capacity and backup planning), etc. I do not really care much about the rest. The forges are your private area, I am not looking at them.

But how to improve the software without knowing correctly its usage? Yesterday, someone submitted some comments through the open survey and then, suddenly I had this bell ringing in my head:

My plans for Indefero 2 are good but they are not connected with the reality of the software usage, because I have no ideas about this usage.

Back to square one and this time I am not going to throw the dices without a clear goal. So, I am going to build a real well defined statistics collector to know a bit more how you use and what you use when working with Indefero.

Better knowledge of your Indefero usage will be good. I will be able to streamline the frequently used part and focus the development where it matters. So Indefero 2 will wait a bit and for a good reason this time.

Oh, by the way, we crossed the line of 2500 active forges, yeah!

Planning is not available with Indefero at the moment, but I am actively working on it. Why? Because I need it to coordinate my work with the nice people who agreed to work with me for the version 2 of Indefero. I will be eating my own dog food. So the first iteration of the planning will be very simple but following the advices of Mike Cohn in Agile Estimating and Planning.

During the next few days, you will get preview of my work in the form of screen shots together with my ideas on getting these features implemented the right way. You can see below that at the moment the translations are not completed.

The users of Indefero on the hosted platform tend to have an important number of projects. 40 or 50 projects is not uncommon. When you create a new project, up-to-now, you had to tune it to match your needs after creation. If you create a new project each week, this is annoying, you need to repeat again and again the clicks and input.

To solve this problem, you have now the ability to create a new project and reuse another one as template. This way, you can reuse your last project or have a special ''template project''. As on the hosted platform, the number of projects is not limited, it costs you nothing to have the template project lying around.

They were good, really good, at least from a personal point of view. They were bad, really bad, from a technical point of view. Basically, I was able to enjoy some really good vacations because a failure in my way to synchronise my data prevented me to work and correctly perform technical support. I will slowly answer again your questions within the next few days, please be patient.

So, I learned something to not reproduce these failures in the future, the something is:

• Ensure to stop working with my desktop tools and switch to the laptop at least 3 days before leaving;
• Setup a web based technical support area to be able to answer even on the go through my mobile phone or a crappy GPRS connection;
• Get a general status page where you can see the status of the systems in real time;
• Get better status of the servers integrated in the general status page.

So, a big part of that will be performed this week, then I will work again on Indefero 2.

Update: Still some lag, still 154 unread emails in my inbox. Arg! With the very slow holiday WE, I suppose I will be able to have everything back in order.

Just for your information, I will be on vacations for a week from now. I will keep an eye on the different services through mobile notifications. Support tickets will be a bit slow. Also, I am not fully satisfied with the notifications with respect to the backup system, that is, I had to reset it this week but I got no notifications of downtime, this is really not nice at all. From here, I cannot control it, so please, download your backups this week and this will be my top priority when back at work the 26th of July. Thank you for your comprehension.

Just a short notice, to help getting Indefero 2 out of the door, I am looking for talented developers and designers for contracted work in July/August. If PHP has no secrets for you and/or if you like good design, drop me an email.

The pay is good, the work is interesting and all the commits in the Indefero code base will be done in your name, you get a nice reference work for the future at the same time.

For the requirements:

• you are really good at what you do, some references are welcomed;
• you can invoice me (yes accounting must be done following the rules;
• you can give me your hourly rate as a starting point.

If you are a PHP developer, the work can start now, for a designer, it is also fine to start only in August.

Update: Found 4 great developers, still looking for a designer. Indefero 2 is going to be great.

As last week was a great week for Cheméo, I need now to take care of a lot of things I left a bit behind when rushing to get everything ready for the annual meeting. The best thing to do is again a new NNW, so here it is, the Nothing New Week 4.

My goals are (and for once they are in the right order of priority):

• clean my 10 cm high stack of papers in my physical inbox;
• go through the 131 unread emails in inbox (and the other 400 read but not cleared);
• get my accounting in order (I still have 20k€ of invoices to issue, so this is critical);
• plan the work for the version 2 of Indefero which I want to go out by the end of August;
• plan the work for Cheméo, especially how to manage feedback from the companies using it. As Indefero is a free software without intellectual property attached to it, the companies using it can without any problem post their needs on the open tracker. With Cheméo, this is not possible as some feedback needs to be handled in private or semi private way. I will need to figure out the right way to do that.

At the moment, I am still really impressed at the synergies I get from developing Cheméo and Indefero at the same time. I learn a lot with one project that I then apply to the other. In terms of quality, they are now definitely better than if I would have developed only one. But I must say, I do not think it is possible to go beyond two projects and I have no ideas if this can last. Anyway, if this cannot last, this would be the results of having many customers. With customers comes money to hire the right people too, so...

Updates:

• Cleaned my physical inbox.
• Cleaned my professional email inbox.
• Migrated my desktop + laptop to the latest Ubuntu (so ugly design but changing to the stock Gnome theme fixes it).
• Had to do the daycare for my youngest son, the poor little one year old had during two days between 38.5⁰C and 39.6 and at the same time it was really hot here (nearly 30⁰C), this slowed down things.
• got nearly all my invoices in order. This means the cash to run the company for another year without even invoice a single time, yeah! My minimal cash value is 6 months and I try to always be above 9 months.

Left to do:

• Resetup of some of my development tools on my desktop and laptop.
• Use more VMs to develop. They are cheap and cost nothing when not in use. This means I can easily use them for testing (way cheaper than EC2) and it will save me a lot of time when reinstalling my systems, as I will not touch them. I think I will move more and more my development tools in the cloud. When I am offline, I usually write my ideas down as I enjoy these distraction free moments.

Today is a good day, the 15 minutes load average of the main Indefero virtual machine is now around 0.35, down from 2.1 yesterday at the same time and your repositories are safer than before as with a real time backup.

Real time is maybe a bit misleading, so here are the details of what is going on with your repository when you commit with Subversion or push with Git:

1. As usual when you push/commit, the rights are checked and if you are authorized, the repository is updated.
2. Just after the update, the Git post-update or the Subversion post-commit hooks are fired.
3. The fired hook informs the warm standby through a REST api powered by node.js that your repository has been updated.
4. The node.js coordinator schedule the retrieval of the latest version of your repository.
5. When it is your turn, your repository is retrieved on the warm standby. At the moment, the backup is completed within 10 seconds of your update.

Why this approach and not a simple drbd to have direct copy of the data? Because of corruption. When you sync the data using git or svnsync, the SCM is doing checksums which are not performed with drbd. Using a block level replication means that if your main storage is getting corrupted, you are going to replicate the corruption and end up with a total loss of your data. It happened to me once and a friend of mine got a corrupted RAID 5 array last month. RAID at the server level or over the network is not a backup.

What is next?

• The same approach to backup your issue attachments and uploaded files (done, see update below);
• calculations of your repository size on the warm standby;
• additional warm standby to get 2 warm standby servers for each master;
• more testing before this approach can be considered as production ready. At the moment, the daily rsync backup is still running and pushing the data in another place.

Friday Update: The attachments and uploaded files are now in the loop. At the moment, the backup is performed at the instant the job arrives (for the repositories too). So, the time to get the backup is basically just the data transfer time and on a 1Gbps link, this means nearly instantaneous backup.

Tonight, I did something wrong. I updated nearly 50% of the users password to access your Subversion repositories with the wrong password. Oups. So let me explain a bit how this Subversion password works and why such error.

As you know, your subversion password is not the same as your standard password, but it is derived from your salted/hashed password and the secret key of your forge. The problem in the update was coming from the secret key of your forge.

The secret key of your forge is a very long string used to perform hash control of a lot of elements in your forge. There is a unique secret key for each forge. The secret key, in the hosted forge setup, is composed of two parts: a baseline key and a unique per forge key. The resulting key is just the concatenation of both keys.

The Subversion password is composed of the first 8 characters of the sha1 hash of your salted/hashed password and the secret key:

substr(sha1($password.$secret_key), 0, 8);

So, now you know how one create your Subversion password.

Indefero is composed of two parts, the public website where you can create a new forge or upgrade your forge. This part has its own secret key. The hosted forges themselves, they have a baseline key + a unique key for each forge.

Now, when I upgraded the access right information tonight, I ran the upgrade script in the context of the public website to get the list of all the forges, then for each forge, I switched to what I called the ''forge context'' to perform the upgrade. But the baseline key was set to use the secret key of the public website as I used the public website configuration file. This is why all the accounts which were touched had their password regenerated with the wrong secret key, preventing you to access your forge. Please accept my sincere apologizes for this error.

Here is what I have setup to prevent such problem in the future:

• Automatic regular test update/commit/checkout against a test repository.
• Skeleton scripts to perform the maintenance with correct configuration.

At the moment your hosted Indefero forge can be a bit slow. Not terribly slow, but slower than what you can expect from the offer. This is the daily dump of each forge which starts to load the system too much. The good news is that for the past 3 days I have been working on getting that under control.

Every night (and it is now stretching itself during the day), the system does that for each active forge:

• dump the content of the database in a JSON format;
• make a full dump of each Subversion project;
• evaluate the size of your forge database;
• evaluate the size of each of your projects and attachments.

This is very CPU and disk intensive. The good point is that the servers running Indefero are powerful, but even very powerful systems cannot support a design which cannot scale.

What as already been done to mitigate this problem:

• when you access your forge stats page, the size of your forge is not computed anymore in real time but in background by the backup process.

What is next on the list:

• automated backup of your repositories at post commit;
• control of the size of your forge on the backup server not to load the master;
• on-demand JSON dump of your forge as it takes less than 300 ms per JSON file (you have two of them) to create them;
• evaluation of the database size, only if the forge had some activity in the last 24h.

It is a hard process to get that done, it requires a lot of thinking to make it safe. I will still rsync twice a day to ensure a backup at the end of the day for both of you working from Europe/Asia and South/North-America.

Yes, scaling issues means that Indefero is growing both as hosted solution and free software. I am especially pleased to see that the average team size using Indefero is growing too. The new version targeting ease of use for teams of 5 to 15 working on 5 to 10 related projects will be spot on the current needs.

After a while, you now have the post commit web hooks with Indefero. The hooks are modelled on the Google web hooks.

Basically, you register an URL in your project management area and after each commit an authenticated POST request is issued against the URL. The payload is a json fragment with information about the commit.

It is already available in the hosted forges. If you are hosting your own instance of Indefero, you need to pull the latest changes from the master branch to get them.

This is with a great pleasure that I can announce the availability of Indefero 1.0. Indefero started as a small project 21 months ago and grew to be now a very robust and easy to use system to manage your projects and code. Like hundreds of users of the hosted version and many more on the server install version (GPL), I manage all my projects with it.

Indefero is not anymore a simple software project, it is also a nice community of users and contributors, a business project and everything done while keeping our values at the center. We keep things simple, we keep respect among the developers and users and we keep the fun with us.

You can try Indefero now and tell me what you think about it.

You have been asking for it, now you have it. Each time you push new code in your git repositories, Indefero will now automatically send the notification emails. The implementation took a bit of time, because I wanted it to be general enough for the next step, that is, the same for Subversion and Mercurial.

A good side effect of this work is that it very nicely lays the ground for the implementation of the webhooks and more asynchronous work. All in all, it will improve your daily experience when using Indefero.

The code is running in production with the hosting offer of Indefero and is available in the master branch of the open source project.

Update: Now available for Subversion repositories.

This is a request from a couple of customers, so here you have, you can now add a custom header and a custom footer to your Indefero hosted forge.

As you already have the ability to use your own CSS and you can use your own domain, this means you have nearly full control on the look and feel of your forge without the need to manage it by yourself. Enjoy!

This afternoon (16h20 CET) a user of Indefero found a security vulnerability in Indefero. This vulnerability affects all the release of Indefero up to 0.8.9, release 0.8.10 (released less than 1h30 after the report of the vulnerability) provides a fix.

The vulnerability is in the git serving component. If a project is marked as private and the source available in read only to extra users, other users of the forge with a valid SSH key can have access to the project in read only mode if they know the short name of the project. In the case of the hosted offer, a user from another forge could not access the projects of your forge, the vulnerability was isolated at the forge level.

If you are using the hosted offer, the issue has already been fixed. If you have your own version of Indefero, here are the three possible ways to fix the vulnerability:

1. Upgrade to the latest 0.8.10 (recommended).
2. Patch the code with the following patch.
3. If you cannot update your code right now, change the access right in the source tab to "members or admin". You lose the read only access to the extra authorized users, but you fix the issue.

I am really sorry for this vulnerability, if you have any questions, do not hesitate to contact me through the mailing list or directly.

• Download the latest version.
• Individual patch.

Yeah! Say what you think about Indefero and you can with a business pack! You simply have to answer the survey. It takes 5 minutes and one from the first 100 answers will get a business pack!

Go answer the survey now!

The Indefero hosting is down, the virtual machine hosting the core of the service has lost access to the net, but can be accessed from the host. I am investigating the situation and will report here. No data loss.

Update: At 9:25AM CET the service is up again. Our provider was doing maintenance work on the routers and had some issues. This is not the first time we have a short period of downtime like that, in January at 1AM CET we had a 10 min downtime because of the reboot of one of the core router. I will check if it is possible to get the warm standby server in another datacenter from the same provider with the ability to switch the ip from one to another across the datacenters to help mitigate such situation.

This is with a great pleasure that I can announce the release of Indefero 0.8.9. This release mainly improves the source browser for Git, Mercurial and Subversion when using files with special characters in the path. The translation work is also starting to pick up some steam, if you speak German, just a little bit is left to get full translations, so come and help.

You can use it with the hosted offer or download it.

Please note that during the upgrade to the new infrastructure, the host RSA key changed. Here is the new finger prints:

dsa  1024 16:60:e9:af:37:0e:0c:68:df:76:7f:05:b8:42:a4:6b
rsa1 2048 74:44:1a:c4:b5:ed:34:ee:94:2c:11:47:2e:e6:a8:1b 
rsa  2048 d4:fb:83:55:fc:9f:99:07:50:8c:21:0c:3d:3a:19:fd

May you have any concern with the security, please contact me directly by email.