A small notice to inform you that Elveos.org is using Indefero:
Elveos.org is a crowdfunding website for open source software. You are a free software developer? Elveos gives you a way to get paid for your work. You are a free software user? Elveos let you fund the features you need.
It reminds me KickStarter but more open. It is very nice to see offers targeting the OSS community.
If you noticed a slow down in the past minutes, one of the routers of our provider had some issues. This slowed down the services for a short period of time. As you can see on the following graph, suddenly our GET requests to monitor the response time of the services went bong. 20 second response time, this is the equivalent of dead...
But I must say, this is where I am really pleased by our provider, OVH. They immediately explained what was going on: a card of the router crashed hard or died.
For your information, we are in the process of migrating Indefero's main backup server on our new infrastructure. The new infrastructure has been running for a while and we are satisfied with the stability.
We are going to do at the same time a server upgrade, moving away from Ubuntu and back to the roots, that is Debian. Once the backup server will be up and running smoothly, the main server will follow.
Update: Got a bit of instability at the same time... upgrading here and there an old server is difficult. Time to get the migration to a better system completed!
Few days ago Cheméo's laboratories went life. The labs are running software experiments in the field of chemical and physical properties. They are kind of sandboxes where ideas can be tried without disturbing the main Cheméo website.
The labs are running on top of Céondo's private Platform as a Service (PaaS). This platform will soon host all the services we deliver, from our products Cheméo and Indefero to simpler websites like ceondo.com. In case of, a status website will be kept independently using another technology with a different provider. I will soon write a bit more about this private PaaS.
These are exciting times, the best to close 2011 and start 2012.
Improving the speed of Indefero is challenging as it requires managing a lot of moving parts, from the git/subversion backends to the database. This week, I have been working on setting up Graphite for the infrastructure. This is working pretty well and provides graph like the following one.
This graph is extracted from a special Nginx log format which includes the time needed for Nginx to send the response back to the client. The only thing missing is that when I see a spike, I need a way to directly access the corresponding logs to figure out why. At the moment, there are no integrations between these metrics and the logs.
To improve a system, one needs to know the current state. Graphite is a bit hard to setup, but afterwards, it is really easy to push data in. A really nice tool.
To run a service like Indefero, you need to log a long list of metrics to follow the load on the system, find the bottlenecks and predict the future needed capacity. To do that, a very powerful system is Graphite, the only issue is that it is only storing and graphing numerical values. Of course, you cannot do different, but the problem is: correlation.
Basically: Once I see that every now and then component is not performing well, how can I drill down in my data to find the reason?
Graphite tells you: this day from 14:05 to 14:07, the rendering of a git tree view was slow. Good to know, the following question is of course: why? If you store more metrics, you can maybe find that I/O was slow on the server X, you can graph together many metrics and visually correlate them. But then, why was I/O slow?
At this point, you need to go one level deeper and take a look at the logs coming from server X from 14:05 to 14:07. This can bring you up to the application level where you figure out that a client repeatedly accessed a page which triggered a git command with a large output, thus loading the server. But to do that you need to access the logs too.
So, Graphite is wonderful, but what I need is that after identifying the subsystem and time range where we have an issue, being able to simply scan through all the corresponding logs in the time range. This would be a kind of integration between Graphite and Graylog2.
My problem now is that Graylog2 is overkill. That is, it tries to provide full text search on the logs, the result is that it requires a very big machinery where I just need aggregation of the logs and the equivalent of a time base search range with a filtering by component, for example webapp.backend.git.
This annoys me, I do not want to build a system by myself.
Around 11:14 UTC today one switch of the private network went down and required a reboot. The problem has been solved but this resulted in a downtime of about 8 minutes.
Note that this is the issue with the new database server, if the link between the application server and the database server goes down, then the service is down. I will contact the support staff of OVH as in my understanding, they had a kind of redundant system to not rely on a single router.
Hello, just to let you know that today November 11, the database migration is starting. You can check this blog post for updates. Here are the steps I will be performing:
This is a bit of cascading but it will always keep several version of the database running and it will always be possible to revert to the original DB server in case of problem.
Last update: The system is now insanely more responsive, pleasure to use is back! If you notice anything unusual, please let me know as soon as possible.
The Decane is a simple molecule but also the name of our new database server. It is a 24GB RAM/240GB SSD server with a lot of power to provide blazing fast data processing. In the next few days it will go through the standard Ganeti setup and the Indefero PostgreSQL database will be migrated over. Depending of the performance, we may migrate more database VMs on it.
This is the first time I am putting a server with SSD drives in production. I have been an heavy user of SSD drives for my desktop/laptop systems in the past two years and I must say, I will never go back to traditional drives, but of course, the amount of data stored is not the same for desktop and for a server.
So, yes, performance increase of Indefero is on the way!
Update: Decane just joined the Ganeti cluster:
# gnt-node list
Node DTotal DFree MTotal MNode MFree Pinst Sinst
node1.ceondo.net 2.7T 2.2T 11.8G 4.0G 8.2G 4 0
node2.ceondo.net 2.7T 2.4T 11.8G 2.9G 9.2G 5 0
node3.ceondo.net 194.3G 194.3G 23.6G 147M 23.4G 0 0
Update 2011.11.09: Base backup of the postgresql database is on the way, this is a huge rsync job and this is of course slowing down the system. Please be patient... thank you.
Update 2011.11.09 20:00 UTC: Ok, the new server is now acting as a warm standby for the database, this will allow fast "failover" to the new database server after the testing period.
Update 2011.11.10 10:28 UTC: The main application server will be unavailable for a short amount of time the time to connect it to its virtual LAN to communicate directly with the warm standby over a private network. Done.
Update 2011.11.10 11:46 UTC: Now that the connection at the switch level is supporting the VLAN, it needs to be configured at the host and vm level. This will again trigger short downtime here and there.
Update 2011.11.10 12:50 UTC: Ok, now setting up a second warm standby which will take over the current one on the new SSD powered server once it will start to act as master. Done.
Update 2011.11.10 13:34 UTC: Ok, things are running as expected, around 21:00 UTC today, Indefero will be stopped for about 15 minutes, this will ensure that we have the warm standby with the latest version. The warm standby will be brought online as master and then the web app will connect to the new master on the new server. Immediately, I will start to populate the new warm standby. Basically a bit of cascading.
Update 2011.11.10 19:20 UTC: Too tired to do the cascading, it is never good to do so when not really fresh. I will perform it tomorrow, ok it will be during the day, but it will be only about 15 minutes of downtime. So expect a downtime of maximum 30 minutes between 09:00 and 12:00 UTC on November 11.
In the past 24h, I have been testing the latest developer version of Indefero on projects.ceondo.com and I must say, I am pretty pleased by the stability of everything. Congratulations to all the developers involved in getting this done. The only problem is that it requires some heavy migrations and update of the database storage and I must say, I do not like that.
So, some read only tests will be done tonight to check how this can be done without disturbing you too much and if things are going well, the upgrade will be performed. The upgrade can take a very long time. Yes, upgrading 3000 databases is a bit painful.
Anyway, if you take at the new version on projects.ceondo.com, I suppose you will accept this small annoyance as it will bring you a lot of goodies.
Today, the 27 of September, around 23h00 Paris/Berlin time some of the background processes of Indefero will not run. Our provider is performing some network maintenance.
Basically, they are rewiring our private network, which means that the background communication between the components will be cut. This also means that some website which have data routed over the private network will not function. The total maintenance time is expected to be about 15 minutes. Thank you for your comprehension.
The Git User's Survey 2011 is now up!
Please devote a few minutes of your time to fill out the simple questionnaire; it'll help the Git community understand your needs, what you like about Git (and what you don't), and overall help us improve it.
The survey will be open from 5 September to 3 October 2011.
The results will be published at GitSurvey2011.
Thanks a lot for your help!
It may seems that we are relatively quiet at the moment and yes we are, in fact we are like little bees, working hard without making noise.
Maybe you are not aware, but the hosted Indefero offer is running a single Indefero installation for the 3000 forges we are hosting. What is happening is that when you request a page of a forge, the domain is matched against a list and automatically the configuration is set to use the right repositories, the right database, etc.
When the system was setup, it was following the: "Do the simplest thing which can possibly work." Nice, but now we are suffering the pain of growth as the simplest way to go was te build everything in a pretty monolithic system. Ouch.
So, we are hard at work on getting a new system setup to break everything in more manageable components. This is a long haul project, but this is needed to recover our agility in getting new versions online several times a day.
The idea is to do all the changes without changing anything, that is, everything will be done without changing the system for you, then once the migration to components will be completed, new features will be rolled in. Small incremental changes are better on the long run.
This year Fall and Winter are going to be very busy seasons.
This is a great pleasure to welcome Bayer as a new customer. Bayer being big, in this case, I am talking about Bayer Technologies and Services also known as BTS.
Happy new year to all of us. This new year is starting very nicely. First, I think most of us had some good holidays, which is really good for our bodies and minds. Second, William (Delkia on IRC) is a new team member with a lot of nice ideas and good at coding, yeah! He has been fixing issues and adding new stuff at an insane rate. This means that at the moment we are three active developers on the code base. Also, the develop branch is merged into master at least weekly and pushed on the forge hosting farm, this allows us to quickly find the little corner case bugs and fix them very fast.
With 20 commits adding new stuff and fixing issues in this first week, this year starts very well!
So, as the master branch is nicely stable, even so we have quite some cleaning to do in the tracker, a 1.1 release will come soon. It is breaking my policy of not release with open bugs, but at the same time, it has been 8 months without an official release, way too long!
For the business side of the project, I am sorry the cost increased due to VAT increase for the EU users without a VAT number. Also, if your account was due to renewal, I have not yet implemented the code for the renewal handling. The first customer started in September or October 2009, so they have a free ride at the moment :D The stream of new customers is stable and slightly increasing, so, this is also good news for all of us. We will check what the renewal rate gives, I must say, I have strictly no ideas what it can/should/will be.
For Indefero 2.0, I have finally a robust test bed to perform some reliability and performance tests of MongoDB. I am using 2 servers (bare metal hardware nodes) with 4 VMs over a virtual LAN. I wanted the test to start this WE, so, this evening will be a long one. I am using OpenVZ for the virtualisation and this works very well. I am not too afraid by the performance as I am already using MongoDB for Cheméo, this is more the reliability of the replication which I want to test. Data loss is not a good thing.
Have a happy new year and keep having fun in what you are doing!
The great United Kingdom is broke. Ok, maybe not totally, but this situation is not brilliant. Yes, all Europe and the US are broke too. Anyway, the state has two ways to have a balanced budget: spend less or get more income. Usually, if you are a family and you have financial problems, you first cut on expenses because it is very hard to earn more money in difficult times. For the state, it is easier, they just get a new law signed by the Queen.
All that to say that after the VAT raise from 15% to 17.5% January the 1st this year, 1 year and 4 days later, the VAT is raised again to reach 20%. In two years we are going to reach the 25% of Denmark.
For US people, the VAT or Value Added Tax is a kind of sales tax but without the negative effects of the sales tax. That is, only the last consumer is paying it and practically it has no effects on businesses as they can reclaim the VAT. You can learn more about the VAT on Wikipedia.
Anyway, this means that if you pay for your forge before the 4th of January (until January the 3rd at midnight UK time) you save 2.5% (but only if you need to pay the VAT, of course).
here we are, the nearly weekly report about Indefero's life. If you want your comments to be part of the next report, do not forget to send me an email before next Friday.
The work on the Photon framework (next Pluf to power Indefero 2) is going ahead nicely. It is still a "closed" project as I do not want people to distract me too much. But if you are a hacker who can try to figure out things alone, then just drop me an email, I will give you access to the source code. At the moment, the code is clean, small and well structured. Basically I am putting all my knowledge of the past 10 years of web development into it. A small example, all the cookies are signed, that is, you can trust the value of the cookies you get from the clients as the unsigned or wrongly signed cookies are automatically discarded. Of course the cookies are automatically signed. That way, session data can use a cookie storage for zero database interaction.
The framework will stay very lean and will use a library of high quality components, the Particles. The goal is to get many frameworks to use the Particles. By doing a clear cut between the framework and the components (form validation, data storage, etc.) one can keep the framework as simple as possible and as fast as possible.
The domains to host these two projects are:
photon-project.com particles-project.com
These domains are not yet active. I need to finish enough of Photon to get them up and running. Oh, and yes, it will be easy to install Indefero 2, very easy.
Another small thing, next week two new servers (12GB RAM, 1Gbps connection, 3TB storage, Intel Xeon i7 W3520) will be setup to test drive the new Indefero together with Photon and Particles (they have been ordered Thursday). And yes, I am also going to get some RIPE ips for the projects. Next year is going to mark the end of the available IPv4, so time to lock 8 or 16 to be sure I can keep the quality of service for another 2 years. I try to not waste public good (IPs are basically a public good now), at the moment I am using 9 IPs coming from my different providers (not counting the dynamic IPs), but some of the IPs could be saved as the new servers are coming with their own private LAN for inter server communication. This means that some of the IPs I am using for VMs doing "system work" could now run on the private LAN. This would reduce the number of public IPs.
So things are moving, maybe not as fast as I wish, but fast enough to have a lot of fun. I hope the next weekly report will have more details.
On more thing. The hosting of Indefero had a 99.95% uptime this month. The 0.05% downtime was due to my provider network, not the Indefero servers which have had a 100% uptime. This also means that for some locations in the world, you enjoyed a 100% uptime this month and 99.95% in the worst case.
Have a nice WE and happy hacking...
Hello,
here we go, my todo list manager tells me that I need to do my every 10 day report about Indefero.
You can see the milestone tag here with the issues left to fix before the 1.1 release (still a few more to mark) and the ones already fixed.
So, things are going ahead smoothly, maybe not as fast as expected, but ahead.
For the moment, not a single line of code was written for Indefero 2 at the exception of the work on the Photon framework (next generation Pluf framework currently used by Indefero 1.x).
You know that Indefero is used to make money too. That is, not a lot, but it is growing nicely. In October I got 9 new customers and 6 in November. At the moment, the forge has 78 paying customers. This means that from an infrastructure point of view, it is making money.
I still need to finish some work for a company using Indefero, some nice addition to the git view of the source code. It has been long overdue, so, I need to move ahead with it. This is going to bring some money to sponsor some of the work for Indefero 2.
As a company, Céondo Ltd has at the moment a nice contract going through the paper work steps (legal dpt in big company) which is bringing 6 to 8 months of cashflow for 2 months of work (4h/day). And another one in the pipeline for April/May for normally the same kind of amount. So, basically, the company is secured until end of 2011. With the growth of Indefero and Cheméo, this also means that I am going to see a slow shift between consulting and product, with more product and less consulting. At the moment, product (Cheméo + Indefero) is less than 10% of the turnover. The goal is to achieve 50% of my turnover with my products by the end of 2011 (calculated on the 2010 turnover). Basically, getting around 3 to 4000€/month with the products. Indefero 2 is going to be critical in achieving this.
I hope these news are of interest for you, do not hesitate to tell me what you would like to read in the next report and I will try to satisfy you.
If I tell you that on my website I have a download link which can be either:
The link is like on the screen shot below:
What is your answer? I must say, I would have put:
So, just for the fun, I ran the test on the download page of Indefero. After a week, the results were not really like expected. So I let the test run until my confidence percentage was stable. Here are the results:
Yes, bold and italic is the converter, and this by a large margin with 9% better, but what surprised me is that the the bold link is not statistically significantly better than the normal link!
So, my best judgment was basically wrong. What a blow, especially for something as simple as the font style off a link. This small experiment as changed a lot my way to think about improving my software. For my scientific work, I always use data, for the design I often trust my feelings. I was wrong, terribly wrong.
Now, the problem is that I cannot test everything because I am not google and I do not have thousands of visitors a day. But at least, I can test the key points in my application, that is, where actions and conversions are performed.
Just a short notice. Our provider is not really happy with the stability of the routing. Basically, they try to get better BGP performance (I have no idea what it really means) and this requires some hardware changes and some experimentations. They are pretty good at doing that within minimum impact, but still, you may notice some downtime here and there. The downtime may not be full as this can come from partial announce of the routes.
Also, they are working on the UPS system because of minor warnings, the UPSes are complaining about fan problems even so the fans are running normally. The maintenance team of the UPS provider is on site to take care of the problem.
I really appreciate how open OVH is when dealing with their internal issues (their bug tracker is simply open to everyone), this is why Indefero is hosted with them.
Update: Everything is under control, no rotuing issues anymore and the UPS are under stricter control for a while.