Céondo's Blog - Embrace Constraints To Evolve Selection of posts with tag InDefero. www.ceondo.com/ecte/feed/ End of Life of Indefero Hosting www.ceondo.com/ecte/2013/07/end-of-life-indefero-hosting 2013-07-05 12:56:39 GMT

Reminder, the end of life of Indefero hosting was reached on the first of July 2013. You had 1 year to adapt, so I hope you took the time to effectively adapt. The hosting will be effectively shut down the 15th of July 2013. If you need a PostgreSQL backup of your forge data on top of the JSON backup you get from your account, please contact me as soon as possible.

Indefero downtime, routing issue www.ceondo.com/ecte/2013/05/indefero-routing-issue 2013-05-15 12:03:30 GMT

Wed May 15 11:53:09 UTC 2013: Hello, we have a routing issue between the frontend and the backend database of Indefero. The database is safe, nothing to worry, this is just that the frontend cannot connect to the database server like if suddenly a firewall was cutting the connection. We are investigating.

Wed May 15 13:54:26 CEST 2013: Our provider is having an issue, we keep an eye on it.

Wed May 15 13:55:39 CEST 2013: Ping between the machines went from 5 ms to back under 1 ms, it looks like our provider is doing something.

Wed May 15 14:02:45 CEST 2013: Our provider changed some configuration of their routers and they are now observing the situation, Indefero is back online.

Reminder: Indefero Hosting End of Life June 30, 2013 www.ceondo.com/ecte/2013/05/indefero-end-of-life-hosting 2013-05-06 08:04:13 GMT

Reminder: The Indefero hosting will stop June 30 this year, yes, 2013. You were informed about a year ago, so I hope you had time to migrate.

If you are a Subversion user, Assembla can import your dump of your repositories. The daily Subversion dumps are available in your account area. They also offer Git hosting. If you are using Git only, you have 1000's of offers.

Of course you can install your own version of Indefero and go ahead with a self hosted solution as Indefero is a free software.

Update of the PostgreSQL server for Indefero www.ceondo.com/ecte/2013/04/indefero-postgresql-update 2013-04-11 07:34:41 GMT

Thu Apr 11 07:27:29 UTC 2013: There is a pretty serious update of PostgreSQL to be done for security reasons. Indefero will be done for a couple of minutes the time to update the server. I am sorry this is an unplanned update.

Thu Apr 11 07:30:54 UTC 2013: The slave has been updated and is correctly picking up the updates from master, now going to update the master. From the slave update sequence you can expect about 2 minutes downtime of Indefero.

Thu Apr 11 07:33:36 UTC 2013: The master has been updated, the effective downtime was less than 20 seconds.

Electrical failure, Indefero down www.ceondo.com/ecte/2013/03/indefero-electrical-down 2013-03-06 16:23:25 GMT

Wed Mar 6 16:21:45 UTC 2013, we have an electrical issue, for the moment Indefero is down, it will come up as soon as the electrical issue is resolved. More details will soon be available from our provider.

Indefero down at the moment www.ceondo.com/ecte/2013/01/indefero-down-rbx4-issues 2013-01-21 14:35:52 GMT

Mon Jan 21 14:22:30 UTC 2013 A large array of servers are down in our datacenter. Saddly, the main database server of Indefero is down too. The backup server is running ok and is up-to-date. We are expecting feedback from OVH to assess if we need to switch over the slave or if we can expect the main server to come back fast enough.

Mon Jan 21 14:24:25 UTC 2013 It was fast, power supply issue for RBX4. Still waiting for more information.

Mon Jan 21 14:35:20 UTC 2013 Everything is up again, sorry for the inconvenience.

Servers Down at the Moment www.ceondo.com/ecte/2012/10/server-down-vlan-issue 2012-10-15 10:47:19 GMT

Our provider has some issues. Not so fun, the servers are down and I hope they will come up again as soon as possible. Basically, we have a split on the internal network, some servers can talk to each other, some cannot. Very annoying because it makes split brain situation. All the data are safe, but we have to wait until this is solved (and they are doing it manually).

Update: Everything up again, down-time of approximatively 30 minutes, I hope it was at the time of your coffee break.

Indefero Hosting Will Keep Going www.ceondo.com/ecte/2012/09/indefero-hosting-keeps-going 2012-09-13 10:40:59 GMT

Very good news, after a lot of discussion (because I want to be sure of the quality of the offer) the Indefero hosting offer will continue as it will be taken over by a small companies already used to work with hosting data and user information under strict security rules. The transition will be totally transparent for the current customers and users. I will keep you informed.

Indefero Hosting Stopping Dec 2012 to June 2013 www.ceondo.com/ecte/2012/08/indefero-hosting-stop 2012-08-02 10:02:59 GMT

Front matters: This is the email I sent to all the current users of the hosting offer.

Dear Indefero Users and Customers,

today is not an easy email I am sending you, this email is to announce you the wind down of the Indefero hosting platform. The Indefero hosting will be stopped at the following dates:

  • free hosting, 31th of December 2012 (in 5 months);
  • paid hosting, 30th of June 2013 (in 11 months). If your renewal date ends before the date, you will get free hosting until the end.

To help you in the changes, the period of transition is as long as possible. Now that you know the meat of the subject, let me provide you with the whys, how and details. But first, stopping a service is not an easy task, it is especially hard because you, as customers, trusted me to provide long term high quality service and by stopping the service I am breaking this trust. For me, it is also hard because it means that I failed to correctly predict the future.

Thanks a lot for the trust you had in using Indefero and please accept my sincere apologizes for not providing you continued services for another couple of years.

If you paid a renewal or a new forge in the past 45 days, I can issue you a full refund. In this case I will ask you to migrate out before the end of the year. 45 days is the limit of the banking system.

How to Save your Data

Simple, login here:

https://www.indefero.net/account/

and download the backups (down the page). You get everything related to your forge and the data are compatible with Indefero, that is, you can install Indefero on your server and import the data.

How to Make a Successful Migration

Login in your account:

https://www.indefero.net/account/

Click on the "configure your forge" link:

https://www.indefero.net/account/bp/

and update the personal domain to use a domain you fully control. If you are working for the foobar.com company, put something like code.foobar.com and get a CNAME record in your DNS pointing code.foobar.com to yourforge.indefero.net. Then, start asking people to use code.foobar.com to access your forge. After a while, nobody will use the indefero.net address and you will have full control over your forge.

The next step is to setup your own Indefero instance and import the data from the hosted forge, then switch the DNS to point to your own Indefero instance.

The end result is a migration without downtime and without disturbing your end users.

Tools to Make a Successful Migration

In October, you will get ready to use Amazon EC2 images which will allow you to do nearly "One Click" migration of your data from the hosted platform to your own Indefero instance. With an EC2 micro instance, it will cost you about $15 per month to run your own Indefero instance. I will also work with the current providers of Debian packages to be sure you can easily setup and import your Indefero forge on a fresh Debian system.

Why Stopping Indefero Hosting?

Because of focus, when I started Céondo Ltd, I had not really a clear picture of where I wanted to go and how, now, I know and the key is "Science", that is, I will fully focus on scientific software and consulting. In the last months, I was able secure 2 to 3 years of consulting pipeline in science, this is a clear indication that this is the way to go, a specialization in an extremely technical area where the barrier to entry is very high.

How Will It Affect the Indefero Software?

Surprisingly, I expect it to be positively affected. The last year I have been slowing down my involvement in the software because changing the software would also mean for me, applying the changes to 3000+ forges on a system not designed in the first place to accomodate so many forges. I was afraid of the consequence of a bad upgrade at such scale.

The time used to manage the hosting will in part be redirected to improve the software and the migration tools will also be used in parallel to allow us to perform automated testing of Indefero. We will be able to start an EC2 micro instance, test and stop.

The Indefero community is very active with an increasing number of users and packages for nearly all the current Linux distributions, the current goal is to have the Indefero packages distributed officially by all the main Linux distributions to ensure long term support. You will soon get the distribution specific packaging scripts be part of the source code of Indefero. This is critical for the long term support of Indefero and will help testing on a larger scale.

Alternatives to Running your Own Forge

The code hosting space is crowded, so crowded that it is hard to recommend someone. First, the real question is:

  • how critical is your code? Can you accept to have it hosted by a third party? Of course you currently have it hosted by a third party right now, but it may be a good time to rethink this question. I think it is critical enough to have full control over it, this is why I gave full control with CNAME, backup and OSS dump compatibility to you when providing Indefero hosting. I could not provide something I could not use personally.

  • then of course, you need to define what you want in terms of functionalities, version management software (Git, Subversion, Mercurial, etc.) and the contractual constraints (hosting location within/outside the US, price, owned by a big/small company independent/partially owned by venture capitalists). This is not simple, I have seen an increase of the number of forge creations since GitHub took venture capital money on board. So, it looks like some of you do not like to be dependent on venture capitalist controlled companies. You have the time to think about it.

If I had only one service to recommend, I would recommend Pikacode under the lead of Benjamin Jordan. They have been hosting code repositories for a long time and have been active contributors to Indefero. I trust them and they are real system administrators, used to managed some of the biggest website in France, saturating Gbps of bandwith during big events. They know their job very well.

Here is the "inline" advertizing from Benjamin for you:

Pikacode.com offers Git and Mercurial repositories hosting. Formerly knows as Intuxication, thousands of repositories have been created by our users since 2008. Pikacode goals are simple : easy, sleek and fast code hosting. We offer you 90 days of free trial for unlimited private repositories and collaborators with the following voucher : HELLO-PIKA.

http://pikacode.com/

If some of you can recommend code hosting companies, just let me know. Note that Pikacode also offer free public repositories.

What is Next?

I will setup in September/October a migration website:

http://www.indefero.net/migration/

this will be your portal to have everything you need to get a successful migration without disturbing your users and losing your data. It will be updated with the latest information, tutorials, possible alternative offers — basically, everything to help you.

Again Thank You

At the end, I can only thank you for your trust and the bit of travel around the Sun we did together. I am proud of what was achieved with Indefero and I am honored you trusted me, I am also sure you will find a good way forward.

Best regards, loïc

Update of Indefero SSL Certificate www.ceondo.com/ecte/2012/07/indefero-ssl-renew 2012-07-31 07:41:15 GMT

The [Indefero|http://indefero.net] SSL certificate expires today, it will be renewed during the afternoon, normally the update is performed without downtime. If downtime, it should a couple of seconds the time to restart the web server.

Update 2012.07.30 11:30 UTC: The renewal procedure is on the way, it should be finished in a couple of hours to have the real update at the server level done just afterwards.

Update 2012.07.31 07:40 UTC: The certificates are now updated and valid for another year.

Creation of New Forges Suspended While Updating the Website www.ceondo.com/ecte/2012/06/creation-suspended 2012-06-25 12:12:50 GMT

Please accept our sincere apologizes but the creation of new Indefero forges will be suspended this week while the website will be updated. The website will be updated by the end of the week (30th of June 2012) but the exact dates when the website will be updated is not yet known. If you want to create a forge, please try doing it by Wednesday the 27th as the update will most likely take place Thursday/Friday.

Indefero Database Failover www.ceondo.com/ecte/2012/05/database-failover 2012-05-25 11:54:56 GMT

I am sorry for the unscheduled downtime this morning, Friday 25th of May around 8 am UTC. A scheduled kernel upgrade of the server went not as expected. The kernel upgrade did went correctly on the slave, including reboot and resync, but the master failed to come up again. For data safety reasons, we performed a backup of the slave before promoting it as master and switching the application to use the new master. This backup is what took a bit more time than expected and resulted in the large downtime.

What is next?

  • Need to promote a new slave and get the master to logship to it, this will again force a small downtime of the master database the time to pickup the new configuration.
  • Need to be a bit more proactive in announcing the issues. I setup a small list for the very active users, but a know central place to publish updates would be better.
  • Need a regular backup of the slave to not have to perform it under pressure.

I will keep you informed. Sorry for the annoyance, sometimes issues happen and this one took me by surprise.

Update: Reviewing the logs, the combination of a VM + Hardware node restart, including KVM upgrade is most likely the culprit.

They use Indefero: Elveos www.ceondo.com/ecte/2012/01/they-use-indefero-elveos 2012-01-31 13:17:10 GMT

A small notice to inform you that Elveos.org is using Indefero:

Elveos.org is a crowdfunding website for open source software. You are a free software developer? Elveos gives you a way to get paid for your work. You are a free software user? Elveos let you fund the features you need.

It reminds me KickStarter but more open. It is very nice to see offers targeting the OSS community.

A router crashed, the websites have been slowing down www.ceondo.com/ecte/2012/01/router-crashed-card 2012-01-27 12:05:57 GMT

If you noticed a slow down in the past minutes, one of the routers of our provider had some issues. This slowed down the services for a short period of time. As you can see on the following graph, suddenly our GET requests to monitor the response time of the services went bong. 20 second response time, this is the equivalent of dead...

Router down

But I must say, this is where I am really pleased by our provider, OVH. They immediately explained what was going on: a card of the router crashed hard or died.

Migration of Indefero's Backup Server www.ceondo.com/ecte/2012/01/indefero-backup-server-migration 2012-01-13 17:03:04 GMT

For your information, we are in the process of migrating Indefero's main backup server on our new infrastructure. The new infrastructure has been running for a while and we are satisfied with the stability.

We are going to do at the same time a server upgrade, moving away from Ubuntu and back to the roots, that is Debian. Once the backup server will be up and running smoothly, the main server will follow.

Update: Got a bit of instability at the same time... upgrading here and there an old server is difficult. Time to get the migration to a better system completed!

Launching Cheméo's Labs www.ceondo.com/ecte/2011/12/labs-chemeo 2011-12-21 08:21:50 GMT

Few days ago Cheméo's laboratories went life. The labs are running software experiments in the field of chemical and physical properties. They are kind of sandboxes where ideas can be tried without disturbing the main Cheméo website.

The labs are running on top of Céondo's private Platform as a Service (PaaS). This platform will soon host all the services we deliver, from our products Cheméo and Indefero to simpler websites like ceondo.com. In case of, a status website will be kept independently using another technology with a different provider. I will soon write a bit more about this private PaaS.

These are exciting times, the best to close 2011 and start 2012.

Improving the Response Time of Indefero www.ceondo.com/ecte/2011/12/indefero-performance 2011-12-02 17:35:14 GMT

Improving the speed of Indefero is challenging as it requires managing a lot of moving parts, from the git/subversion backends to the database. This week, I have been working on setting up Graphite for the infrastructure. This is working pretty well and provides graph like the following one.

Current response time of Indefero

This graph is extracted from a special Nginx log format which includes the time needed for Nginx to send the response back to the client. The only thing missing is that when I see a spike, I need a way to directly access the corresponding logs to figure out why. At the moment, there are no integrations between these metrics and the logs.

To improve a system, one needs to know the current state. Graphite is a bit hard to setup, but afterwards, it is really easy to push data in. A really nice tool.

The Problem with Performance Logging www.ceondo.com/ecte/2011/11/performance-logging-debugging 2011-11-30 08:32:02 GMT

To run a service like Indefero, you need to log a long list of metrics to follow the load on the system, find the bottlenecks and predict the future needed capacity. To do that, a very powerful system is Graphite, the only issue is that it is only storing and graphing numerical values. Of course, you cannot do different, but the problem is: correlation.

Basically: Once I see that every now and then component is not performing well, how can I drill down in my data to find the reason?

Graphite tells you: this day from 14:05 to 14:07, the rendering of a git tree view was slow. Good to know, the following question is of course: why? If you store more metrics, you can maybe find that I/O was slow on the server X, you can graph together many metrics and visually correlate them. But then, why was I/O slow?

At this point, you need to go one level deeper and take a look at the logs coming from server X from 14:05 to 14:07. This can bring you up to the application level where you figure out that a client repeatedly accessed a page which triggered a git command with a large output, thus loading the server. But to do that you need to access the logs too.

So, Graphite is wonderful, but what I need is that after identifying the subsystem and time range where we have an issue, being able to simply scan through all the corresponding logs in the time range. This would be a kind of integration between Graphite and Graylog2.

My problem now is that Graylog2 is overkill. That is, it tries to provide full text search on the logs, the result is that it requires a very big machinery where I just need aggregation of the logs and the equivalent of a time base search range with a filtering by component, for example webapp.backend.git.

This annoys me, I do not want to build a system by myself.

Small downtime of Indefero www.ceondo.com/ecte/2011/11/short-downtime-8-minutes 2011-11-16 12:06:14 GMT

Around 11:14 UTC today one switch of the private network went down and required a reboot. The problem has been solved but this resulted in a downtime of about 8 minutes.

Note that this is the issue with the new database server, if the link between the application server and the database server goes down, then the service is down. I will contact the support staff of OVH as in my understanding, they had a kind of redundant system to not rely on a single router.

Database Migration Starting & Done www.ceondo.com/ecte/2011/11/database-migration 2011-11-11 11:34:56 GMT

Hello, just to let you know that today November 11, the database migration is starting. You can check this blog post for updates. Here are the steps I will be performing:

  1. test of the web application software upgrade. 09:55 UTC - Done.
  2. shutdown of the PHP processes to prevent update of the database. 09:56 UTC - Done.
  3. force the creation of a PostgreSQL WAL to have consistent backup. 09:57 UTC - Done. Now waiting for the log to be shipped to the warm standby. 10:00 UTC - Done.
  4. start of the warm standby as a new master. 10:04 UTC - Done.
  5. check that the new master is consistent. 10:21 UTC - Done.
  6. update the web application configuration to access the new master. 10:23 UTC - Done.
  7. start the PHP processes again. 10:25 UTC - Done.
  8. immediately clone the new master to have it log ship on another warm standby. 10:37 UTC rsync in progress
  9. 10:52 UTC - Done, with a warm standby in another datacenter nicely getting the updates from the master.

This is a bit of cascading but it will always keep several version of the database running and it will always be possible to revert to the original DB server in case of problem.

Last update: The system is now insanely more responsive, pleasure to use is back! If you notice anything unusual, please let me know as soon as possible.