Reviewing the Data in Cheméo

... the way a referee reads a paper

In the 2000s, at a Fluid Phase Equilibria conference near Salt Lake City, I was discussing the ThermoML project with someone from NIST. One remark has stayed with me since: even the best property databases carry about 5% of errors. This was about the content, a database takes its values from papers, handbooks and compilations, wrong ones included.

If the figure holds for Cheméo, about 5,000 of its 102,000 compounds carry something wrong. And a wrong value in a database ends up in somebody's simulation or paper. So, over the past months, we have been doing a lot of checking.

Checking the copy

For years, the validation at Cheméo answered one question: did we read the source correctly? Value, unit and compound must be the ones of the original document. We still do this, and we found errors of our own doing it (more on that below).

But if a source gives hydrogen chloride a Gibbs energy of formation of +95.3 kJ/mol, you can transcribe it perfectly, the value is still wrong. So now we also review the data itself, the way a referee reads a paper. Does the value agree with the other values of the same source, with the other sources, with its neighbours in the homologous series? Is it even physically possible? When something does not fit, we go back to the original document and to independent data (IUPAC evaluations, ATcT, the primary literature), and we decide.

How the checks work

  • The structure must agree with itself. The formula computed from the structure must be the stated formula, the molar mass must match within 0.05 g/mol, and one structure or one CAS number means one compound. This runs on every build and blocks the release on any failure without a recorded explanation.
  • Every doubtful case goes back to its source. We first let automatic rules propose the fixes, then checked samples of their proposals: 30% to 65% of them were wrong (one rule would have stripped the deuterium of 172 labelled compounds). So we opened all the 5,662 cases by hand, at the source.
  • Single values are cross-checked against the source's own other values, against other sources and against simple order checks: a compound melts before it boils, and boils below its critical temperature.
  • Every build is compared row by row with production. Nothing moves or disappears without a recorded reason.

The first run of these checks on the previous database version did not look good: 2,347 groups of compounds sharing one structure, with 3,091 extra ids, and 716 compounds whose formula disagreed with their own structure.

A small gallery of errors

Most errors in the sources are small and specific. Each one below was established by going back to the source and comparing it with independent data.

  • A wrong sign. KDB gives hydrogen chloride a Gibbs energy of formation of +95.3 kJ/mol. ChemSep gives -95.3 kJ/mol, and KDB's own enthalpy of formation, -92.36 kJ/mol, agrees with ATcT.
  • A value copied from the neighbour. KDB gives dibromomethane the enthalpy and Gibbs energy of formation of 1,2-dibromoethane, listed a few lines away in the same source.
  • A value in the wrong field. KDB gives 1-chlorooctadecane a normal boiling point of 292.65 K, which is its freezing point. It boils near 621 K.
  • A study overturned by later work. The NIST WebBook and ChemSep give propylene carbonate an enthalpy of vaporization of 30.7 kJ/mol, from a 1971 study. Everything measured since gives about 61 kJ/mol, twice as much.
  • A value inconsistent with the source itself. ChemSep's Gibbs energy of formation of methyl hexanoate, -357.0 kJ/mol, does not follow from ChemSep's own enthalpy of formation and entropy, which give about -302 kJ/mol.
  • A structure of another substance. The WebBook's structure file for ammonium sulfate draws a covalent N-O-S-O-N molecule, while 43 literature records describe the salt.

In total, we now leave out 36 values: 13 from ChemSep, 10 from KDB, 10 from the NIST WebBook and 3 from open datasets. For these properties, Cheméo shows the values of the other sources.

I use these sources every day and Cheméo would not exist without them. 36 values is a tiny fraction of what they provide.

The mirror in the NIST WebBook

Take the WebBook page of L-alanine. The page shows an InChI and an InChIKey, the standard identifiers of the compound. The same page also offers a structure file to download. Compute the InChI of that structure file and you get L-alanine. The InChI shown on the page is the one of D-alanine, its mirror image.

We found the same mirror on 8,718 WebBook pages of chiral compounds. The structure file is usually the right side, about four times in five, but not always: for D-glucose, the page is right and the structure file draws L-glucose.

Cheméo had taken its identifiers from the pages, so data measured on L-amino acids landed on D compounds, and the reverse. We even served a "D-Alanine + Water" mixture whose measurements are all of L-alanine.

So each compound is now decided on evidence: its name when it states the configuration, PubChem looked up by CAS number, the natural configuration of steroids and sugars. What the evidence cannot decide is checked by hand. I also sent a short report to the NIST WebBook team, with a three-step reproduction for L-alanine anyone can repeat, so that the identifiers get fixed for everybody who relies on them.

Errors of our own

The liquid density of cyclohexane in Cheméo was 13% too high near the critical point. The source file, from ThermoML (by the way, this was a revolutionary idea at that time, because back then, all databases were very expensive to access), was fine: it gives these densities at a constant 5,000 kPa, and our parser dropped that condition. Looking at that part of the code, we found two more defects. First, only the first dataset of each file was kept, and since a refactoring in 2023, this lost 290,540 data rows. Second, heat capacities at constant volume were filed as heat capacities at constant pressure. All three are fixed for the next version, which brings back about 263,000 rows.

Our old pipeline also reversed the double-bond geometry in 464 InChIs, this is fixed too.

Measuring the error rate

The checks above only find the kind of errors we already know about. For the others, we draw compounds at random and check each one by hand at its sources.

On the database before the current version, 7 of 196 random compounds had an error, 3.6% (between 1.7% and 7.2% at 95% confidence). On the current version, v28, released on 9 November, the probe found 6 errors in 49 compounds, and 4 of the 6 were the WebBook mirror. Before that probe, I did not know the mirror existed.

So, at the moment, Cheméo sits right around the 5% of my NIST memory. The target is about 3% or less, as the upper bound at 95% confidence, and we are not there yet.

Every correction is public

Every decision about a compound is recorded with its reason and the sources opened to reach it. Compound ids never change meaning, and none is reused. The 1,234 duplicate ids now redirect to the compound they duplicated, with its data, names and mixtures.

The corrected source values, the changed structures and the retired ids are listed on the errata page, with the reason for each. If you cited a Cheméo value that appears there, you will know why it changed.

Next

Version 29 brings the WebBook stereo corrections, the recovered ThermoML data and a new random probe. I am curious to see where it lands compared to the 5%.

Fluid Phase Equilibria, Chemical Properties & Databases
Back to Top