So, as regular readers know, we have declared war on PAINS on the blog. As part of that effort, I (we?, not sure if Dan wants to be associated directly with it) introduced PAINS Shaming. Well, thanks to Angelo Pugliese and Duncan McArthur at the Beatson we have the latest paper to shame.
The nice thing is that they come right out and call it like it is: a Rhodanine. To reiterate, from the comment by Baell and Walters:
Rhodanines exemplify the extent of the problem. A literature search reveals 2,132 rhodanines reported as having biological activity in 410 papers, from some 290 organizations of which only 24 are commercial companies. The academic publications generally paint rhodanines as promising for therapeutic development. In a rare example of good practice, one of these publications (by the drug company Bristol-Myers Squibb) warns researchers that these types of compound undergo light-induced reactions that irreversibly modify proteins. It is hard to imagine how such a mechanism could be optimized to produce a drug or tool. Yet this paper is almost never cited by publications that assume that rhodanines are behaving in a drug-like manner.
And as predicted, this paper does not cite Voss et al. They cite dose-depedent responses for their compounds. Does it matter? Not to me. To their credit, they call these molecules tools, but also tout them for future therapeutic development. A PAIN can be a useful tool or even lead to non-PAIN containing compounds, but it requires a higher level of proof. I don't see that here.
So, here is your PAINS Shaming (Holiday themed):
Problem I guess is the reluctant of biochemist, biologist and HTS groups to show the structures to a medicinal chemist before sending the manuscript out.
ReplyDeleteand computational chemists ;)
ReplyDeleteIt's a multi-factorial problem, methinks.
ReplyDeleteIn a rare role reversal, I'm going to take an even harsher stance than Teddy. These molecules are reminiscent of the fictional compound hitinane I introduced as an April Fools' joke. Ironically, one of the commentators on that post suggested that it should have been published in Chem. Biol., and here we are.
ReplyDeleteFurthermore, I think it is important to emphasize that PAINS do not make good tool compounds. If anything, tools should be more selective than drugs because they are used to analyze pathways and understand biology. It is hard to make sense of anything when your "tool" hits every other protein in the cell.
What's really embarrassing is that the authors do not cite any of the significant literature on rhodanines, nor do they even consider whether these are covalent inhibitors.
This paper was published online a few weeks before the high profile Baell & Walters Comment in Nature. Hopefully we will see less of this sort of pollution of the literature in the future. At the very least, more people should know to disregard it.
What's next? Dogs and Cats living together?
ReplyDeleteAnd in Chemistry & Biology too! What a surprise!
ReplyDeleteAs the editors (and even some of the readers) of Practical Fragments will be aware, I agree that errors in scientific studies should be highlighted. Nevertheless, naming and shaming in this context places the a burden of proof on you to demonstrate the guilt of the specific compounds described in this study. Will the Bad Cop accept the challenge of the (reluctant) Public Defender?
ReplyDeletePete,
ReplyDeleteI think the weight of literature precedent falls on the authors to prove that these PAINS are in fact well behaved. Their lack of citations to PAINS literature indicates (to this reader) that they did not do a sufficient literature search for their compounds and thus are not even aware of the potential liabilities.
In court, it's not what the Bad Cop thinks but what the judge and jury think. Will the prosecution present some evidence or can the defense move to have the charges dropped?
ReplyDeleteIn a US criminal court, the standard is "innocent until proven guilty." This is because people usually don't indict themselves; they are charged with a crime and try to disprove it.
ReplyDeleteScience is different.
In science, people make a claim and provide evidence for it. If that evidence is weak, or if the researchers ignore alternative explanations, it is incumbent upon the scientific community to respond. Ideally this happens before a paper is published, but peer review, like any institution, is fallible.
In the current case, none of the literature on PAINS was cited, nor were any experiments described to show that the compound behaves specifically. It is formally possible that this compound differs from the numerous close analogs that behave pathologically, but the researchers provide no evidence for this. As Carl Sagan was fond of saying, "extraordinary claims require extraordinary evidence."
This sort of thing matters because chemical suppliers will no doubt start selling the compound, and, unless warned, other researchers will use it to probe biology, thereby leading to further pollution of the literature. As an example, the "chemical probe" SJ-172550 is now sold by over a dozen vendors, several of which cite the original paper but not the excellent follow-up paper by the authors in which they state that its "mechanism complicates the interpretation of experiments."
Pete, you seem to be arguing that, until a weak hypothesis is disproved, it should be accepted. This low bar provides an excellent opportunity for ignorant or unethical careerists to pad their publication record at taxpayer expense. Perhaps I'm an idealist, but I believe we should all carefully consider alternative explanations. In the case of compounds with substructures previously shown to be pathological, this means assuming any hit is guilty of being an artifact until proven innocent.
Hi Dan, the issue here is not so much about the acceptability of rhodanine derivatives and when I presented the challenge to Teddy, I described the public defender as reluctant. For the record, if those were the only hits from an HTS campaign, I would be joining the queue to put the project out of its misery as swiftly and mercifully as possible. I also believe that it would be useful to share concerns about particularly chemotypes even when the information is relatively soft. An interesting challenge for the cheminformaticians, perhaps?
ReplyDeleteHowever, I believe that if you’re going to trash somebody’s work in a blog post then you need to be more explicit than Teddy has been about what is wrong with the work. The other thing that you need to be concerned about when bashing PAINS is that some will be more PAIN-ful than others and there will be less certainty about the unwholesomeness of specific chemotypes. This will become more of an issue if Practical Fragments decides to fire a few salvoes at targets further down the PAIN-chain. For example, should one be just as apologetic about using catechols as probes in NMR experiments as reporting rhodanines to be screening hits?
But let’s get back to rhodanines because there’s a point that I’d like to develop a bit further. First, I need to ask a question and we can return to the discussion once you’ve responded. Do you and Teddy believe that there are any circumstances under which biochemical assay results for a rhodanine would be admissible?
Do I "believe that there are any circumstances under which biochemical assay results for a rhodanine would be admissible?" I'm afraid I can't answer that without asking, admissible for what? As we state often on this blog, biochemical results in general are just a first step, subject to confirmation by orthogonal assays, biophysics, cell biology, and, ultimately, animal studies.
ReplyDeleteIf your question is whether I believe that a rhodanine will inhibit a protein in a biochemical assay, I'll respond absolutely yes. The problem is that it will inhibit lots of proteins. So will iodoacetamide. The difference is that even an idiot won't call iodoacetamide a chemical probe, whereas sadly the same can't be said for rhodanines.
This paper differs in two important respects from the NMR paper we recently highlighted. First, in that case, the authors acknowledged the potential for catechols to be problematic, and ran appropriate controls. More importantly though, the authors were not suggesting that their catechols would be useful in general; they simply used them to validate their technique.
The current paper, however, goes far beyond reporting rhodanines as screening hits. In fact, the last line of the Significance section states:
"We present... a class of small molecule inhibitors of SOS1 useful as both biological tools and for future therapeutic development."
This is a dangerous and irresponsible statement.
These are not useful tools, as the authors show only minimal evidence for selectivity, and the chemical class has been shown repeatedly to be non-selective and to act through multiple pathological mechanisms, a fact that the authors neglect to mention.
This is an important biological target, and it is unfortunate that people will likely be misled once they start using this "tool".
If the authors had run sensible follow ups all of this could be avoided.
ReplyDelete1. Show formal pseudo 1st order kinetics
2. Change reducing strength of the buffer system
3. Show formal competition
4. Show selectivity against an unrelated target
These are not difficult experiments, which take maybe a couple of weeks, many labs have a 'standard' unrelated target which hits can be run against (GST/Calpain etc...). As chemistry journals now regularly require a higher level of proof to determine whether claims of compounds synthesized are true, biology should follow suit.
Hi Dan, I would have been very surprised if Teddy would have launched into a naming-and-shaming rant if the authors of that NMR article had failed to mention the fact that catechols have been described as PAINS. However, that would be a separate discussion and my next question is what about rhodanines with saturated (sp3) carbon in the ring? Do you believe that a rhodanine with a saturated carbon in the ring will inhibit a protein in a biochemical assay?
ReplyDeleteThe question "do you believe that a rhodanine with a saturated carbon in the ring will inhibit a protein in a biochemical assay?" is a bit vague. One might as well ask "do you believe that a heterocycle will inhibit a protein in a biochemical assay?", to which the only correct answer is, "maybe."
ReplyDeleteThe more relevant question is "Do you believe that a rhodanine with a saturated carbon in the ring is a PAINS compound?" I would say in general no, as it no longer contains the Michael acceptor. In fact, I recall a manuscript discussing a series of alkylidene rhodanines in which reduction of the double bond obliterated activity. That's not to say that we should accept such a compound at face value, and the -S-(C=S)- moiety may complicate development, but (always assuming proper controls are run) it could potentially be useful as a chemical probe.
All that said, it is worth noting that the "lead compound" in this paper (NSC-658497) contains not just an alkylidene rhodanine but also another potential Michael acceptor. Indeed, this may be responsible for the activity observed, as reducing this double bond gives an inactive compound (A3).
Hi Dan, I deliberately phrased the question as “Do you believe that a rhodanine with a saturated carbon in the ring will inhibit a protein in a biochemical assay?” because you wrote in your previous comment, If your question is whether I believe that a rhodanine will inhibit a protein in a biochemical assay, I'll respond absolutely yes”. I disagree that the question is “a bit vague” since it is actually more specific (requirement for saturated carbon in the rhodanine ring) than the question to which your response was “absolutely yes”. This is probably a good time to say what PAINS-folk call rhodanines probably shouldn’t be called rhodanines since this implies a saturated carbon in the ring (as is the case for the structural prototype). Instead, we should probably be talking about benzylidenerhodanines (or whatever name is appropriate to the specific substructure that you want to trash) because this makes the presence of the exocyclic double bond absolutely explicit.
ReplyDeleteI took a look in Supplementary Table S5 for the original PAINS paper and found two SLN definitions relevant to the current discussion which are labelled rhodanine ( C=C[1]SC(=S)NC(=O)@1 ) and rhodanines ( C=C[1]SC(=Het)NC(=O)@1 ) and both of these indicate that an exocyclic double bond is necessary for substructural match. The second is a sensible attempt to generalize the first (more specific) substructural query but probably shouldn’t be labelled ‘rhodanines’.
This is probably a good point to say that I would be wary of any screening hits that had an electron-withdrawing group (carbonyl, cyano, sulfonyl…) bonded to unsaturated carbon. I would certainly be wary of screening hits matching C=C[1]SC(=S)NC(=O)@1 but also matching C=C[1]SC(=O)NC(=O)@1 , C=C[1]OC(=S)NC(=O)@1 , C=C[1]OC(=O)NC(=O)@1… That said, forming covalent bonds with protein targets (e.g. to cysteine thiol) is a perfectly acceptable molecular design tactic even if it gets some of the traditionalists spitting feathers.
From the early days of HTS people in Pharma recognized that screens were picking up some funky looking structures. One of the problems we faced from the start was not so much recognizing that the structures looked funky but more demonstrating objectively what was wrong with the compounds (and in some cases their structural classes). Secondly, problem compounds didn’t usually (on the projects I worked on) reveal themselves in as frequent hitters. For example, compounds showing as hits in a small number of assays rang bells when they were observed as hits in a cysteine protease assay and a PTP assay.
The reason for my challenge to Teddy’s post is that it is not clear how strong the evidence actually is that assay results for rhodanines (which we will take to mean rhodanine derivatives in which all ring carbon atoms are unsaturated) can be summarily dismissed. The best way to respond to such a challenge is to present the evidence.
Pete, It is incumbent upon the scientists to prove that their molecules are good. It is my job to point out when they do not do it, to my satisfaction. Whether or not, it meets your criteria, that's another story.
ReplyDeleteHi Teddy, Your satisfaction (i.e. opinion) will carry a lot less weight if you are unwilling (or unable) to explain the evidence on which you trash this article. You need to be very careful in the opinion business.
ReplyDeletePete, you are correct that to be precise we should be referring to "alkylidene rhodanines" instead of merely "rhodanines". That said, it is relevant that, of the 117,187 rhodanines reported in SciFinder, 88% of them are alkylidene rhodanines, and of the 60,347 commercial rhodanines, >97% are alkylidene rhodanines. Because these are PAINS, they will be enriched above their proportional representation among screening hits. In other words, when a rhodanine shows up as active, it is overwhelmingly likely to be an alkylidene rhodanine.
ReplyDeleteNow I have two questions for you. Given the issues with this specific paper, do you still "move to have the charges dropped"? And do believe that these compounds should, as the authors propose, be used as tools to probe biology?
Hi Dan, One key question is whether the charges would have been made in the first place if the compounds in question were not ‘ene-rhodanines’ (I suggest this rather clumsy term as a convenient shorthand for ‘rhodanines with an exocyclic carbon-carbon double bond’ and will use it to make the substructural context clear). Were you suggesting (without reference to PAINS and ene-rhodanines) that the authors had not done enough to justify using the compounds as tools then I might be a plea bargain. However, the actual offence for which the article is accused is to have interpreted experiments that made use of alleged PAINS. As such, I call on the prosecution to present their evidence.
ReplyDeleteAs an aside, I should point out that most of my knowledge of US courtroom procedures was gained as a teenager in Trinidad by watching episodes of Owen Marshall, Counsellor at Law and that was some years ago. On a minor technical point, I need to point out that the arresting officer must state exactly what crime was being committed by failing to cite Voss et al (from the perspective of a fellow blogger, this is just plain sloppy and that Practical Fragments can do better). In response to your comment “Because these are PAINS, they will be enriched above their proportional representation among screening hits”, I’ll leap to my feet and say, “Objection. Wild speculation by the State”. (I’m guessing that one addresses the prosecution in the USA as the ‘State’ but Owen Marshall was a long time ago and I’ve since forgotten. In the UK it would be ‘my learned friend’).
I suggest that we start with the original PAINS article and, to simplify matters, I’ll ask whether you believe that there is enough evidence in that publication to secure a conviction. Also feel free to subpoena Teddy.
I'm not too fond of shaming people. In this case, it is indeed a pity that the authors spend their time and energy on a scaffold that is widely seen as intractable. Still...
ReplyDeleteIn one of my former jobs, I happened to see some of those rhodanines derivatives popping up from a virtual screening exercise (as is often the case when you don't filter them out from the start). I was aware of the potential issues, and this was mentioned when discussing results of the VS. But the decision was still to buy some and test them. They popped up in the screening, in their typical low µM range. SAR was flat on the identified derivatives, but the most potent one appeared competitive vs. substrate. Again, there were multiple reasons to stop, including IP. But still I had to move on with those.
So in the end, shaming the authors altogether is an easy process. But what part of the blame should be put on a PhD student that ignores this PAINS story and/or has pressure to publish and/or has to follow directions given by his/her supervisor? What part of the blame should be put on the more experienced supervisor, who should know about those issues, but decides to keep working with those untractable compounds? Then, what is the responsibility of a lab director?
And finally, what part of the blame should be put on the journal reviewers? Wasn't it their mission to question such findings? They should at least have asked the authors to comment on this PAINS issue.