PPI benchmark evaluation

Data integrity

1,815 rows removed during cleaning (2.8%)

536 duplicate pairs (4 reversed), 1,279 self-pairs and 0 pairs with conflicting labels were removed; 64,103 canonical pairs remain.

For model building: Reversed duplicates (A–B and B–A) placed on both sides of a random split leak labels directly. Train on the canonical pair list, not the raw file.

Degree bias in supplied negatives

Endpoint degree SMD 0.83 between positives and supplied negatives

Positive pairs join proteins with mean positive-network degree 25.26, supplied negatives 6.77. Degree-matched N2 brings the SMD to 0.13.

For model building: A model can tell the classes apart by how well-studied the proteins are, without learning anything about interaction. Compare against degree-matched negatives before trusting a score.

Constructed negatives are putative (unobserved) pairs, not experimentally established non-interactions.

Degree-only shortcut

Degree alone reaches ROC-AUC 0.849 on the reference labels

A logistic model on four endpoint-degree summaries, with no sequence information, scores 0.849 on N0 and 0.528 on degree-matched N2 negatives.

For model building: Most of the benchmark's label separation is reproducible from network degree. Headline scores on these labels mix interaction signal with annotation density.

Degree is computed from the same incomplete positive graph that supplies the labels; it is a retrospective audit feature, not an input available for a protein with no known interactions.

Length-only shortcut

Sequence length alone: ROC-AUC 0.517

Endpoint sequence-length summaries used as the only features.

For model building: Sequence length does not separate the labels.

Performance on unseen proteins

ROC-AUC falls by 0.260 from seen (R1) to unseen proteins (R3)

AAC+CTD MLP with the training set held fixed: R0 0.962, R1 0.962, R3 0.702.

For model building: Random pair splits overstate performance on new proteins. Report the protein-disjoint R3 split if the model is meant for proteins without known interactions.

R3 test sets are small; see the uncertainty chapter.

Signal after degree matching

Modest signal (0.584) with degree-matched negatives on unseen proteins

Best representation AAC+CTD, MLP trained and tested with N2 negatives at R3: ROC-AUC 0.584 (N0: 0.702). Component bootstrap 95% CI 0.472–0.636; above chance under both corrected tests: no.

For model building: Some sequence signal remains, but it is small or uncertain; treat performance claims cautiously.

A non-significant test does not establish equivalence to chance; few connected components limit what can be concluded.

Independent test units

R3 test graphs contain 17–66 connected components

Test pairs that share proteins are not independent. Uncertainty is computed by resampling whole connected components of the test-pair graph.

For model building: Confidence intervals are wide and tests have little power; differences between models of a few hundredths are not interpretable.

A non-significant test does not establish equivalence to chance; few connected components limit what can be concluded.

Taxonomic shortcut

'Same species?' alone scores ROC-AUC 0.728 on pooled degree-matched negatives

Scoring a pair 1 when both proteins come from the same organism gives 0.728 on N2 and 0.500 on species-matched N2-species negatives.

For model building: Pooled negatives that join organisms can be recognised from taxonomy alone; pooled performance partly measures species identification. Use species-matched negatives (N2-species) or evaluate within species.

Constructed negatives are putative (unobserved) pairs, not experimentally established non-interactions.

Cross-species negatives

45.3% of pooled N2 negatives join two species (positives: 0.2%)

Fraction of pairs whose endpoints are assigned to different organisms, per negative set.

For model building: Negatives and positives differ in species composition, which a model can exploit.

Within-species signal

Within-species AAC+CTD MLP with N2 negatives: dmelanogaster 0.533, celegans 0.535, ecoli 0.589

Degree statistics and negatives are rebuilt inside each organism; positives and negatives are split 80/20 at random (proteins may recur, as in the source protocol).

For model building: Within-species estimates remove the taxonomic shortcut but not protein reuse; compare with the pooled R3 result.