PPI benchmark evaluation

PPI Benchmark Studio · engine 0.1.0 (framework 1.0.0) · 2026-10-14T17:12:48+0000 · seed 42 · preset standard

Needs controls — At least one artefact materially affects what models trained here will learn.

Readiness score 35/100. Heuristic summary of the diagnostics below (100 − 15 per concern − 5 per caution); not a statistical quantity.

Diagnostics

pass

Data integrity

12 rows removed during cleaning (0.3%)

12 duplicate pairs (0 reversed), 0 self-pairs and 0 pairs with conflicting labels were removed; 3,600 canonical pairs remain.

For model building: Reversed duplicates (A–B and B–A) placed on both sides of a random split leak labels directly. Train on the canonical pair list, not the raw file.

fail

Degree bias in supplied negatives

Endpoint degree SMD 0.99 between positives and supplied negatives

Positive pairs join proteins with mean positive-network degree 16.12, supplied negatives 2.21. Degree-matched N2 brings the SMD to 0.13.

For model building: A model can tell the classes apart by how well-studied the proteins are, without learning anything about interaction. Compare against degree-matched negatives before trusting a score.

Constructed negatives are putative (unobserved) pairs, not experimentally established non-interactions.

fail

Degree-only shortcut

Degree alone reaches ROC-AUC 0.981 on the reference labels

A logistic model on four endpoint-degree summaries, with no sequence information, scores 0.981 on N0 and 0.511 on degree-matched N2 negatives.

For model building: Most of the benchmark's label separation is reproducible from network degree. Headline scores on these labels mix interaction signal with annotation density.

Degree is computed from the same incomplete positive graph that supplies the labels; it is a retrospective audit feature, not an input available for a protein with no known interactions.

pass

Length-only shortcut

Sequence length alone: ROC-AUC 0.553

Endpoint sequence-length summaries used as the only features.

For model building: Sequence length does not separate the labels.

fail

Performance on unseen proteins

ROC-AUC falls by 0.193 from seen (R1) to unseen proteins (R3)

AAC+CTD MLP with the training set held fixed: R0 0.898, R1 0.906, R3 0.713.

For model building: Random pair splits overstate performance on new proteins. Report the protein-disjoint R3 split if the model is meant for proteins without known interactions.

R3 test sets are small; see the uncertainty chapter.

warn

Signal after degree matching

Modest signal (0.712) with degree-matched negatives on unseen proteins

Best representation AAC+CTD, MLP trained and tested with N2 negatives at R3: ROC-AUC 0.712 (N0: 0.713). Component bootstrap 95% CI 0.000–0.717; above chance under both corrected tests: no.

For model building: Some sequence signal remains, but it is small or uncertain; treat performance claims cautiously.

A non-significant test does not establish equivalence to chance; few connected components limit what can be concluded.

fail

Independent test units

R3 test graphs contain 2–7 connected components

Test pairs that share proteins are not independent. Uncertainty is computed by resampling whole connected components of the test-pair graph.

For model building: Confidence intervals are wide and tests have little power; differences between models of a few hundredths are not interpretable.

A non-significant test does not establish equivalence to chance; few connected components limit what can be concluded.

Recommendations

Train on the canonical pair list (canonical_pairs.tsv): Duplicates and reversed duplicates have been removed; use this file instead of the raw input.
Use degree-matched negatives (negatives/N2.tsv): Train and evaluate with the same degree-matched definition (N2, or N5 to also match length) and report the degree-only baseline.
Report a protein-disjoint split (splits/R3/): R1/R2/R3 share one training set; R3 test pairs contain only proteins never seen in training. Report R3 for claims about new proteins.
Report component-aware uncertainty (tables/component_aware_uncertainty.csv): Quote bootstrap and jackknife intervals over connected components, and add an independent benchmark where possible.
Report both controls (results.json): Protein reuse and negative construction are distinct problems: a protein-disjoint split can keep biased negatives, and balanced negatives can still let proteins recur.

Negative-set balance

negative_set	n_negatives	positive_mean	negative_mean	smd	ks_statistic	length_smd
N0	1800	16.117	2.208	0.991	0.674	-0.116
N1	1800	16.117	5.768	0.695	0.368	-0.099
N2	1800	16.117	13.739	0.134	0.047	-0.007
N3	1800	16.117	10.800	0.301	0.185	0.002
N5	1788	16.117	13.779	0.132	0.045	-0.001
N2-exact	1549	16.117	13.041	0.172	0.060	0.017
N2-nearest	1800	16.117	15.444	0.036	0.014	0.018

Splits and leakage gates

regime	train_pairs	test_pairs	train_proteins	test_proteins	protein_overlap	test_components	passed
R0	2880	720	600	531	531	9	True
R1	1945	486	480	399	399	15	True
R2	1945	1056	480	539	419	1	True
R3	1945	113	480	104	0	7	True

Protein-novelty curve

representation	regime	model	test_pairs	roc_auc	pr_auc	mcc
AAC+CTD	R0	lr	720	0.851	0.863	0.585
AAC+CTD	R0	rf	720	0.858	0.851	0.577
AAC+CTD	R0	mlp	720	0.898	0.910	0.656
AAC+CTD	R1	lr	486	0.853	0.861	0.537
AAC+CTD	R1	rf	486	0.865	0.848	0.560
AAC+CTD	R1	mlp	486	0.906	0.914	0.671
AAC+CTD	R2	lr	1056	0.731	0.757	0.335
AAC+CTD	R2	rf	1056	0.805	0.805	0.458
AAC+CTD	R2	mlp	1056	0.810	0.825	0.478
AAC+CTD	R3	lr	113	0.564	0.639	0.179
AAC+CTD	R3	rf	113	0.824	0.860	0.530
AAC+CTD	R3	mlp	113	0.713	0.782	0.215

Topology-only and length-only controls

feature	evaluated_on	roc_auc	pr_auc	mcc
degree	N0	0.981	0.981	0.841
degree	N1	0.825	0.817	0.453
degree	N2	0.511	0.533	-0.022
degree	N3	0.667	0.654	0.204
degree	N5	0.491	0.523	-0.012
degree	N2-exact	0.527	0.579	-0.017
degree	N2-nearest	0.492	0.516	-0.032
length	N0	0.553	0.527	0.123
length	N1	0.576	0.559	0.131
length	N2	0.561	0.566	0.074
length	N3	0.534	0.536	0.051
length	N5	0.506	0.510	0.017
length	N2-exact	0.549	0.584	0.074
length	N2-nearest	0.529	0.525	0.042

Consistent-negative revalidation

representation	regime	negative_set	test_pairs	roc_auc	pr_auc	mcc
AAC+CTD	R0	N0	720	0.892	0.899	0.598
AAC+CTD	R3	N0	113	0.713	0.782	0.215
AAC+CTD	R0	N2	720	0.602	0.575	0.136
AAC+CTD	R3	N2	130	0.712	0.702	0.292
AAC+CTD	R0	N5	718	0.557	0.529	0.116
AAC+CTD	R3	N5	108	0.726	0.785	0.338

Component-aware uncertainty (R3)

representation	negative_set	n_test	n_components	roc_auc	boot_low	boot_high	jack_low	jack_high	boot_p_bh	jack_p_bh
AAC+CTD	N0	113	7	0.713	0.698	0.727	0.695	0.732	0.000	0.000
AAC+CTD	N2	130	2	0.712	0.000	0.717	-3.844	5.267	0.508	0.660
AAC+CTD	N5	108	6	0.726	0.167	0.741	0.117	1.336	0.508	0.575

Methods

The dataset (600 proteins, 3,612 pairs) was audited and canonicalized to 3,600 unique undirected pairs (1,800 positive, 1,800 negative). Following the protein-novelty and negative-sampling control framework, a random pair split (R0) and a matched protein-novelty curve (R1 both proteins seen, R2 one unseen, R3 both unseen; shared training set of 1,945 pairs) were built from one partition of identical-sequence groups, with automated leakage gates. Putative negative sets (N0, N1, N2, N3, N5, N2-exact, N2-nearest) were constructed by one-sided substitution matched on positive-network degree and/or sequence length deciles. AAC+CTD descriptors (167-d) with symmetric pair fusion were used with lr, rf, mlp models. Degree-only and length-only classifiers were trained on the reference negatives and evaluated on each alternative set, and an MLP was re-evaluated with each negative definition used consistently for training and testing at R0 and R3. Uncertainty was quantified with a connected-component bootstrap (1000 replicates) and a delete-one-component jackknife, with Benjamini–Hochberg adjustment. Seed 42.