Simulated Annealing Searches Brain-Signal Ensembles, but a Noisy Signal Undercuts It

Brain–computer interfaces that translate imagined movement into control signals promise communication channels for people with severe paralysis and new tools for neurorehabilitation. But decoding motor imagery from scalp electroencephalography remains stubbornly hard: the sensorimotor rhythms that imagination modulates — a dampening of the μ (8–13 Hz) and β (14–30 Hz) oscillations over the contralateral cortex — are buried in noise, vary dramatically from person to person, and drift between sessions for the same individual. A new study in Machine Learning with Applications asks a deceptively simple question: instead of fixing the architecture of an EEG decoder by convention, why not search for the best one for each subject?

Researchers Salwa Khalid Abdulateef, Armaneesa Namaan Hasoon and Israa Rafaa Abdulqader built a framework called DAG–SA that treats ensemble construction as a discrete optimisation problem. From a pool of 420 trained base learners — support-vector machines and linear discriminants paired with feature views drawn from Common Spatial Pattern (CSP), Common Spatio-Spectral Pattern (CSSP) and raw channel log-variance extractors across four frequency bands — the method assembles committees of four to eight members. The committees are structured as small directed acyclic graphs: two internal nodes, each grouping a subset of learners, feed a root node, and every node applies one of five fusion operators, from soft voting and majority voting to a conservative minimum rule and a stacking meta-learner.

The scale of the search space is staggering. A formally specified graph grammar generates roughly 9.7 × 10^19 distinct topologies, ranging from about 2.6 × 10^11 four-member ensembles to nearly 10^20 eight-member ones. Exhaustive enumeration is hopeless, so the team turned to simulated annealing, the classic metaheuristic introduced by Kirkpatrick and colleagues in 1983, which wanders rugged discrete landscapes while its temperature-controlled acceptance of worse moves lets it escape local optima. Each candidate topology is scored on held-out validation data, and the annealer perturbs its current graph through five moves: swapping a leaf, changing a node operator, changing the root operator, adding a leaf, or deleting one.

Crucially, the authors designed the evaluation to be airtight. Spatial filters were fitted only on training splits, support-vector probability calibration used internal cross-validation, stacking meta-learners were fitted on out-of-fold predictions so no node was ever scored on data it had seen, and the test split was read exactly once per method. Every comparison was paired: twenty random seeds controlled the data partition, the search randomness and classifier training simultaneously, so all six methods under a given seed were trained and tested on identical trials. The benchmarks were the calibration recordings of BCI Competition IV Dataset 1, with four subjects, and the nine-subject Dataset 2a, restricted to a binary left- versus right-hand task.

The headline result is a negative one, and the authors report it with unusual candour. On Dataset 1, DAG–SA reached 67.2 percent accuracy and ranked fourth of six methods, behind single-best selection (70.2 percent), a Riemannian tangent-space classifier (69.9 percent) and even random search from the same sampler (69.2 percent). Under a stricter repeated cross-validation protocol, the ordering barely changed: DAG–SA again ranked fourth at 68.2 percent. On Dataset 2a it ranked last, at 64.2 percent, while the compact convolutional network EEGNet dominated at 77.2 percent. The elaborate searched ensembles did not beat simply picking the single best pool member.

Why did a sophisticated search lose to a coin-flip-style baseline? The diagnostic experiments point squarely at the selection signal. Validation accuracy estimated from just 22 to 31 calibration trials correlates only weakly with test accuracy — mean Pearson coefficients of 0.39 on Dataset 1 and 0.28 on Dataset 2a, with unit-level values swinging from −0.66 to 0.87. A single validation trial shifts the score by 3.2 to 4.5 percentage points, so choosing the maximum over thousands of topologies systematically favours candidates whose validation scores are inflated by chance. When the authors gave the annealer budgets up to 10,000 evaluations, its validation score climbed steadily while its test accuracy stayed flat.

The annealing schedule itself also contributed. With an initial temperature of 5 on an objective bounded between 0 and 1, plus a reheating rule that multiplies the temperature by 1.5 whenever progress stalls, 96 percent of proposed moves were accepted at the default budget — the annealer was effectively performing a correlated random walk rather than a targeted descent. Lowering the initial temperature to 0.05 cut acceptance to 40–53 percent and raised validation accuracy to near random-search levels, but test accuracy did not follow. On an enumerable sub-pool of ten learners, where all 78,750 four-member topologies could be scored exactly, both searches reached the true optimum in comparable fractions of runs, confirming that the optimiser was not the bottleneck.

Two further findings sharpen the picture. A same-family rule, which confined each annealing run to either the CSP/CSSP family or the log-variance family, turned out to hurt: half the runs were locked into the less accurate family for each dataset, and relaxing the constraint raised test accuracy by 2.4 and 3.5 points on the two datasets — though even the unconstrained search never surpassed the strongest baselines. And a transfer analysis, rebuilding each subject’s selected topology on every other subject, found only a small and statistically uncertain subject-specific advantage: the own topology won by 2.7 points on Dataset 1, concentrated in a single subject, and by a negligible 0.6 points on Dataset 2a.

The study’s real contribution may be methodological rather than empirical. Every returned ensemble is a compact, fully specified object — named learners, their division between two nodes, three fusion operators — that can be inspected, compared across subjects and rebuilt on new data. Inspecting the outputs is what exposed the in-sample stacking bias and the family confinement in the first place. The grammar, shared sampler and fixed budget make the search auditable: every topology the method can return is counted, and the quality of the space can be measured independently of any search. Perhaps most valuably, the paper offers a transferable lesson for the growing field of search-based neural decoders: the reliability of the selection score, measurable directly as the agreement between selection and held-out accuracy over sampled candidates, bounds what any larger search budget or cleverer optimiser can deliver.

For brain–computer interfaces hoping to reach clinical use, the message is sobering but constructive. Before proposing a bigger search space or a smarter metaheuristic for subject-specific calibration, researchers should first ask whether a short calibration block can rank candidate models at all. The authors suggest variance-reduced selection scores — repeated or bootstrapped cross-validation estimates, log-loss, or penalties on ensemble diversity — as the first target for future work, alongside moves that let the annealer escape its initial feature family and pools enriched with learned representations. In a field where deep decoders are often benchmarked on a single split, a rigorously matched, fully released negative result with a quantified explanation may prove more influential than another incremental accuracy claim.

Subject of Research: Simulated-annealing search over directed-acyclic-graph ensembles for subject-specific motor-imagery EEG decoding

Article Title: Annealed search over directed-acyclic-graph ensembles for subject-specific motor-imagery EEG decoding

Article References: Abdulateef, S. K., Hasoon, A. N., & Abdulqader, I. R. (2026). Annealed search over directed-acyclic-graph ensembles for subject-specific motor-imagery EEG decoding. Machine Learning with Applications, 26, Article 101040. https://doi.org/10.1016/j.mlwa.2026.101040

Image Credits: AI Generated

DOI: 10.1016/j.mlwa.2026.101040

Keywords: brain-computer interface, motor imagery, EEG, simulated annealing, ensemble learning, Common Spatial Patterns, directed acyclic graph, model selection, validation overfitting, BCI Competition IV, subject-specific decoding, metaheuristics