Continually self-improving AI

Modern language model-based AI systems are remarkably powerful, yet their capabilities remain fundamentally capped by their human creators in three key ways. First, although a model’s weights can be updated via fine-tuning, acquiring new knowledge from small, specialized corpora after pretraining remains highly data-inefficient. Second, the training of these systems relies heavily on finite, human-generated data from across history. Third, the pipelines used to train AI models are confined by the algorithms that human researchers can discover and explore. This thesis takes a small step toward overcoming these inherent limitations, presenting three chapters aimed at breaking these dependencies to create continually self-improving AI. First, to overcome this data-efficiency barrier in knowledge acquisition, we propose a synthetic data approach that diversifies and amplifies small corpora into rich knowledge representations, enabling a model to effectively update its parameters from limited source material. Second, to reduce reliance on human data, we show that given a fixed amount of such data, the model can self-generate synthetic data to bootstrap its fundamental pretraining capabilities without distillation from any off-the-shelf, instruction-tuned LM. Finally, to transcend human-engineered training paradigms, we demonstrate that by scaling search during test time over the space of algorithms, AI can search over a larger space of learning algorithm configurations than human researchers can explore manually.

💡 Research Summary

This dissertation tackles three fundamental bottlenecks that limit modern large language models (LLMs): (1) data‑efficiency in knowledge acquisition, (2) reliance on a finite amount of human‑generated pretraining data, and (3) confinement to human‑designed training pipelines. To address these, the author presents three self‑contained research chapters that together form a roadmap toward “continually self‑improving AI.”

Chapter 2 – Continual Knowledge Acquisition
The first contribution introduces EntiGraph, a synthetic‑data generation pipeline that transforms a small, domain‑specific corpus into a structured entity‑relationship graph. By sampling diverse paths through this graph, the system automatically produces a large, varied set of sentences, questions, and answers that preserve the semantic core of the original material while dramatically expanding surface‑form diversity. Experiments on the challenging QuALITY reading‑comprehension benchmark show absolute gains of 3–5 % over standard fine‑tuning, even when the synthetic data are generated by a relatively weak language model. Ablation studies reveal that both the fidelity of the entity graph and the lexical diversity of the generated text are critical to downstream performance.

Chapter 3 – Synthetic Bootstrapped Pretraining (SBP)
The second contribution tackles the scarcity of human‑curated pretraining data. Under a “data‑constrained pretraining” regime (e.g., only 10 B tokens of real data), the author proposes a bootstrapping loop: (1) pre‑train a base model on the limited real data, (2) use that model to generate a massive synthetic corpus, and (3) continue pre‑training on a mixture of real and synthetic data. This loop effectively doubles data efficiency: the same downstream performance is achieved with roughly half the amount of genuine data. The author quantifies token‑level diversity and perplexity reductions, and demonstrates that a 1 T‑token synthetic experiment matches the performance of a full‑scale 2 T‑token real‑data run. Scaling experiments (7 B → 70 B parameters) indicate that SBP’s benefits grow with model size.

Chapter 4 – Test‑time Algorithm Search for AI‑Designed AI
The third contribution moves beyond data to the algorithmic level. The author builds a meta‑learning framework that, at inference time, searches a huge space of learning‑algorithm configurations (optimizers, learning‑rate schedules, regularizers, etc.). Using a budget‑forcing strategy and execution‑guided evolutionary search, the system evaluates many candidate configurations within a fixed compute budget. Compared to a best‑of‑N (BO) baseline, the method converges 1.8× faster and yields 2–3 % higher accuracy on the same tasks. Importantly, the automatically generated “ideas” (algorithmic proposals) are more diverse than those produced by human researchers, suggesting a pathway to AI‑designed training pipelines that surpass human intuition.

Integration and Limitations
Together, these three strands reduce dependence on human data, improve data efficiency, and expand the algorithmic search space beyond what humans can manually explore. The dissertation acknowledges several open challenges: (i) ensuring factual correctness of synthetic text to avoid contaminating the model with hallucinations; (ii) preventing self‑reinforcing biases during the SBP loop; (iii) managing the computational cost of large‑scale test‑time search.

Future Directions
The author proposes several avenues: (a) integrating automated fact‑checking or truth‑verification modules into the synthetic generation pipeline; (b) designing self‑correcting feedback loops that detect and mitigate bias amplification; (c) combining reinforcement learning or Bayesian optimization with evolutionary search to improve meta‑learning efficiency; and (d) unifying EntiGraph‑based continual learning with SBP to create a truly infinite‑scale pretraining regime that feeds directly into the test‑time algorithm search.

Overall, the dissertation makes a compelling case that synthetic data generation and automated algorithmic exploration can jointly break the three traditional constraints on LLMs. The empirical results across reading comprehension, instruction following, and benchmark summarization demonstrate tangible performance gains, while the conceptual framework points toward a future where AI systems can iteratively improve themselves with minimal human intervention.