For niche academic researchers, systematic literature reviews (SLRs) are both essential and time-consuming. Screening thousands of abstracts for a handful of relevant records, then manually extracting data, can consume weeks. AI automation, specifically using active learning tools like Rayyan and ASReview, offers a practical path from theory to efficient practice. Here is a step-by-step implementation guide grounded in real algorithmic strategies.
1. The Core Challenge: Imbalanced Data
In niche fields, the ratio of relevant to irrelevant records is often extremely low (e.g., 1:100). This imbalance can confuse standard machine learning models. The solution? Dynamic resampling. Both Rayyan and ASReview use this technique to artificially balance the training set during active learning, ensuring the model doesn’t simply learn to predict “irrelevant” for everything. This keeps the screening process focused on finding your rare, high-value papers.
2. Feature Extraction: Why TF-IDF Works
Before a model can learn, it needs to convert text into numbers. TF-IDF (Term Frequency-Inverse Document Frequency) is the default method in ASReview and a strong option in Rayyan. Unlike simple word counts, TF-IDF down-weights common words (e.g., “study,” “method”) and highlights terms uniquely important to your specific research niche. This makes it ideal for academic abstracts where domain-specific jargon matters.
3. The Model: Start with Naive Bayes
For initial screening, you don’t need a deep neural network. Naive Bayes is surprisingly fast and effective for text classification, especially with small datasets typical of niche reviews. It assumes word independence (a “naive” assumption) but performs well in practice, often outperforming more complex models when data is sparse. Use it as your baseline model in ASReview or Rayyan’s built-in classifier.
4. The Query Strategy: Uncertainty Sampling
Active learning is the engine behind these tools. The classic query strategy is uncertainty sampling. Instead of randomly showing you records, the AI prioritizes texts it is most unsure about—those with a predicted probability near 50%. By labeling these “borderline” cases, you quickly teach the model the subtle distinctions in your niche. Both Rayyan (via its “Suggestion” mode) and ASReview (as the default strategy) implement this elegantly.
5. Practical Workflow: Step-by-Step
Step 1: Import. Upload your RIS or CSV file of abstracts into Rayyan or ASReview. Both handle PubMed, Scopus, and WoS exports.
Step 2: Seed. Manually label 5–10 clearly relevant and 10–20 clearly irrelevant records. This initial “seed” kickstarts the model.
Step 3: Screen with Active Learning. Let the tool run. With uncertainty sampling, it will present the most ambiguous records first. You label each one, and the model updates in real time. Expect to screen only 30–50% of the total pool to find 95%+ of relevant records.
Step 4: Validate. After the AI suggests stopping, manually review a random 10% of the “irrelevant” pile to confirm no false negatives.
Step 5: Data Extraction. Export the final included set. For extraction, use Rayyan’s built-in notes feature or integrate with tools like EPPI-Reviewer. The AI doesn’t extract data for you, but it reduces your screening load by 70%, freeing time for meticulous extraction.
By combining dynamic resampling, TF-IDF, Naive Bayes, and uncertainty sampling, you turn a bottleneck into a streamlined, reproducible workflow. Start with Rayyan for its user-friendly interface or ASReview for full transparency and customization.
For a comprehensive guide with detailed workflows, templates, and additional strategies, see my e-book: AI for Niche Academic Researchers: How to Automate Systematic Literature Review Screening and Data Extraction.