# I Built a Twitter Bot Detector in 2022. I Finally Evaluated It Honestly in 2026.

Published: 2026-08-25 · Tag: AI
Canonical: https://vondraysanford.com/writing/2026-08-25-i-built-a-twitter-bot-detector-in-2022-i-finally-evaluated-i.html
Author: Vondray Sanford (https://vondraysanford.com)

> A look back at the Python NLP pipeline I built in 2022: sentiment analysis, XGBoost bot detection, and the evaluation protocol I should have written four years ago.

Going back to old code is humbling in a specific way. It's not that the code is embarrassing. It usually works. It's that you can see exactly where you stopped short. You shipped the thing, moved on, and never asked whether it was any good. That's what happened with my [Twitter Sentiment Analysis Bot](https://github.com/vondraysanford/TwitterSentimentAnalysisBot), a project I built in 2022 and didn't properly evaluate until this year.

Here's what the project does, what I went back and measured, and what four years of building harder systems taught me to see.

## What the Project Does

The bot has two jobs. First, it scores tweet sentiment using TextBlob plus a custom model I trained on Amazon Fine Food Reviews and a Twitter dataset. Second, and this is the more interesting part, it classifies Twitter accounts as bot or human using a 13-feature XGBoost classifier. Both functions are exposed through a Discord bot built with `discord.py`: a `!sentiment` command and an `!analyze` command that post live results directly into a channel. Data comes from the Twitter API v2 via Tweepy, which dates the project all by itself. This was back when the platform was still called Twitter and the API was still something a hobbyist could build against.

The one thing 2022-me did that I still think was the right call: I RSA-signed the pickled model using `SignPickle.py` and `VerifyPickle.py`. If the `model.pickle` file had been tampered with, the bot would refuse to load it. That's a real supply-chain concern with serialized ML artifacts, and honestly I'm a little surprised I took it seriously back then. It tells me I was already treating the model as infrastructure, not just a script output.

## The Evaluation I Should Have Done in 2022

The original repo had no honest metrics. No held-out test set, no confusion matrix, no AUC. Just a model file and a working interface. That's the gap I finally went back to close, four years late.

I ran a proper evaluation protocol: 35,874 accounts split 70/30, with 11,923 labeled bots and 23,951 humans. The results came back at **ROC AUC 0.890**, accuracy 81.8%, bot precision 71.6%, and bot recall 76.6%. Five-fold cross-validation gave me AUC 0.891 ± 0.003, so the model isn't fitting one lucky split. It generalizes consistently.

I also wrote `GenerateEvalFigures.py`, a reproducible script that regenerates the ROC curve, confusion matrix, and feature importance chart from the deployed model. Anyone who clones the repo can run it and get the same figures. Small thing, but it's the difference between "trust me, it works" and showing your work.

I used `scale_pos_weight` in XGBoost to handle the class imbalance, since bots are the minority class in real-world data. That parameter trades some precision for higher recall on bots, which I did on purpose. For a screening tool, missing a bot costs more than flagging a legitimate account for review. A false negative lets a bot slip through undetected. A false positive flags a human who can be checked again. That was the right tradeoff for this use case, and I'd make the same call today.

## The Feature That Did the Heavy Lifting

This is the part I'm most proud of, and it came from thinking about how bots behave rather than throwing raw columns at the model.

The strongest feature by a wide margin is one I engineered:

```
network = log(followers) × log(following)
```

Bots follow aggressively but attract almost no followers back. That creates a specific pattern: high following counts, low follower counts. Neither raw number captures it cleanly on its own because the scale varies wildly. The log-product separates the pattern well. A human with 1,000 followers and 800 following scores very differently from a bot with 12 followers and 4,000 following, and the log transformation keeps those gaps meaningful without letting large raw numbers dominate.

It leads the feature importance chart by a noticeable margin over everything else. I didn't find it by trying a fancier model. I found it by thinking about the behavior I was trying to catch and writing that behavior down as math. When your dataset isn't enormous, feature engineering that encodes domain understanding beats model complexity.

## What I'd Do Differently Now

I build RAG pipelines and multi-agent systems now, and that context makes a few gaps in this project obvious.

There's no experiment tracking. I trained the model, it worked, I shipped it. I have no record of what I tried before that, what hyperparameters I swept, or what features I dropped. If I built this today I'd have MLflow or at minimum a structured log of every run. Going back to understand *why* a model performs the way it does is nearly impossible without that history.

There's also no probability calibration. XGBoost outputs scores in the right direction, but the raw probabilities aren't calibrated. A 0.72 score doesn't necessarily mean 72% confidence in any meaningful sense. For a screening tool that's probably fine, but if I were building something that fed into an automated action rather than a human review queue, I'd want Platt scaling or isotonic regression on top.

And there's no drift monitoring. Account behavior in 2022 isn't account behavior in 2026. The platform has changed its name, its API pricing, and its bot population since I trained this model. Bot operators adapt. The feature distribution shifts. I'd want something watching for that: baseline distributions, PSI scores, alerting when incoming data stops looking like the training data. That's the exact problem I'm building DriftWatch around right now, and going back to this project made me appreciate why it matters.

## What Revisiting Old Work Teaches You

This was early work, and I want to be clear about that. An AUC of 0.890 on a bot detection task is decent, not exceptional. The sentiment model is simple. The Discord interface is more demo than production. I knew that when I built it.

But two things hold up. One: I shipped an end-to-end system with a real interface. Not a notebook, not a script that outputs to stdout. A working bot that a user could query and get results from in a channel. That instinct to build the whole thing and not just the interesting part is something I still lead with. Two: I treated the model artifact as a supply-chain risk and did something about it. The RSA signing wasn't required for the project to work. I did it because it was the right call.

What revisiting old work really shows you is your own taste. The decisions you made when you were moving fast, before you had the vocabulary to explain them, reveal what you actually value. I cared about shipping complete systems and treating artifacts carefully. I didn't yet care about reproducibility, evaluation rigor, or long-term observability. Now I care about all of it.

If you've got an old project sitting in a repo somewhere with no real metrics attached to it, go back and do the evaluation. Not to embarrass yourself. Do it to find out which of your instincts were right before you could explain them, and which habits you still haven't built. Mine sat for four years before I checked, and that second list turned out to be worth more than the metrics.