Designing a Real-Time Sentiment Analysis Service
Architecture for scoring streaming text: queue, micro-batching, model, storage, alerts. A lexicon baseline fails on sarcasm, and batching cut scoring cost per item about 150 times.
A real-time sentiment analysis service puts incoming texts on a durable queue, has workers score them with a model in micro-batches, stores each score with the raw text and model version, and serves results and alerts. Interviewers expect the systems trade-offs: batching raises throughput but adds wait time, and accuracy must be measured per class on labeled traffic.
A sentiment service takes a stream of texts (reviews, tweets, support chats), scores each one, and exposes results and alerts within seconds. The design questions are mostly systems ones: absorbing bursts with a queue, batching model calls for throughput, and measuring accuracy honestly. A lexicon baseline scored sarcasm wrong, which is why you evaluate on labeled data.
- In a micro-benchmark of a linear model on 4,096 sparse vectors, scoring cost fell from 31.9 microseconds per item at batch size 1 to 0.2 at batch size 512.
- Micro-batching in a sentiment service trades latency for throughput, because a text can wait up to the batching window before scoring starts.
- When messages arrive at 1,000 per second and workers handle 800 per second, the queue backlog grows by 200 messages every second.
- A lexicon sentiment baseline with a negation rule scored the sarcastic text "Oh great, another crash" as +2, which is wrong.
- If 80% of messages are neutral, a sentiment model that always predicts neutral scores 80% accuracy and is useless, so per-class precision and recall are needed.
01Requirements first
Start with numbers. How many texts per second at peak? How fresh must results be: under a second for a live dashboard, minutes for a daily report? Is the label three-way (positive, neutral, negative), a score from -1 to 1, or per aspect ("battery good, screen bad")? Which languages? How accurate does it need to be, and what does a wrong answer cost?
These answers decide everything below. A support-ticket alerting system that pages a human needs high precision on the negative class. A weekly brand-mood chart can tolerate noise because errors average out.
02The pipeline
Producers (an API, a webhook, a scraper) put raw messages on a durable queue such as Kafka or a cloud queue. The queue decouples bursts from processing: if traffic doubles for a minute, messages wait instead of being dropped. Workers read from the queue, clean the text, call the model, and write results to storage. A serving layer exposes recent results, aggregates and alerts to dashboards and APIs.
Keep the raw text, the model version and the score together. When the model improves, you can re-score history, and when a result is disputed you can show what produced it. Make processing idempotent (writing the same message id twice gives the same row) so retries after a crash do not create duplicates.
03A baseline you can build in ten minutes
Before a neural model, build a baseline. It gives you a number to beat and a fallback when the model is down.
import re
LEX = {'good': 1, 'great': 2, 'love': 2, 'excellent': 2, 'fast': 1,
'bad': -1, 'terrible': -2, 'slow': -1, 'hate': -2, 'broken': -2}
NEG = {'not', 'never', "isn't", "wasn't", "don't", 'no'}
def score(text):
toks = re.findall(r"[a-z']+", text.lower())
s = 0
for i, t in enumerate(toks):
v = LEX.get(t, 0)
if v and i > 0 and toks[i - 1] in NEG:
v = -v # flip after a negator
s += v
return sResults: "The app is great and fast" scores 3, "Terrible, slow and broken" scores -5, "not good" scores -1, and "I do not hate it" scores 2. The negation rule works for these. But "Oh great, another crash" scored +2, which is wrong, because the sarcasm is invisible to a word list. The same weakness appears with domain words ("sick" in a slang review, "unpredictable" for a film versus a car). A trained model handles more of this, though sarcasm remains hard for those too.
04Micro-batching the model
Model inference has a fixed overhead per call, so scoring one text at a time wastes most of the work. A micro-batcher collects messages for a short window (say up to 32 texts or 20 ms, whichever comes first), scores them together, and returns the results.
To see the effect I scored 4,096 sparse feature vectors with a linear model, at different batch sizes, on this machine:
batch size 1: 31.9 microseconds per item
batch size 8: 4.3
batch size 64: 0.6
batch size 512: 0.2Batching cut the cost per item by about 150 times. This is a micro-benchmark of the arithmetic only, and the numbers will differ on your hardware and model. The gain for a neural model on a GPU depends on the model and hardware, so benchmark your own. The trade-off is latency: a text can wait up to the window length before scoring starts. Set the window from your latency target, not from the largest batch the hardware likes.
05Back-pressure and scaling
A quick model shows why monitoring queue depth matters. Messages arrive at 1,000 per second and workers handle 800 per second. After 10 seconds the backlog is 2,000 messages and it grows by 200 every second. With workers able to do 1,200 per second the backlog stays at zero.
Two controls help. Scale workers out when the queue depth or the age of the oldest message passes a threshold. And decide what to do when you cannot keep up: process newest first, drop low-priority sources, or fall back to the cheap baseline model. Alert on queue age, since a short queue of old messages is a bigger problem than a long queue that drains.
06Evaluating accuracy honestly
Accuracy on a public dataset does not tell you accuracy on your traffic. Take a few hundred recent messages, have people label them, and compute precision and recall for each class, not just overall accuracy. If 80% of messages are neutral, a model that always says neutral scores 80% accuracy and is useless.
Re-check regularly. Language drifts, a product launch brings new vocabulary, and a new source behaves differently from the training data. Track the share of each label over time and sample outputs for review. A sudden shift in the distribution is usually a data problem before it is a change in real opinion.
07Privacy and limits
User text may include names, emails and phone numbers. Decide what you store, mask what you do not need, and set a retention period. Be clear when presenting results that a sentiment score is a model's estimate, not a fact about the writer. Be careful about using it to make decisions about individual people, where a mistake has real consequences.
08Common interview questions
Q: Why put a queue in front of the model?
A: To absorb bursts, decouple producers from workers, and allow retries without losing messages.
Q: What is micro-batching and what does it cost?
A: Grouping several requests into one model call to raise throughput, at the cost of a small added wait for each request.
Q: How do you make the pipeline safe to retry?
A: Make writes idempotent, keyed by message id.
Q: How do you evaluate the model?
A: Labeled samples from real traffic, with per-class precision and recall, not accuracy alone.
Q: What fails in simple lexicon methods?
A: Sarcasm, domain-specific words, and context beyond simple negation.
Q: How do you detect that the model has degraded?
A: Monitor label distribution and sampled human review, and compare against a held-out labeled set.