This post is about why that was the right call, and when you should think the same way.

What I was building

The project was a medical question answering system. A user asks a clinical question, the system retrieves relevant passages from a medical corpus, reranks them, runs a hallucination check, and generates a grounded answer using Llama-2-7B.

The retrieval layer needed to search across 249,984 BioBERT-embedded vectors quickly and accurately. That is the job FAISS was built for.

The case for FAISS

Vector databases like Pinecone, Weaviate, and Chroma are genuinely useful products. They give you managed infrastructure, real-time updates, metadata filtering, and a REST API out of the box. For a production SaaS product with a team behind it, that makes total sense.

But I was one person, building a research system, running on free Google Colab, with the corpus already sitting in Google Drive as a prebuilt FAISS index.

Every feature a vector database offers over FAISS is a feature I did not need:

Managed infrastructure. I had no server to manage. The entire backend was a Colab notebook. Adding a cloud vector database would mean one more external dependency, one more thing that could go wrong during a demo.

Real-time updates. My corpus was static. Medical textbooks and PubMed abstracts do not change mid-session. The only dynamic retrieval in the system was the live PubMed API call for time-sensitive queries, and that had nothing to do with the index.

REST API. FAISS runs in-process. There is no network hop, no authentication header, no rate limit. You call index.search() and you get results in milliseconds.

Cost. Free.

What FAISS actually looks like in practice

Building the index is straightforward. Embed your corpus, stack the vectors, call faiss.write_index:

import faiss
import numpy as np

dimension = 768  # BioBERT output size
index = faiss.IndexFlatL2(dimension)
index.add(np.array(embeddings).astype("float32"))
faiss.write_index(index, "medical_faiss.index")

Loading it back takes one line:

index = faiss.read_index("medical_faiss.index")

And searching:

query_emb = embedding_model.encode(query,
                convert_to_numpy=True,
                normalize_embeddings=True).astype("float32")
distances, indices = index.search(query_emb.reshape(1, -1), k=32)

That is the entire retrieval layer. No SDK to install beyond faiss-cpu, no API key, no account.

When FAISS is the right call

FAISS makes sense when:

• Your corpus is fixed or updated infrequently
• You are running on constrained infrastructure (local, Colab, a small VPS)
• You want zero external dependencies
• Speed and simplicity matter more than a management UI

Vector databases make sense when:

• Multiple services need to read and write to the index simultaneously
• You need metadata filtering at query time
• The corpus updates continuously in production
• You have a team and do not want to manage the infrastructure yourself

The honest answer

I picked FAISS because it was simple, free, and fit the problem exactly. There was no tradeoff to agonize over.

A lot of engineering decisions look complicated from the outside but are obvious once you are clear about your constraints. My constraints were: no budget, no team, Colab as the runtime, static corpus. FAISS is the answer to that set of constraints every time.

If I were building a production medical platform with a team and paying customers, the answer might be different. But for a research project running on free infrastructure, FAISS did everything I needed and nothing I did not.

This is the technical breakdown of how I built a full RAG-based medical assistant using BioBERT, FAISS, a CrossEncoder reranker, Self-RAG, and Llama-2-7B and deployed it with a React frontend without spending a rupee on compute.

What I Actually Built

Before getting into the how, here is the what.

The system takes a medical question, retrieves relevant passages from a medical corpus, reranks them by relevance, runs a Self-RAG verification step to filter hallucinations, and then generates a grounded answer using Llama-2-7B. The frontend is a React+Vite app that talks to a FastAPI backend running inside Google Colab, exposed publicly via ngrok.

The full stack:

• Retrieval: BioBERT embeddings + FAISS index (249,984 vectors)
• Live retrieval: PubMed API via Biopython for time-sensitive queries
• Reranking: CrossEncoder (ms-marco-MiniLM)
• Hallucination filtering: Self-RAG using cosine similarity scoring
• Generation: Llama-2-7B (4-bit quantized)
• Backend: FastAPI inside Google Colab
• Tunnel: ngrok
• Frontend: React + Vite

Evaluated on PubMedQA: 76% accuracy, 0% hallucination rate. On MedQA: 33.5% accuracy, 0% hallucination. The hallucination number is the one I care about most for a medical system.

The Core Problem: Llama-2-7B Does Not Fit in Free Colab

The free tier of Google Colab gives you a T4 GPU with 15GB of VRAM. Llama-2-7B in full float16 precision needs roughly 14GB just to load. That leaves almost nothing for inference, context, or the rest of the pipeline sitting alongside it.

The fix is 4-bit quantization using bitsandbytes.

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-chat-hf",
    quantization_config=bnb_config,
    device_map="auto",
)

With NF4 quantization and double quantization enabled, the model drops to around 4-5GB VRAM. That is the entire trick that makes this possible. Everything else in the pipeline fits comfortably once Llama is squeezed down.

The Retrieval Pipeline

The retrieval side uses BioBERT to embed both the corpus and incoming queries into the same vector space, then FAISS handles the similarity search.

from sentence_transformers import SentenceTransformer
import faiss

embedding_model = SentenceTransformer(
    "pritamdeka/BioBERT-mnli-snli-scinli-scitail-mednli-stsb",
    device="cuda"
)

def retrieve_faiss(query, k=32):
    query_emb = embedding_model.encode(query,
                    convert_to_numpy=True,
                    normalize_embeddings=True).astype("float32")
    distances, indices = index.search(query_emb.reshape(1, -1), k)
    results = []
    for idx, score in zip(indices[0], distances[0]):
        meta = metadata[idx]
        results.append({
            "chunk_id":    meta["chunk_id"],
            "text":        chunk_lookup.get(meta["chunk_id"], ""),
            "source":      meta["source"],
            "pmid":        meta.get("pmid"),
            "faiss_score": float(score)
        })
    return results

The model variant used pritamdeka/BioBERT-mnli-snli-scinli-scitail-mednli-stsb is fine-tuned for semantic similarity on medical and scientific NLI tasks, which makes it significantly better for retrieval than the base BioBERT or a general-purpose sentence encoder. The FAISS index holds 249,984 vectors built from a chunked medical corpus stored in Google Drive.

The FAISS index holds 249,984 vectors built from a chunked medical corpus stored in Google Drive. But a static corpus has a hard limitation: it goes stale. Medical guidelines get updated. New drugs get approved. Clinical trials publish results. A system that can only search its training-time corpus cannot answer questions about any of that.

Hybrid Retrieval: Bringing PubMed Live Into the Pipeline

This is the part of the system I think is underappreciated in similar projects.

When a user asks something like "what are the latest treatment guidelines for type 2 diabetes" or "recent FDA approvals for immunotherapy", FAISS alone is not enough. The corpus was built at a fixed point in time. The answer might not exist in it.

The solution is a time-sensitivity detector that routes queries to a hybrid retrieval path. If the query contains keywords like "latest", "recent", "current", "updated", "guideline", "recommendation", "approval", or a specific year, the system makes a live call to the PubMed API using Biopython, fetches recent abstracts, embeds them using the same BioBERT model, and merges them with the FAISS results before reranking.

from Bio import Entrez
 
Entrez.email = "your@email.com"
 
TIME_KEYWORDS = ["latest", "recent", "current", "new", "updated",
                 "guideline", "recommendation", "approval"]
 
def is_time_sensitive(query):
    q = query.lower()
    if any(k in q for k in TIME_KEYWORDS): return True
    if re.search(r"\b(19|20)\d{2}\b", q): return True
    if re.search(r"last\s+\d+\s+(months|years)", q): return True
    return False
 
def pubmed_search(query, retmax=20, year_from=2020):
    q = f'{query} AND ("{year_from}"[Date - Publication] : "3000"[DP])'
    handle = Entrez.esearch(db="pubmed", term=q, retmax=retmax)
    record = Entrez.read(handle); handle.close()
    return record["IdList"]
 
def pubmed_fetch(pmids):
    if not pmids: return []
    handle = Entrez.efetch(db="pubmed", id=",".join(pmids),
                           rettype="abstract", retmode="xml")
    records = Entrez.read(handle); handle.close()
    docs = []
    for art in records["PubmedArticle"]:
        med = art["MedlineCitation"]
        art_data = med["Article"]
        title = art_data.get("ArticleTitle", "")
        abstract = " ".join(art_data["Abstract"]["AbstractText"]) \
                   if "Abstract" in art_data else ""
        docs.append({
            "text": f"{title}. {abstract}",
            "source": "PubMed_Live",
            "pmid": str(med["PMID"])
        })
    return docs

Once the live abstracts are fetched, they go through the same chunking and embedding pipeline as the static corpus — split into 512-character chunks with 100-character overlap, embedded with BioBERT, scored against the query, and then merged with the FAISS results. The combined candidate pool then goes into CrossEncoder reranking, which treats static and live chunks equally and picks the top 5 regardless of source.

def hybrid_retrieve(query):
    faiss_results = retrieve_faiss(query, k=32)
    if not is_time_sensitive(query):
        return faiss_results
    live_docs = pubmed_fetch(pubmed_search(query))
    live_chunks = process_live_docs(live_docs, query)
    return faiss_results + live_chunks

The result is a system that answers from its static corpus by default (fast, no API overhead) but automatically expands to live literature when the question demands it. The reranker handles the blending — it does not matter whether a chunk came from the static index or a PubMed abstract published last week, the best passages surface to the top.

This matters more than it sounds. Most RAG demos work on static datasets and look impressive on benchmarks. In a real clinical setting, a doctor asking about a drug approved six months ago would get nothing from a static system. Hybrid retrieval closes that gap without requiring any retraining or re-indexing.

Reranking With CrossEncoder

FAISS retrieves by vector similarity, which is fast but imprecise. The top-k results are not always the most relevant passages for the specific question. CrossEncoder fixes this by doing a proper pairwise comparison between the question and each retrieved passage.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

def rerank(query, passages, top_n=3):
    pairs = [(query, p) for p in passages]
    scores = reranker.predict(pairs)
    ranked = sorted(zip(scores, passages), reverse=True)
    return [p for _, p in ranked[:top_n]]

This two-stage approach, fast retrieval then precise reranking, is standard in production search systems. Running it on free Colab is fine because CrossEncoder is small and runs on CPU without issues.

Self-RAG: The Hallucination Filter

This is the part that took the most iteration, and the part that actually makes the system trustworthy.

The original Self-RAG paper uses a separately trained critic LLM to score whether a generated answer is grounded in the retrieved documents. That is the right idea but the wrong tool for this setup running a second LLM for verification when the first one is already consuming 4-5GB of a 15GB T4 is not feasible.

The approach I used replaces the critic LLM with a two-stage verification pipeline: a support check followed by a utility score. Together they decide whether the answer is good enough to return or whether the system should retry with a refined query.

Stage 1: Support Verification

The support check computes a blended cosine similarity score. A good medical answer needs to satisfy two conditions at once. It should be grounded in the retrieved documents, and it should actually address what was asked. Checking only document similarity is not enough; a model can produce a passage that sounds medically coherent but drifts from the original question entirely.

The score is:

support_score = (avg_doc_sim × 0.4) + (max_doc_sim × 0.3) + (query_sim × 0.3)

Where avg_doc_sim is the mean cosine similarity between the answer embedding and all retrieved document embeddings, max_doc_sim is the similarity to the single most relevant document, and query_sim is the cosine similarity between the answer and the original query. All embeddings come from the same BioBERT model used in retrieval, so the semantic space is consistent throughout the pipeline.

def verify_support(query, answer_text, docs):
    answer_emb = embedding_model.encode(answer_text[:1000],
                     convert_to_numpy=True, normalize_embeddings=True)

    doc_sims = []
    for doc in docs:
        doc_emb = embedding_model.encode(doc["text"][:500],
                      convert_to_numpy=True, normalize_embeddings=True)
        doc_sims.append(float(np.dot(answer_emb, doc_emb)))

    query_emb = embedding_model.encode(query,
                    convert_to_numpy=True, normalize_embeddings=True)
    query_sim = float(np.dot(answer_emb, query_emb))

    avg_doc_sim = float(np.mean(doc_sims))
    max_doc_sim = float(np.max(doc_sims))

    score = round((avg_doc_sim * 0.4) + (max_doc_sim * 0.3) + (query_sim * 0.3), 3)

    if score >= 0.78:   label = "Fully Supported"
    elif score >= 0.58: label = "Partially Supported"
    else:               label = "No Support"

    return {"support_label": label, "support_score": score}

Stage 2: Utility Scoring

Support alone is not enough. An answer can be technically grounded but still be too short or fail to cite its sources. The utility scorer evaluates the answer on three independent axes and produces a score out of 5:

utility_score = length_score + citation_score + support_bonus − penalty

The breakdown:

def score_utility(query, answer_text, support_result):
    score = 0
    word_count = len(answer_text.split())
    num_cite = len(set(re.findall(r'\[\d+\]', answer_text)))
    support_score = support_result.get("support_score", 0.0)
    label = support_result.get("support_label", "No Support")

    if   word_count >= 250: score += 2
    elif word_count >= 130: score += 1

    if   num_cite >= 3: score += 2
    elif num_cite == 2: score += 1

    if support_score >= 0.78: score += 1

    if label == "No Support":
        score = min(score, 2)

    return {"utility_score": max(1, min(5, score))}

Any answer scoring below 4/5 triggers a query refinement and re-retrieval. The system retries up to 3 times, each time expanding the query with additional clinical keywords. If none of the attempts cross the threshold, the best-scoring attempt across all three runs is returned.

How hallucination rate was actually measured

The 0% hallucination rate is not a vague claim, it was measured on 200 samples each from PubMedQA and MedQA. For each sample, the system either generated a response that passed the utility threshold (utility score >= 4) or it did not return an answer at all. An answer was marked as a hallucination if it contained a factual claim that had no correspondence in the retrieved source documents, verified by checking the support_label output. Any answer with a "No Support" label that still passed through would count as a hallucination.

In practice, no such answer passed through. The utility scoring structure makes it structurally impossible: a "No Support" label caps the utility score at 2, which is below the pass threshold of 4. So the system either returns a grounded, cited, detailed answer or it retries. It never surfaces an unsupported response. That is the mechanism behind the 0% number.

Exposing Colab as a Public API

This is the deployment problem nobody talks about. Colab does not give you a public IP. Your FastAPI server runs on localhost inside a VM that disappears every 12 hours.

ngrok solves the first problem. The second one you just have to live with on free tier.

from pyngrok import ngrok
import uvicorn
import threading
from fastapi import FastAPI

app = FastAPI()

@app.post("/query")
async def query_endpoint(request: QueryRequest):
    context = retrieve_and_rerank(request.question)
    answer = generate_with_verification(request.question, context)
    return {"answer": answer, "sources": context}

def run_server():
    uvicorn.run(app, host="0.0.0.0", port=8000)

thread = threading.Thread(target=run_server, daemon=True)
thread.start()

tunnel = ngrok.connect(8000)
print(f"Public URL: {tunnel.public_url}")

The React frontend just points to whatever URL ngrok prints. When the Colab session restarts, you update the URL in the frontend config and redeploy. Not elegant, but it works.

What This Setup Cannot Do

Being honest about the limitations:

The ngrok URL on free tier expires after 8 hours. The Colab session itself disconnects after 12 hours of inactivity or after the daily GPU quota is hit. This means the system is not persistently available, it requires someone to keep the Colab notebook running. For a research demo and conference submission, that is fine. For anything production-facing, you need persistent compute.

Model loading takes 3-4 minutes every time the session starts because Llama-2-7B has to download and quantize on the fly. Caching to Google Drive cuts this down somewhat.

The T4 GPU runs inference at roughly 8-12 tokens per second with this pipeline. Not fast, but usable for research purposes.

The Numbers

Evaluated against two standard medical QA benchmarks:

The MedQA accuracy reflects a known issue: the model shows answer-choice bias toward option A. That is a Llama-2 fine-tuning artifact, not a retrieval problem. Retrieval quality on MedQA is actually reasonable, the generation step is where accuracy drops.

Why I Am Writing This

The point is not that free infrastructure is good. It is that hardware constraints are solvable engineering problems, not blockers. If you are sitting on a similar project idea waiting until you can afford an A100, you probably do not need to wait.

The code is on GitHub: github.com/BasithMrasak