Introduction
Since earlier this year, we've been building AI search / chat systems using Retrieval-Augmented Generation (RAG) for a number of client companies. Several of these projects used open-weight LLMs rather than OpenAI or Gemini, so here we'll introduce, through anonymized case studies, why we chose open-weight LLMs and what business value they actually delivered.
Why we adopted open-weight LLMs
- Minimizing confidentiality and information-leak risk Even though SaaS-based APIs (OpenAI, Google Gemini, etc.) offer settings to opt out of having data used for training, some clients who require fully offline operation still have concerns.
- Ability to close off and isolate the network Deploying the LLM to our own cloud or on-premises environment lets us integrate access control and audit logging with existing policies.
- Freedom to customize It's easy to fine-tune the model — for example through model compression or enhanced multilingual support — to fit the use case.
That said, for clients who are fine with API-based usage, we also quickly deliver RAG systems built around OpenAI's, Google's, and similar SDKs.
System architecture (representative example)
Layer | Technology used (example) | Notes |
|---|---|---|
LLM | Google Gemma, Meta Llama 3, Mistral, etc. | Choice of 2B–70B depending on requirements |
Vector DB | Qdrant | Fast search + scalar value filtering |
Main DB | PostgreSQL | User management, access control, etc. |
API | FastAPI | Python-based, high-performance ASGI |
Frontend | Remix (React) | Supports both CSR/SSR, proven track record with in-house SSO integration |
- A full-text search setup combining Elasticsearch/OpenSearch is also an option
What it can do — AI that can be "embedded" into your workflow
- An in-house version of NotebookLM Users drag and drop documents to upload → a dedicated RAG is generated within minutes Even without uploading anything, a regular ChatGPT-like conversation is also possible
- Collection and permission management Control visibility by department or project, and share results with a single click
- Shared conversation logs Quote and forward "the question I asked and the answer I got" to teammates to build up shared knowledge
- Meeting minutes summarization & task extraction Feed in a meeting transcript recorded via a voice memo app on your iPhone or similar, or text from a Zoom recording, and instantly generate a summary plus a to-do list
Through this kind of setup, document creation, planning, and inquiry handling can all be completed via the shortest possible route. Results reported include things like "cut search time by 80%" and "shortened proposal-draft creation to 1/5 of what it used to take."
For those considering adopting AI
- A speed-first plan built around cloud APIs
- A security-first plan with an open-weight LLM built entirely within a closed, in-house environment
—— We'll propose the optimal solution that doesn't compromise on either your information governance or your development speed. Feel free to reach out for a consultation. Our consultants will hear out your use case and deliver a PoC (proof of concept) in as little as 5 business days.
For those with concerns about adopting AI
We'll propose an environment suited to your situation, whether that means using an LLM like OpenAI's in line with your company's security policy, or using an open-weight LLM. Please feel free to reach out for a consultation first!
