Learning AI engineering with Ruby

[u/hitthatliq]

I'm learning more about AI engineering, specifically the application/production side rather than ML research or training models.

The areas I'm interested in are things like:

• RAG, embeddings, vector search and reranking
• tool/function calling
• agentic workflows and orchestration
• MCP clients/servers
• structured outputs
• evals and observability
• persistent agent state / human approval
• building production AI features into normal web applications

I originally learned to code with Ruby, but since then all my jobs have been Python or TypeScript based, and I really miss working with Ruby.

I'm curious what Ruby libraries/tools people are using for RAG, agents, MCP, evals, tracing, etc. Are there good Ruby alternatives to frameworks like LangChain? What does your stack look like?

And if you were learning this stuff today, would you recommend doing it in Ruby, or sticking with Python because that's where most of the ecosystem and learning material is?

I'm not trying to avoid Python entirely. I'm more curious about how viable Ruby is for the application/agentic side of AI development.

Side note: I realise some of these areas aren't particularly language-specific. For example, with RAG a lot of the retrieval/storage work may ultimately be handled by pgvector, a vector database, or another service.

[u/Minimum_Scholar_8136]

Ruby is in a much better place for this than it was eighteen months ago, but the honest shape of the ecosystem is "good primitives, no monolithic framework." That's mostly a feature — a lot of what LangChain does is abstraction you end up fighting — but it does mean you assemble rather than adopt.

Roughly how the pieces map:

LLM clients. ruby_llm is the closest thing to a default now: one interface across providers, with streaming, tool calling and structured output. ruby-openai is older and very solid if you're single-provider, and Anthropic ships an official Ruby SDK. langchainrb exists and is the most direct LangChain analogue if you specifically want that shape.

Embeddings and vector search. This is the strongest area, mostly thanks to Andrew Kane. neighbor gives you pgvector through ActiveRecord and it is genuinely pleasant — nearest-neighbour scopes on a normal model. If you're on SQLite there's sqlite-vec. For local embedding generation without a Python service, informers runs sentence-transformers models through ONNX in-process. A Postgres-backed RAG store in Rails is a couple of hours of work, not a project.

Reranking is the actual gap. There's no well-adopted Ruby reranker; people call Cohere's rerank endpoint or a hosted cross-encoder over HTTP. Fine in practice, just don't expect a gem.

Tool calling / structured output. Handled at the client layer rather than by a framework — ruby_llm and raix both do function calling and schema-constrained output. Obie Fernandez's Patterns of Application Development Using AI is the closest thing to a Ruby-native book on the application patterns, and worth reading even where you disagree with it.

MCP. There are Ruby server implementations and they work, but this is the youngest corner — expect to read source.

Evals and observability. This is where you'll feel Python's lead most. There's no Ruby equivalent of a mature eval harness. What works: OpenTelemetry Ruby is good, so instrument spans yourself around model calls and log prompt/response pairs to a normal table; then your "eval suite" is Minitest or RSpec over a fixture set of inputs with assertions on the parsed structured output. It sounds primitive next to a dedicated tool and it is roughly what the dedicated tools do anyway. Being able to write evals as ordinary tests against your real models and real database is a genuine Rails advantage.

Persistent agent state and human approval is the part where I'd argue Rails is actually the best environment of the three, not the worst. Agent state is just rows. Human-in-the-loop approval is a state machine, a background job that halts, and a UI — all things Rails already does better than a Python service. Solid Queue gives you durable job state without extra infrastructure.

The one thing to know before you commit, because it will bite you in production: streaming responses via ActionController::Live hold a Puma thread for the entire request. For a thirty-second answer that's a thread gone for thirty seconds, and under load it presents as unrelated endpoints getting slow, which sends you debugging the wrong thing. Size the pool for it, or push the streaming onto Action Cable instead. Also set X-Accel-Buffering: no on any SSE response or your proxy will buffer the whole "stream" into one lump and nothing in the logs will tell you why.

If I were starting today: ruby_llm + neighbor on Postgres + Solid Queue, evals as plain tests, OTel spans for tracing. Skip the framework layer until something actually hurts.

[u/arup_r → u/Minimum_Scholar_8136]

Everything you said very interesting and all went above my head. How did you learn these all? Can you guide me ? All I do prompts to Claude. But I’m interested to looking beyond what I am doing. I’m equally feeling so behind these races. 😏

[u/hitthatliq → u/arup_r]

I definitely wouldn't feel behind. A lot of this is just terminology, and once someone explains what the pieces actually are it becomes much less intimidating.

I'd probably start by getting a basic mental model of the main concepts:

• RAG is essentially a process of grabbing information from your own database or knowledge base to supplement a prompt with information the model (e.g. Claude) doesn't already have access to. To find the information that's most relevant to your prompt, a RAG system will often use embeddings and vector search.
• Embeddings are representations of text, images, etc. as numbers. I think of them as coordinates in a high-dimensional space, where things with similar meaning end up closer together. You can send some data to an embedding model and get back an embedding vector, which you can store in a database or use for vector search.
• Vector search means searching those embeddings to find the most similar or relevant information.
• Reranking is a second pass over the initial search results, using a more accurate (and often more computationally expensive) technique to move the most relevant results to the top.

So a simple RAG flow might look like:

user submits a prompt → embedding model turns the prompt into an embedding → database uses that embedding to find relevant information → results are reranked → original prompt + relevant information are sent to the LLM → user gets a response.

• Tool/function calling means giving the LLM functions in your application that it can choose to call, such as querying a database, calling an API or sending an email.
• MCP is a standard way for applications to expose tools and context to AI models.
• Agents are generally systems where the model can decide what steps to take and which tools to use, rather than you hard-coding every step of the process.

I learned a lot of these general ideas from DeepLearning.AI. Their RAG course is particularly good:
https://www.deeplearning.ai/courses/retrieval-augmented-generation

It's not Ruby-specific, but the concepts themselves aren't really Ruby-specific either.