Semantic Search Without a Second Database

Introduction

This post is about customer search on a multi-merchant food delivery platform — diners, restaurants, menus, and a delivery radius — not a generic document corpus.

The product looks familiar: a customer app (Expo / React Native) and a merchant-facing surface, a Spring Boot catalog API that already serves nearby stores, an order API that records checkouts and DELIVERED events, and DynamoDB as the system of record. Runtime is AWS: ECS Fargate behind a shared ALB, DynamoDB on-demand, no extra always-on search cluster. Nearby already works as a zone fan-out plus a haversine radius. Filter chips for cuisine and dietary already exist on the search screen.

That is the environment this pattern is built for. A diner opens the app, shares location, and types something a restaurant would never print on a menu: cheep vegan sushi thats fast, piza, hangover. They expect restaurants and dishes inside delivery range, ranked by meaning and distance — not a substring match on restaurant names, and not a full menu download so the phone can scan items client-side.

Why search is geolocated

This is the single assumption that shapes the whole design, so it is worth stating up front: customer search on a food delivery platform is inherently local. A few reasons stack up.

Delivery has a range. A restaurant can only deliver so far before the food is cold, the driver economics break, and the ETA is unacceptable. A restaurant 40 km away is not a valid result no matter how well it matches "vegan sushi." The product only ever shows restaurants that can actually reach the customer.

The customer's location is the primary filter. The very first thing that happens is resolving the customer's address or GPS into an area. Everything after that — browse, nearby, search — is already scoped to that area. Search is not a separate global thing; it is "search within what I can order from."

Relevance is distance-aware. Even among valid matches, closer usually wins: faster delivery, lower fees, more reliable. Distance is part of ranking, not just a filter. That only makes sense if the query is tied to a location.

It matches how the data is already organized. In this architecture the catalog is partitioned by zone (zonePk). The app resolves lat/lng to a zone, then queries that zone. Search reuses the same zoning, so it naturally stays local.

The payoff: because every query is tied to a delivery area, no customer search ever has to rank against the entire catalog at once. That is why per-zone vectors on DynamoDB fit this access pattern, and why a global vector database is more machinery than the problem needs.

The rest of this article is that design: in-process ONNX, vectors on the catalog items you already write, DynamoDB SearchVectors for retrieval, and a fail-open parser for typos and dietary slots. The failure mode it replaces is specific: typos never match, dishes stay invisible until the client downloads the menu, and every new search product wants a new database.

Locked decisions

What you are replacing is ZoneIndex + substring match, plus a client-side menu scan. What you are not replacing: the nearby-discovery feed, the existing filter chips, or the geo model.

What the naive path actually does

A typical nearby search is already geo-correct and semantically blind:

1. Resolve intersecting zonePk values from lat/lng
2. Load store summaries from those partitions
3. Filter with name.contains(query) / cuisine.contains(query)
4. Haversine-drop anything outside radiusMiles
5. If the user meant a dish, the phone iterates menus client-side

That is cheap and honest for exact store names. It cannot recover piza → pizza, cannot rank "margherita" against a store whose name does not contain the word, and it burns mobile battery and payload size as soon as dish search matters.

Four layers

Ship them in this order. Order-again does not need vectors and can go out first. Taste uses a small sibling table; everything else stays on the catalog.

The mobile client prefers the search API. /nearby + client contains() is the fallback when the feature flag is off or the API errors.

Document builder

Search quality is the document, not the model. Build one text blob per entity and keep the id stable.

Store document (searchText): name, description, cuisines, neighborhood, top dish names from the current menu, dietary flags. Rule-based tags (vegan, halal, …) — do not wait for a model to invent them.

Dish document: name, description, category, parent store name. Same dietary rules.

Stable dish ids are a write-path constraint. If the upstream catalog already has a durable uuid, use it. If the dish arrived from a scrape, hash storeId + normalizedName. Random UUIDs on every scrape mean you cannot UpdateItem and you cannot delete the stale SK. The index fills with ghosts.

Budget the document against the model's context, not against what you have. bge-small-en-v1.5 accepts 512 tokens; anything past that is truncated by the tokenizer with no error. A store blob of name + description + cuisines + neighborhood + every dish name will silently lose its tail. Order the blob by signal — name, cuisines, dietary flags, neighborhood, then as many dish names as fit — and cap it explicitly rather than letting truncation choose for you.

Write the document as-is. Do not prefix store or dish text. The query prefix is a retrieval trick, not a document trick. Use the same ONNX file, pooling, and prefix policy on backfill and on the live query. Mixing a quantized index with a full-precision query (or CLS with mean pooling) looks well-typed and ranks badly.

In-process embedder

Keep embedding inside the catalog service that already owns the write.

public float[] embedQuery(String text) {
  String prefixed = "Represent this sentence for searching relevant passages: " + text;
  return embed(prefixed); // tokenize → OrtSession → CLS → L2
}

public float[] embedDocument(String text) {
  return embed(text);
}

Budget 50–150 MB native for the session. On a 512 MB Fargate task that already runs Spring AOT plus JNI, watch RSS before you declare victory. The lever is -Xmx256m on that task, or 1024 MB on this service only — not a fleet-wide bump. Catalog embed is CPU on write; the live path embeds one query string.

No Python interpreter in the image. No HTTP round-trip to an embed microservice. The model file is a build artifact, same as a Flyway script.

DynamoDB shape

Vectors live on the items you already key by store.

Store item (existing PK / SK):

Dish items under the same partition:

PK = STORE#{storeId}
SK = DISH#{stableId}

Copy zonePk onto every dish. Same embedding fields, plus name, price, image, availability, storeName. entityType=DISH.

Query cache (optional, same table or a sibling):

PK = QCACHE#{normalizedQuery}
TTL = 7 days

Payload: parser JSON + query vector. Cache the raw string, not the corrected one — the expensive work is "what did the user actually type?"

User-taste table: PK = customerId, attributes tasteEmbedding, topCuisines, reorder maps, updatedAt. Rebuild on DELIVERED, not on every page view.

Vector index

Bump the AWS SDK for Java to 2.51.0 or later (vector indexes landed there). 2.49.x / 2.50.x do not have DynamoDbClient.searchVectors. Call it on the client you already use for GetItem / Query. Do not introduce a second Dynamo mapper for this.

Radius stays a haversine post-filter. The index answers "nearest in embedding space inside this zone," not "inside 3.0 miles." Inline filters cannot express a range, so meters cannot live in SearchConditionExpression. That has a recall cost: if the borough is much larger than radiusMiles, topK 100 may be spent on semantically close restaurants across the borough, then haversine drops most of them, and a closer restaurant that was 101st in embedding space never appears. Over-fetch (topK 100 is the API max) and accept the hole, or split the vector partition finer than the borough.

The table must be on-demand. That is a vector-index requirement, not a cost preference. The vector index is billed separately for data written, data stored, and data examined per search — so a hot downtown zone costs more per query than a quiet suburb even at the same topK.

QueryParser: slot-fill, then get out of the way

The model on the search path is not a chatbot. It is a 400 ms JSON completion that turns a messy string into slots the rest of the pipeline already understands.

Timeout 400 ms. Temperature 0. max_tokens ~150. Circuit-break the HTTP client. Cache on the raw string.

"cheep vegan sushi thats fast"

{
  "corrected": "cheap vegan sushi that's fast",
  "embedText": "cheap vegan sushi that's fast",
  "cuisines": ["sushi", "japanese"],
  "dietary": ["vegan"],
  "intent": "search",
  "filters": { "price": "low", "speed": "fast" }
}

Map cuisines / dietary onto the existing filter chips. Do not invent a second filter UI. Keep intent words in embedText — stripping "fast" so the embedding is "cleaner" throws away signal the dense model can use. Chips are for filters; the embedding is for meaning.

Fail open. Parser timeout, 5xx, or malformed JSON → embedText = raw query and continue. Dense models already tolerate some typos (piza often still sits near pizza). A 400 ms parser that 504s the whole search is worse than embedding the messy string. The parser is not on the embed path and must not sit on the API's critical timeout budget.

Keep a small HTTP JSON helper in the catalog API. Do not take a Maven dependency on the order service to call an LLM.

Search execution

SemanticSearchService replaces substring matching inside nearby search. The HTTP contract already has lat/lng/query. Add includeDishes (default true when a query is present).

1. Parser (cache / fail open)
2. embedQuery(embedText) (cache)
3. Intersecting zonePks from the existing geo helper
4. Parallel SearchVectors per zone: entityType=STORE, optional dineSafeStatus=PASS, topK 100. This is ANN, not exact k-NN — recall is high, not 100%
5. If a query is present: same call for DISH, group by storeId
6. Haversine drop outside radiusMiles (post-filter; see the recall hole above)
7. Union a cheap lexical candidate set (store name / cuisine contains on the zone GSI) so an exact name match cannot fall out of topK
8. Blend and cut
9. Return StoreSummaryDTO or StoreSearchGroup

DynamoDB COSINE scores are distances. Invert and normalize before you add anything, or the miles term dominates:

similarity = 1 - cosineDistance / 2          # 1 = identical, 0 = opposite
proximity  = 1 - min(dMiles, radius) / radius
ratingNorm = (rating - 1) / 4                # 1–5 stars → 0–1; skip if missing
score = 0.55 * similarity
      + 0.25 * proximity
      + 0.15 * ratingNorm
      + 0.05 * openNow                       # 1 or 0

Those weights are a starting point, not a measured ranking model. Tune them on a judged set (the golden queries below). Similarity without proximity will rank a great vegan sushi restaurant across town over the one 0.4 miles away. Proximity without similarity is the old nearby sort. The blend is the product.

Parser miss or ONNX miss: fall back to ZoneIndex + contains(). That fallback is a feature, not an apology. Feature-flag the semantic path so you can turn it off without a rollback.

Observability that operators can use

Golden queries to keep in the smoke/bench set: piza, pizza, margherita, cheap vegan sushi, hangover, plus a handful of real store names. Typos and dish names are the regressions substring search will never catch.

Index pipeline

Backfill is a script, not a request-path job:

1. Page stores
2. Build documents (store + current menu)
3. Skip if searchModel and searchText hash are unchanged
4. Embed, UpdateItem the store, write DISH# rows, delete stale SKs

Incremental rebuild on create, save, menu scrape, and POS sync: that store and its dish set only.

Never mix two models in one index. A model-file change bumps searchModel and triggers a full backfill. Cosine over mixed 384-d spaces — or CLS vs mean, or prefixed vs unprefixed queries against the same index — is well-typed garbage.

# operator check after backfill: every STORE item should share one model id
aws dynamodb scan \
  --table-name stores \
  --projection-expression "searchModel, entityType" \
  --filter-expression "entityType = :s" \
  --expression-attribute-values '{":s":{"S":"STORE"}}' \
  --max-items 50

Taste and reorder without a recommendation service

Order again is not semantic search. GET /orders/reorder-suggestions from the customer-orders index. Home rail links to /store/{id}. Hide the rail when the list is empty. This can ship in parallel with the embedder.

Taste is a weak personalization signal, not a recommender. On DELIVERED, take a recency-weighted mean of the dish embeddings (or embedDocument(itemName) if the DISH# row is not there yet), then L2-normalize the mean. An average of unit vectors is not unit length; DynamoDB COSINE will still run, but magnitude-sensitive downstream math and any later DOT_PRODUCT index will not.

Query the DISH index with that vector, then group by store. Do not SearchVectors(tasteEmbedding) against STORE items: store documents (name + neighborhood + many dishes) and dish documents live in different regions of the same 384-d space, and a dish centroid will not rank stores cleanly. Empty taste → existing featured endpoint. Do not block first-open on a cold taste row. A centroid of "pizza, ramen, salad" also washes toward the middle of the space — that is expected, not a bug in DynamoDB.

Cost and what stays out of scope

The expensive architecture is a dedicated vector database plus a dedicated embed container plus a NAT path so neither can sit in a public subnet. For a catalog that already lives in DynamoDB, that is idle spend. Vector search here is an index on items you were going to store anyway.

Out of scope on purpose: a second vector DB, a remote embed API, replacing the nearby discovery feed, and any reranker that adds a model call per candidate.

Scale: millions of rows, not millions on the request path

On a food-delivery catalog, “millions of rows” is almost never millions of restaurants. It is dishes. Ten thousand restaurants with eighty menu items is already ~800k vectors. A large metro is 1–5M DISH# items. The design has to be honest about that number.

The live path does not walk the table. ONNX embeds the query once. The parser is one bounded JSON call. SearchVectors is ANN, topK 100, one zonePk per call. Haversine and the rank blend run only on the hits that come back. Catalog size is not JVM heap size.

That is the opposite of loading every restaurant in the intersecting boroughs into memory and contains()-filtering them. That path is what runs out of memory when menus get large. Vector search is the scale-up from it.

The lever is the vector partition key

DynamoDB vector search is built to hold a very large index. The contract that makes latency and cost predictable is the vector index partition key. Each SearchVectors call searches one zonePk. A diner in one borough does not examine another city’s dishes.

Use a partition key with medium cardinality. Too high (store id) and each partition has no neighbors — recall dies. Too low (a boolean, or one “global” token) and you have a single hot shard. Borough / delivery zone is the same grain nearby already uses. That is why the index is partitioned on zonePk, not on a global catalog key.

AWS bills for data examined in that partition, not for the 100 rows you return. A downtown zone with two million dishes costs more per query than a suburb with twenty thousand, even when both return the same topK.

What to watch after the first backfill

SearchVectors topK maxes at 100. That is enough for nearby store and dish ranking. It is not “best 5,000 in the country” in one call. National search without a zone in the request is a different product.

When to split a zone — before you add a second database

If one zonePk is an order of magnitude larger than the next and p95 climbs with it, split the vector partition (borough → neighborhood or grid) without changing the model and without standing up Qdrant. Incremental writes (one store + its DISH# set on scrape or POS sync) stay cheap. A model-file change that rewrites every vector is an ops job, not something the request-path task does.

A dedicated vector database is the lever after a single zone is too hot and you still need one global ANN. For diner + radius + a handful of zones, keep the vectors on the catalog table.

Ship order

1. SDK bump + embedder + document builder + store attributes + store backfill
2. Vector index + SearchVectors in /stores/search + flag + mobile prefers API
3. QueryParser + caches + lexical candidate union
4. DISH# rows + dish SearchVectors + drop the client menu scan
5. Order-again rail (can parallel step 1)
6. Taste table + /stores/recommended

When an external vector database earns its place

Keeping vectors on the catalog table is the right call for geolocated food-delivery search. It is not a claim that dedicated vector databases — Qdrant, Weaviate, Milvus, pgvector, OpenSearch kNN, Pinecone — are the wrong tool. They exist because they are very good at things this design deliberately does not do. It is worth being honest about where they pull ahead.

Global, cross-zone search. The moment a query is not tied to a delivery area — "search every restaurant in the country," an internal catalog tool, analytics over the whole corpus — the per-zone assumption breaks. A dedicated engine is built to rank against the entire dataset in one query, which is exactly the case DynamoDB's per-partition model is avoiding.

Richer retrieval than approximate nearest neighbor. Hybrid search (BM25 + vectors) with learned weighting, multi-vector documents, filtered ANN over many attributes at once, and payload-heavy queries are first-class features in a purpose-built store. Reproducing them by hand on top of a general database gets expensive.

Reranking and ML-heavy relevance. If search becomes a product in its own right — cross-encoder rerankers, per-user model blends, online experiments on ranking — a specialized engine gives you the knobs and the throughput. Our design intentionally stops at a linear blend and a fallback.

Tuning the index itself. HNSW/IVF parameters, recall-vs-latency trade-offs, quantization strategies, and per-index sharding are exposed and adjustable. DynamoDB abstracts that away, which is a benefit until the day you actually need to tune it.

Decoupled scaling and write volume. When embeddings churn constantly, or a single zone genuinely holds tens of millions of vectors under heavy query load, isolating search into its own horizontally scaled system protects the operational database and lets each scale on its own curve.

The trade-off is real on the other side too: a second datastore to run, a synchronization pipeline to keep vectors consistent with the catalog, extra network hops, and usually a private-subnet/NAT cost footprint. For a scaling food-delivery platform whose searches are geolocated, that machinery buys little. The honest rule of thumb: reach for a dedicated vector database when your access pattern stops being local, when relevance outgrows a simple blend, or when one partition alone is too hot to serve — not before.

Conclusion

A diner on a food-delivery app is not searching a document corpus. They want a restaurant in range that can make what they typed — including the typo, the dish name that is not on the storefront, and the dietary constraint. That job already has a catalog table, a zone model, and a Spring Boot API on ECS. Semantic search should use those, not invent a second data plane.

The production pattern is small on purpose: embed with bge-small ONNX inside the catalog service, write 384-d vectors onto the same store and dish items, retrieve with DynamoDB SearchVectors partitioned by zonePk, and put a 400 ms fail-open parser in front for slots. The app renders the list. The API ranks. Order-again and taste ride orders you already store.

Scale is a zone problem, not a “millions of rows” problem. The request path embeds once and blends at most a hundred hits. If one borough gets hot, split that partition before you add a vector database.

Industry-standard marketplace search is not “we added a vector database.” It is one write path, one model version, a timeout budget on the parser, and a fallback that still returns nearby stores. Embed in-process. Index where the catalog already lives. Keep ranking on the server.