Search
8 results for “hallucinations”
Search results
LLM Evaluation & Reducing Hallucinations
Why "it feels better" isn't a shippable justification for a prompt change, what an LLM-graded eval actually measures, and why explicitly allowing "I don't know" is one of the most effective, cheapest hallucination fixes available.
Why is an evaluation suite necessary before shipping a prompt change, instead of just testing it manually on a few examples?
A prompt change can improve one success dimension while quietly breaking another, better accuracy but worse consistency, or improved tone at the cost of latency, and manually eyeballing a handful of examples won't reliably catch a regression on a dimension you weren't specifically looking at. An evaluation suite tests against a defined, ideally large set of cases, including deliberately hard edge cases like sarcasm or mixed sentiment, and scores every dimension that actually matters, which turns "it feels better" into a measurable, defensible, trackable claim, and catches regressions before they reach production rather than after.
What is an "LLM-graded" eval, and when would you reach for it instead of exact-match or similarity-based grading?
An LLM-graded eval uses a separate model call to judge a subjective quality of the output, tone, empathy, professionalism, on a numeric scale or binary classification, rather than checking it against one fixed correct answer. Exact-match grading only works when there's one right answer (a category label); similarity-based grading (cosine similarity between embeddings) works when wording can vary but meaning should match a reference. LLM-graded evals are the right tool specifically for qualities that are inherently subjective and hard to define with a fixed rule or reference string, at the cost of being noisier and more expensive to run than a simple string comparison.
Why does explicitly telling a model it's allowed to say "I don't know" measurably reduce hallucination, and what's a second, complementary technique for the same problem?
Without that explicit permission, a model under an implicit expectation to always produce a confident, complete answer will sometimes fill a real information gap with a plausible-sounding but fabricated one; stating outright that uncertainty is an acceptable answer removes that pressure and lets the model surface "I don't have enough information" instead of guessing. A complementary technique for long documents is asking the model to first extract direct, word-for-word quotes relevant to the task before generating any analysis, grounding its response in verifiable text it actually has in front of it, and then citing which quote supports each claim, rather than generating an answer freely and hoping it stayed faithful to the source.
Why does Docker's official build guidance recommend running as a non-root user inside a container, and why not just use sudo when root is needed?
Running as root inside a container means that a successful application-level compromise (a code-execution vulnerability, an unsafely deserialized payload) hands the attacker root inside that container immediately, with no privilege-escalation step required, and depending on the container runtime's configuration, root inside a container can sometimes be leveraged toward the host. Creating a dedicated non-root user via `USER` in the Dockerfile means a compromise still has to escalate privileges to do serious damage. `sudo` is specifically discouraged in Docker's own guidance because of its unpredictable TTY and signal-forwarding behavior inside a container, not because privilege separation itself is unnecessary.
PostgreSQL detects a deadlock between two transactions. What does it actually do, and can you predict which transaction survives?
PostgreSQL automatically detects the circular wait (a deadlock) and resolves it by aborting one of the involved transactions, letting the other(s) proceed. The documentation is explicit that which transaction gets aborted is difficult to predict and shouldn't be relied upon, there is no guarantee it's the "smaller" transaction, the one that started the wait, or any other predictable rule. Applications need to handle a deadlock-abort the same way they'd handle a serialization failure: catch it and retry the aborted transaction, rather than assuming a specific transaction will always be the one sacrificed.
Why does using a monotonically increasing field (a sequential ID, a timestamp) as a shard key concentrate all new writes onto one shard, even with range-based sharding across many shards?
With range-based sharding, chunks are assigned contiguous ranges of shard-key values, and a monotonically increasing key means every new document's value is higher than every previously inserted one, so all new inserts land in whatever chunk currently owns the highest range, which lives on one specific shard. Every other shard, holding older, lower-valued ranges, receives none of the new write traffic at all, the exact "hot shard" problem, all insert load concentrated on a single shard regardless of how many total shards the cluster has.
Why does calling a blocking function inside an async function defeat the purpose of using asyncio?
A blocking call (synchronous file I/O, a synchronous HTTP request, `time.sleep`) occupies the single thread the event loop runs on, and unlike `await`, it does not yield control back, the entire event loop is frozen for the duration of that blocking call, so every other coroutine that could otherwise be making progress is stalled too. This is why async code requires async-compatible libraries throughout the I/O path; a single accidental blocking call anywhere in a hot path can silently serialize what was supposed to be concurrent work, and the bug often doesn't show up until real concurrent load exposes it.