Search
23 results for “evaluation”
Search results
LLM Evaluation & Reducing Hallucinations
Why "it feels better" isn't a shippable justification for a prompt change, what an LLM-graded eval actually measures, and why explicitly allowing "I don't know" is one of the most effective, cheapest hallucination fixes available.
Why is an evaluation suite necessary before shipping a prompt change, instead of just testing it manually on a few examples?
A prompt change can improve one success dimension while quietly breaking another, better accuracy but worse consistency, or improved tone at the cost of latency, and manually eyeballing a handful of examples won't reliably catch a regression on a dimension you weren't specifically looking at. An evaluation suite tests against a defined, ideally large set of cases, including deliberately hard edge cases like sarcasm or mixed sentiment, and scores every dimension that actually matters, which turns "it feels better" into a measurable, defensible, trackable claim, and catches regressions before they reach production rather than after.
What is an "LLM-graded" eval, and when would you reach for it instead of exact-match or similarity-based grading?
An LLM-graded eval uses a separate model call to judge a subjective quality of the output, tone, empathy, professionalism, on a numeric scale or binary classification, rather than checking it against one fixed correct answer. Exact-match grading only works when there's one right answer (a category label); similarity-based grading (cosine similarity between embeddings) works when wording can vary but meaning should match a reference. LLM-graded evals are the right tool specifically for qualities that are inherently subjective and hard to define with a fixed rule or reference string, at the cost of being noisier and more expensive to run than a simple string comparison.
Why does explicitly telling a model it's allowed to say "I don't know" measurably reduce hallucination, and what's a second, complementary technique for the same problem?
Without that explicit permission, a model under an implicit expectation to always produce a confident, complete answer will sometimes fill a real information gap with a plausible-sounding but fabricated one; stating outright that uncertainty is an acceptable answer removes that pressure and lets the model surface "I don't have enough information" instead of guessing. A complementary technique for long documents is asking the model to first extract direct, word-for-word quotes relevant to the task before generating any analysis, grounding its response in verifiable text it actually has in front of it, and then citing which quote supports each claim, rather than generating an answer freely and hoping it stayed faithful to the source.
Why can naive cache invalidation cause a race condition?
If a write invalidates (deletes) a cache key and then updates the database, a concurrent read between those two steps can repopulate the cache with the old value right after it was deleted, leaving stale data cached until the next TTL expiry or invalidation. Ordering matters: update the database first, then invalidate the cache, and even then, a very unlucky interleaving with a concurrent cache-aside read can still occur, which is why short TTLs are used as a safety net rather than relying on invalidation alone for strict consistency.
Retrieval-Augmented Generation (RAG)
Why a document chunk embedded in isolation loses the context that made it meaningful, and how prepending explanatory context to each chunk before embedding measurably cut retrieval failures in Anthropic's own published results.
A custom class defines __eq__ based on a value field but leaves __hash__ using default identity-based hashing. What actually goes wrong when you use an instance as a dict key?
In Python 3, simply defining `__eq__` without touching `__hash__` doesn't leave the old identity-based hash in place, Python automatically sets `__hash__` to `None` on that class, making instances unhashable, so using one as a dict key raises `TypeError` immediately rather than corrupting anything. This happens even if a parent class defines `__hash__`: overriding `__eq__` in a subclass sets that subclass's `__hash__` to `None` regardless of what the parent provides, ordinary inheritance does not carry the parent's hash forward. The silent, no-exception version of this bug only happens if the class explicitly keeps or re-supplies an identity-based `__hash__` alongside the value-based `__eq__` (e.g. `__hash__ = object.__hash__`, or, to retain a parent's hash on purpose, `__hash__ = Parent.__hash__`). In that case, two instances with the same value compare equal (`a == b` is `True`) but hash differently, and inserting under key `a` then looking up with an equal-but-distinct key `b` lands in the wrong hash bucket and returns nothing, because the hash mismatch never gave the lookup a chance to even check equality against the right entry.
AWS IAM
How IAM users, roles, and policies control access in AWS, why roles with temporary credentials are preferred over long-lived access keys, and how policy evaluation actually decides allow versus deny.
How does IAM decide whether a request is allowed, and what wins if there is a conflict between an Allow and a Deny?
IAM evaluation starts from an implicit deny, nothing is allowed unless some applicable policy explicitly allows it. It then checks every applicable policy for an explicit Deny; if one exists anywhere, identity-based, resource-based, a permission boundary, an SCP, or an RCP, the request is denied regardless of how many Allow statements also apply. Beyond that, identity-based and resource-based policies are the only policy types that actually grant permissions, an Allow from either is required. Permission boundaries, SCPs, and RCPs never grant anything themselves, they only cap what a grant can reach, so the request also needs every applicable one of those to independently allow the action; any one of them failing to allow it denies the request even with a valid Allow in place. An explicit Deny always wins over an explicit Allow, no matter which policy or how specific the Allow is.
How does a Network Security Group (NSG) evaluate traffic rules?
An NSG contains a prioritized list of allow/deny rules, evaluated in priority order (lowest number first) until the first rule matching the traffic's source, destination, port, and protocol is found, that rule's action wins, and no further rules are evaluated. NSGs can attach to a subnet, a network interface, or both, and are stateful, meaning an allowed inbound connection's return traffic is automatically permitted without needing a matching outbound rule. Because evaluation stops at first match, rule priority ordering is the actual logic, a broad allow rule placed before a specific deny rule silently makes that deny unreachable.
Why does wrapping several statements in `BEGIN`/`COMMIT` matter, even for a multi-step operation that's logically one action?
Without an explicit transaction, PostgreSQL commits each statement independently the moment it succeeds; if a multi-step operation (debit one account, credit another) fails halfway through, the database is left in an inconsistent, partially-applied state with no way to undo the completed step. Wrapping the statements in `BEGIN`/`COMMIT` groups them so they can be committed or rolled back together, but that alone isn't automatic protection against every failure mode: PostgreSQL only aborts a transaction on its own when a statement raises an actual SQL error, an `UPDATE` that runs successfully but matches zero rows (a mistyped account id, for instance) is not an error at all, and would otherwise be committed as if the transfer had actually happened. Making the transaction genuinely safe means checking that each `UPDATE` affected exactly the one row expected and issuing an explicit `ROLLBACK` if either check fails, only issuing `COMMIT` once both checks pass.
AI Engineer Roadmap
From LLM fundamentals and prompt engineering through embeddings, retrieval-augmented generation, tool use, and evaluation, the core path for building real LLM-powered applications, linked into Cloud Tech by Victor topic references.
Why does a real ring hash implementation give each host many positions on the ring instead of just one, and what problem would a single position per host cause?
A host thrown onto the ring at just one point can end up, purely by chance, owning a disproportionately large or small arc of the ring if the hash values happen to land unevenly, since with few points there's no averaging effect smoothing out the randomness. Assigning each host many positions on the ring, scaled by that host's intended weight, so a double-weight host gets roughly twice as many ring entries as a single-weight one, averages out that randomness across many smaller arcs per host, producing a much more even overall traffic distribution than a single coin-flip-like placement per host would.
When does semantic search (via embeddings) actually outperform traditional keyword search, and when might keyword search still win?
Semantic search wins when a query and the relevant document use different words for the same idea, "car won't start" matching a document about "vehicle fails to ignite", since embeddings compare meaning rather than literal tokens, which keyword search cannot do at all. Keyword search still wins, or at least remains necessary, for exact-match needs, an error code, a product SKU, a specific proper noun, where the literal string matters and a semantically "close" but textually different result is actually the wrong answer. This is why production search systems commonly combine both (hybrid search) rather than treating embeddings as a strict replacement for keyword matching.
A socket shows up in the LISTEN state versus ESTABLISHED. What is actually different about what that socket is doing?
LISTEN means a server-side socket is bound to a port and passively waiting to accept incoming connection requests, it isn't attached to any specific remote peer yet. ESTABLISHED means a full connection has completed its handshake and is actively associated with a specific remote address and port, data can flow in both directions. Seeing a service you expect to be running not show a LISTEN socket on its expected port is usually the very first, most direct signal that the service isn't actually up or isn't bound where you think it is.
What does "at-least-once delivery" actually guarantee, and what does it not guarantee, and why does that mean a consumer must be idempotent?
At-least-once delivery guarantees a message will eventually be delivered and processed at least one time, it does not guarantee exactly one delivery, the same message can legitimately be delivered and processed more than once, for example if a consumer's deletion request is lost after it already finished processing, or a visibility timeout expires just as processing completes. Because duplicate delivery is a normal, expected outcome of this model rather than a rare edge case, a consumer has to be written so that processing the same message twice produces the same end result as processing it once (an idempotent operation), rather than assuming the queue itself will prevent duplicates.
PostgreSQL's documentation says it's "impossible to suppress nested-loop joins entirely" even with enable_nestloop off. What does that tell you about relying on planner hints to force a specific join algorithm?
That specific guarantee is documented only for `enable_nestloop`: turning it off only discourages the planner by making nested loops look artificially expensive in cost estimation, it can't hard-disable them, because for some queries a nested loop is the only viable plan at all (for example, certain correlated subquery shapes), so the planner will still use one if it must. `enable_hashjoin` and `enable_mergejoin` don't carry that same caveat, disabling either one actually can prevent the planner from choosing that join type, since a nested loop (or the other remaining method) is always available as a fallback plan. In practice, though, all three settings are best treated as a debugging/diagnostic tool for understanding planner behavior, not a reliable production mechanism for forcing a specific join algorithm, the actual fix for a bad plan is almost always better statistics (via `ANALYZE`) or a better index, not overriding the planner's method choice.
DHCP Server Deployment on Hyper-V: From Scope Creation to Failover Configuration
Step-by-step deployment of a highly available DHCP service in a Hyper-V virtual environment
Why does effective incident investigation in Azure usually require writing KQL rather than relying only on dashboards?
Dashboards are built in advance for questions someone anticipated asking; they're monitoring, not investigation. A real incident usually raises a question nobody built a dashboard for ("which specific requests failed, and what did they have in common?"), and answering that requires querying the underlying log data directly. KQL is Log Analytics' query language, and being able to write ad hoc queries, joining traces, filtering by custom dimensions, correlating across data types, is what turns "I can see something is wrong" into "I know why," which is the actual goal of observability, not just monitoring.
What does contextual retrieval actually change about the chunking process, and what measurable difference did it make?
Instead of embedding each chunk as extracted, contextual retrieval prepends a short, chunk-specific explanatory context before embedding it, turning "The company's revenue grew by 3%..." into something like "This chunk is from an SEC filing on ACME Corp's performance in Q2 2023; the company's revenue grew by 3%...", generated automatically per chunk rather than written by hand. In Anthropic's published results, this reduced retrieval failures by 49%, and combining it with reranking (a second relevance-scoring pass over retrieved candidates) reduced failures by 67%, a substantial, measured improvement from addressing the isolated-chunk context problem directly rather than only tuning the embedding model or retrieval algorithm.
Why would you choose session pooling over transaction pooling even though it scales to fewer concurrent clients per server connection?
Session pooling is the only mode of the two that supports every PostgreSQL feature without exception, prepared statements, session variables set via SET, LISTEN/NOTIFY, session-level advisory locks, because the server connection genuinely stays with the client for as long as their session lasts. If an application depends on any of those session-level features and can't be refactored around them, session pooling is the correct choice despite its lower connection-reuse efficiency, trading raw scalability for full feature compatibility rather than working around broken session state.
A banking system typically favors CP behavior during a partition, while a chat application typically favors AP. What does each system actually do differently when a partition occurs, and why does the choice fit each use case?
A CP system, during a partition, pauses or rejects requests that can't be guaranteed consistent, a bank stopping a transfer rather than risking two nodes independently approving withdrawals against the same balance, since a duplicated or lost transaction is a correctness failure worse than a temporary outage. An AP system keeps responding during the partition, accepting the risk of temporarily inconsistent state, a chat app still accepting and displaying messages on both sides of a network split, reconciling them once the partition heals, because staying available and eventually consistent matters more to users than every message reappearing everywhere in a strict, immediate order.
A page has a fast average LCP but users still frequently report the page feeling slow. What could the percentile-based measurement reveal that an average wouldn't?
If a substantial slice of real page loads, say the slowest 25%, badly miss the 2.5 second LCP threshold (a slow connection, a busy device, a cold cache), the average can still look fine because it's dominated by the faster majority, while a real, sizable group of users are having a genuinely bad experience the average is actively hiding. Checking the 75th-percentile value directly (rather than the mean) surfaces that gap: if the 75th percentile is well above 2.5 seconds even though the average looks fine, that's a concrete signal that meaningful numbers of real users are missing the threshold, not a false alarm.