Search
30 results for “sqs”
Search results
Message Queues & Event-Driven Architecture
Why a visibility timeout, not a delete, is what actually protects a message from being processed twice, and why "at-least-once delivery" means your consumer has to handle duplicates even when everything is configured correctly.
What is a visibility timeout, and what problem does it solve that simply having multiple consumers poll a queue would create?
When a consumer receives a message, the queue doesn't delete it immediately, it starts a visibility timeout during which that message is hidden from other consumers, so a second consumer polling the same queue won't also pick up and process the same message concurrently. The consumer is expected to finish processing and explicitly delete the message before the timeout expires; if it does, the message never reappears. Without this mechanism, multiple consumers competing for messages would routinely double-process the same one, exactly the race condition visibility timeouts exist to prevent.
A consumer receives a message, starts processing, but crashes before deleting it. What happens to that message, and why is this actually the desired behavior?
Once the visibility timeout expires without the message being deleted, it automatically becomes visible again in the queue and can be picked up by the same or a different consumer for another processing attempt. This is deliberate, not a bug: the alternative (a crashed consumer's message vanishing permanently) would silently lose data, whereas reappearing after timeout guarantees eventual processing at the cost of a possible duplicate attempt, which is exactly the trade-off "at-least-once delivery" describes. Setting the visibility timeout too short causes premature, unnecessary reprocessing of messages still legitimately being worked on; too long delays legitimate retries after a real crash.
What does "at-least-once delivery" actually guarantee, and what does it not guarantee, and why does that mean a consumer must be idempotent?
At-least-once delivery guarantees a message will eventually be delivered and processed at least one time, it does not guarantee exactly one delivery, the same message can legitimately be delivered and processed more than once, for example if a consumer's deletion request is lost after it already finished processing, or a visibility timeout expires just as processing completes. Because duplicate delivery is a normal, expected outcome of this model rather than a rare edge case, a consumer has to be written so that processing the same message twice produces the same end result as processing it once (an idempotent operation), rather than assuming the queue itself will prevent duplicates.
Why should variables almost always be quoted, e.g. `"$name"` instead of `$name`?
An unquoted variable expansion undergoes word-splitting (on whitespace) and globbing (on `*`, `?`, etc.) before the command sees it, so a value containing a space or a shell metacharacter silently becomes multiple arguments or an unintended file-glob expansion instead of one literal string. Quoting (`"$name"`) suppresses both, so the variable's value is always passed through as exactly one argument, which is why nearly every Bash style guide treats an unquoted variable expansion as a latent bug rather than a style preference.
What's the practical difference between Repeatable Read and Serializable, given that PostgreSQL's Repeatable Read already prevents phantom reads?
PostgreSQL's Repeatable Read goes beyond the SQL standard's minimum and already prevents phantom reads via snapshot isolation, but it can still allow a specific class of anomaly called a serialization anomaly, where the combined effect of several concurrently-committed transactions is not equivalent to any possible serial (one-at-a-time) ordering of them, even though each transaction individually looks consistent. Serializable adds predicate locking on top of snapshot isolation specifically to detect and prevent that remaining anomaly, guaranteeing that the outcome is always equivalent to transactions having run one at a time in some order. The cost is the same as Repeatable Read's, more serialization failures the application must retry, in exchange for the strongest correctness guarantee available.
What is the difference between a parameterized query and simply concatenating a sanitized string?
A parameterized query sends the SQL command and the user-supplied values as two separate things to the database driver, the driver (or the database itself, for prepared statements) binds the values into the query plan without ever treating them as part of the SQL text. String concatenation, even "sanitized," still builds one text string where user input and SQL syntax share the same channel, any gap in the sanitization logic can be exploited. Parameterization removes that shared channel entirely rather than trying to police it.
SQL Joins & Query Planning
Why the same JOIN can execute as a nested loop, a hash join, or a merge join depending entirely on table size and available sort order, and why "it's impossible to suppress nested loops entirely" is a real, documented planner constraint.
Can an ORM fully prevent SQL injection on its own?
An ORM prevents injection for the queries it builds using its own query API, because those are parameterized under the hood. It does not protect against injection if raw SQL is still used, for example, string-interpolating a value into a raw query method, or building dynamic column/table names from user input (which parameterization cannot help with, since identifiers cannot be bound as parameters and need allowlisting instead). ORMs reduce the attack surface; they do not eliminate the need to think about it.
Why does binary search require the input to already be sorted, and what actually happens if you run it on unsorted data?
Binary search's core logic is comparing the target against a midpoint and eliminating the entire half that can't possibly contain it, an elimination step that's only valid if elements are ordered, so everything below the midpoint really is smaller and everything above really is larger. Run it on unsorted data and there's no error or exception, the algorithm has no way to detect the invariant is broken, it just keeps halving the search space based on comparisons that no longer imply anything real, and returns an insertion point or "not found" result that has no actual relationship to whether or where the target exists in the list.
Why can't partition pruning work with a WHERE clause like WHERE logdate >= CURRENT_TIMESTAMP, and what does that tell you about writing partition-friendly queries?
Partition pruning at plan time requires the comparison value to be known and fixed when the plan is built, and `CURRENT_TIMESTAMP` is not immutable, its value depends on when the query actually executes, not when it's planned, so the planner can't statically prove which partitions it will or won't match. (Pruning can still happen at execution time for genuinely parameterized values, like a join parameter from an outer query, just not for volatile functions like this one.) This means writing partition-friendly queries means filtering on the partition key with values the planner can actually reason about, a literal, a bound parameter, or an immutable expression, not a function whose result varies by when the query runs.
What is PostgreSQL's default transaction isolation level, and what specific anomaly does it still allow that a stricter level would prevent?
PostgreSQL defaults to Read Committed, where each individual statement within a transaction sees a fresh snapshot of everything committed as of that statement's start, not the transaction's start. This prevents dirty reads (seeing another transaction's uncommitted changes) but still allows non-repeatable reads, running the same SELECT twice in one transaction can return different results if another transaction committed a change in between, because each statement gets its own snapshot rather than the transaction using one snapshot throughout. Repeatable Read fixes this specific anomaly by taking one snapshot at the start of the transaction and using it for every statement within it.
Why does Python's documentation say a class defining mutable objects with a custom __eq__ should not implement __hash__ at all?
If an object's hash is derived from fields that can change after the object is already stored as a dict key, mutating it changes its hash value, but the object stays in whatever bucket it was originally placed in based on the old hash, so a subsequent lookup computes the new hash, looks in the new (wrong) bucket, and fails to find an object that is, in fact, still in the dictionary. Making a mutable object unhashable by default (which not defining `__hash__` effectively signals) prevents this specific class of bug entirely, at the cost of not being able to use that object as a dict key or set member at all, a deliberate, documented trade-off favoring correctness over convenience.
A client sends a payment request, the network times out before a response arrives, and the client retries with the same idempotency key. What actually happens on the server?
If the original request already completed (successfully or even with an error like a 500) before the retry arrives, the server returns the exact same result it returned (or would have returned) the first time, the same status code and response body, without re-executing the underlying operation, so the customer isn't charged twice just because the client never saw the first response. This is the entire point of an idempotency key: it lets a client safely retry an operation whose actual outcome it's genuinely uncertain about (did the first request even reach the server? did it complete before the timeout?) without that retry risking a duplicate side effect.
Why does explicitly telling a model it's allowed to say "I don't know" measurably reduce hallucination, and what's a second, complementary technique for the same problem?
Without that explicit permission, a model under an implicit expectation to always produce a confident, complete answer will sometimes fill a real information gap with a plausible-sounding but fabricated one; stating outright that uncertainty is an acceptable answer removes that pressure and lets the model surface "I don't have enough information" instead of guessing. A complementary technique for long documents is asking the model to first extract direct, word-for-word quotes relevant to the task before generating any analysis, grounding its response in verifiable text it actually has in front of it, and then citing which quote supports each claim, rather than generating an answer freely and hoping it stayed faithful to the source.
What distinct question does each of logs, metrics, and traces answer?
Metrics answer "what is happening, in aggregate, over time", cheap to store, good for dashboards and alerting thresholds, but they lose individual event detail. Logs answer "what exactly happened in this specific event", full detail but expensive to store and search at scale. Traces answer "where did time go across this one request as it moved through multiple services"; they reconstruct causality and latency across service boundaries that neither logs nor metrics show on their own. A mature observability setup uses all three together, correlated by shared identifiers like a request or trace ID.
A client sends a sudden spike of requests that exceeds the configured rate, but the API doesn't immediately reject any of them. Why not, and when does rejection actually start?
The token bucket has accumulated capacity up to the burst limit, if the client had been under the rate limit recently, unused tokens built up in the bucket, and that reserve absorbs a short spike without any request failing, exactly the point of separating burst capacity from steady-state rate. Rejection (a 429 response) only starts once the bucket is actually empty, every accumulated token has been consumed and the spike is sustained long enough that the rate of token consumption keeps exceeding the rate of token replenishment. This is why a token bucket, unlike a hard per-second cap, tolerates brief bursts gracefully while still enforcing a real steady-state ceiling.
What problem does GraphQL solve that REST does not, structurally?
Over-fetching and under-fetching. A REST endpoint returns a fixed shape, so a mobile client that only needs a user's name either gets the whole user object (over-fetching) or the API team has to add a new endpoint or query params for every new shape a client needs (under-fetching, solved by proliferating endpoints). GraphQL lets the client specify exactly which fields it wants in the query itself, so one flexible schema serves many different client needs without new endpoints.
Why does a chunk like "The company's revenue grew by 3% over the previous quarter" cause a retrieval failure even if it's embedded correctly?
That sentence is only meaningful with context the isolated chunk doesn't carry, which company, which quarter, information that likely lived in a heading or preceding paragraph that didn't survive the chunking boundary. Embedded on its own, the chunk's vector represents a generic, context-free statement about revenue growth, which means it won't be retrieved reliably for a query about "ACME Corp Q2 2023 earnings" even though it's exactly the right chunk, because nothing in its embedding actually encodes that it's about ACME Corp or Q2 2023 at all.
What are the three join algorithms PostgreSQL's planner can choose between, and what does each one actually do?
A nested loop join takes each row from one table (the outer side) and scans the other table (the inner side) for matches, the simplest algorithm, and cheap specifically when the inner side is small or has a usable index so that scan is fast per outer row. A hash join builds an in-memory hash table from one input keyed on the join column, then probes it with rows from the other input, efficient for large inputs with no useful sort order or index. A merge join requires both inputs already sorted on the join key (or sorts them first) and then walks both sorted streams in lockstep, efficient specifically when that sorted order already exists or is useful for something else in the query anyway.
PostgreSQL's documentation says it's "impossible to suppress nested-loop joins entirely" even with enable_nestloop off. What does that tell you about relying on planner hints to force a specific join algorithm?
That specific guarantee is documented only for `enable_nestloop`: turning it off only discourages the planner by making nested loops look artificially expensive in cost estimation, it can't hard-disable them, because for some queries a nested loop is the only viable plan at all (for example, certain correlated subquery shapes), so the planner will still use one if it must. `enable_hashjoin` and `enable_mergejoin` don't carry that same caveat, disabling either one actually can prevent the planner from choosing that join type, since a nested loop (or the other remaining method) is always available as a fallback plan. In practice, though, all three settings are best treated as a debugging/diagnostic tool for understanding planner behavior, not a reliable production mechanism for forcing a specific join algorithm, the actual fix for a bad plan is almost always better statistics (via `ANALYZE`) or a better index, not overriding the planner's method choice.
If a microtask itself queues another microtask while it's running, does that new microtask wait for the next event loop iteration, or does it still run before the next macrotask?
It still runs before the next macrotask. The rule isn't "run the microtasks that were queued when this iteration started", it's "drain the microtask queue completely," and if executing a microtask adds another one, that new one is still part of the queue being drained. This means a chain of microtasks that keep scheduling more microtasks can, in principle, starve the event loop from ever reaching the next macrotask (a real, documented way to accidentally block timers and rendering), which is different from a single flat batch of microtasks all queued up front.
A page has a fast average LCP but users still frequently report the page feeling slow. What could the percentile-based measurement reveal that an average wouldn't?
If a substantial slice of real page loads, say the slowest 25%, badly miss the 2.5 second LCP threshold (a slow connection, a busy device, a cold cache), the average can still look fine because it's dominated by the faster majority, while a real, sizable group of users are having a genuinely bad experience the average is actively hiding. Checking the 75th-percentile value directly (rather than the mean) surfaces that gap: if the 75th percentile is well above 2.5 seconds even though the average looks fine, that's a concrete signal that meaningful numbers of real users are missing the threshold, not a false alarm.
PostgreSQL Fundamentals
How tables, joins, aggregates, window functions, and transactions fit together in everyday PostgreSQL work, from the actual internals of a single index (covered in PostgreSQL Indexes) to the SQL you write day to day.
PostgreSQL Indexes
How B-Tree, Hash, GIN, BRIN, and covering indexes work in PostgreSQL, when to create them, how to inspect usage, and the write-overhead trade-offs at scale.
How does IAM decide whether a request is allowed, and what wins if there is a conflict between an Allow and a Deny?
IAM evaluation starts from an implicit deny, nothing is allowed unless some applicable policy explicitly allows it. It then checks every applicable policy for an explicit Deny; if one exists anywhere, identity-based, resource-based, a permission boundary, an SCP, or an RCP, the request is denied regardless of how many Allow statements also apply. Beyond that, identity-based and resource-based policies are the only policy types that actually grant permissions, an Allow from either is required. Permission boundaries, SCPs, and RCPs never grant anything themselves, they only cap what a grant can reach, so the request also needs every applicable one of those to independently allow the action; any one of them failing to allow it denies the request even with a valid Allow in place. An explicit Deny always wins over an explicit Allow, no matter which policy or how specific the Allow is.
Why does effective incident investigation in Azure usually require writing KQL rather than relying only on dashboards?
Dashboards are built in advance for questions someone anticipated asking; they're monitoring, not investigation. A real incident usually raises a question nobody built a dashboard for ("which specific requests failed, and what did they have in common?"), and answering that requires querying the underlying log data directly. KQL is Log Analytics' query language, and being able to write ad hoc queries, joining traces, filtering by custom dimensions, correlating across data types, is what turns "I can see something is wrong" into "I know why," which is the actual goal of observability, not just monitoring.
Why might a team deliberately choose IaaS over PaaS even though PaaS requires less operational work?
PaaS trades control for convenience; it constrains you to whatever runtimes, configurations, and scaling behavior the platform supports. Teams choose IaaS when they need capabilities a PaaS doesn't expose (custom OS-level tuning, unusual networking topologies, specific compliance requirements that mandate control over the underlying host), when they're running workloads a PaaS wasn't designed for, or when the cost model of many small PaaS instances doesn't make sense at their scale compared to self-managed infrastructure.
An application uses PREPARE to create a reusable prepared statement, then relies on it across multiple requests. What happens if it's deployed behind a transaction-pooled PgBouncer, and why?
It breaks. A SQL PREPARE statement is a session-level feature, it lives on whatever specific server connection issued the PREPARE, but transaction pooling reassigns the underlying server connection to a different client (or the same client's next transaction) as soon as each transaction ends, so there's no guarantee a later request lands on that same server connection where the prepared statement actually exists. This is explicitly documented as one of the session-based features transaction pooling breaks, along with SET/RESET, LISTEN, WITH HOLD cursors, and session-level advisory locks, all of which depend on state tied to one specific, persistent server connection. This is distinct from protocol-level named prepared statements issued via the extended query protocol (what most driver-level "prepared statements" actually are), which PgBouncer can support under transaction pooling when `max_prepared_statements` is set to a non-zero value.
PostgreSQL detects a deadlock between two transactions. What does it actually do, and can you predict which transaction survives?
PostgreSQL automatically detects the circular wait (a deadlock) and resolves it by aborting one of the involved transactions, letting the other(s) proceed. The documentation is explicit that which transaction gets aborted is difficult to predict and shouldn't be relied upon, there is no guarantee it's the "smaller" transaction, the one that started the wait, or any other predictable rule. Applications need to handle a deadlock-abort the same way they'd handle a serialization failure: catch it and retry the aborted transaction, rather than assuming a specific transaction will always be the one sacrificed.