TL;DR
- Control calls at the shared provider/credential boundary. Ten workers each respecting a local limit can still exceed one account's quota.
- Keep short-window rate limits, daily allowances and paid credits separate. A 429 does not mean the account is absent, and a successful HTTP response does not mean the result is usable.
- Queue work with tenant fairness, deadline and retained progress. Reduce arrivals when capacity falls instead of creating an unlimited retry storm.
- Save accepted paid results before downstream CRM writes. Recovery should resume from those results rather than purchase the same enrichment again.
The quota belongs to a boundary
A pipeline can discover accounts, enrich contacts and update a CRM through several APIs. Each has its own account, endpoint, time window and allowance. Treat those limits as a contract to model, not a single number attached to the whole workflow.
Keep the provider quota key explicit: provider, credential/account, endpoint and any relevant tenant scope. A local sleep in each worker cannot coordinate other workers using the same credential. A shared limiter should decide when the next request is eligible.
The B2B data API guide covers the underlying purchase. Here the question is how to preserve useful throughput when those APIs constrain demand. Our cloud platform guide concerns hosting the surrounding system.
Do not mark rate-limited rows as “not found”. Preserve pending state and the next eligible attempt. Otherwise a capacity incident silently becomes a data-quality conclusion and can exclude valid accounts from the sales programme.
Provider and queue choices
Apollo: inspect endpoint and plan limits
Apollo's rate-limit reference describes per-minute, per-hour and per-day limits with 429 responses when exceeded. Its usage endpoint explains that plan affects endpoint limits. The current reference says these are per team and endpoint, shared by all keys and users; child workspaces share the parent workspace limits. Use the actual account's current limits rather than a copied generic rate.
This supports a pipeline that checks capacity before scheduling a backfill. Separate API request quota from the credit schedule. A plan can permit many requests while the intended enrichment consumes a limited paid allowance.
Choose the plan based on endpoint access, sustainable throughput and useful output. Avoid buying additional credits as a presumed fix for a rate limit unless the proposal explicitly changes capacity too. Save successful responses with provenance so later destination failures do not force another paid request.
HubSpot: avoid conflicting retry layers
HubSpot's official Node client documentation describes retry behaviour for certain server errors and rolling-limit responses. An SDK can therefore already be retrying while your outer job retries the same operation.
Choose one coordinated retry policy or understand how the layers combine. Preserve attempt counts and classify auth, validation, quota and temporary server failures differently. A revoked token should go to reconnection, not exponential backoff forever.
For CRM writes, confirm whether retrying the operation is safe. A create request that timed out after succeeding requires reconciliation or an idempotent application key. The integration should respect the account's current documented limits, not assume every HubSpot edition has identical capacity.
Amazon SQS: retained work rather than more provider capacity
Amazon SQS provides a managed queue, with pricing for API actions and related services. Its message quotas describe queue limits. A queue buffers work but does not increase the external vendor's quota.
This fits an AWS estate needing retained messages and decoupled workers. Price sends, receives, deletes, visibility changes, empty polling and payload/storage services. Do not treat one account row as one billable queue request.
Choose a managed queue when persistence and operations justify it. An existing database-backed job queue may suffice for a smaller workload with a capable owner. In either case, define retention, dead-letter handling and tenant fairness at the application layer rather than assuming a queue automatically makes the business process correct.
Design shared limits and client fairness
A token bucket is a useful model: requests consume capacity and the bucket refills over time. Add longer-window counters where the provider has hourly or daily limits. A minute bucket alone can exhaust the day's allowance before breakfast.
Use per-client queues or weighted scheduling behind the shared provider limit. A large import should not block every small customer update. Reserve capacity for time-sensitive work and make the weights visible to operations. Keep an expiry policy for stale triggers; an old event may no longer warrant a sales action when capacity returns.
Respect a documented Retry-After value where supplied. Otherwise use bounded exponential backoff with jitter for eligible transient failures. Cap attempts and elapsed time. Jitter spreads retries; it does not make an impossible throughput target feasible.
Record queue age, provider-specific pending count, accepted-result count and final destination completion. A falling 429 rate can mean the pipeline stopped doing useful work, so pair error metrics with throughput and age. The agent framework comparison helps place this control inside the broader orchestration design.
Recover from a quota reduction
Assume a hypothetical provider allows 600 requests per minute and the pipeline receives 480 jobs per minute, each requiring one call. Capacity initially exceeds demand by 120 per minute. The provider then reduces usable capacity to 300 for two hours. Backlog grows at 180 per minute, reaching 21,600 jobs.
If capacity returns to 600 while arrivals remain 480, clearing that backlog takes 180 minutes, or three hours. The recovery denominator is spare capacity, not total capacity. If every job requires two calls, the original arrival load is 960 calls per minute and the architecture is overloaded even before the reduction.
Introduce backpressure upstream. Pause a nonurgent import, reduce discovery intake or defer lower-priority refreshes. Tell the customer which work is delayed rather than presenting a partial result as complete. Retain paid enrichment outputs already accepted during the incident, then resume the CRM write stage separately.
If a daily quota is exhausted, minute-level retries cannot restore it. Wait for the documented reset or obtain authorised capacity. Keep a projection of remaining allowance against queued demand so an operator can make that decision before the queue ages beyond its business value.
Price useful throughput
Suppose 100,000 monthly jobs average 1.5 paid calls at an assumed $0.02 per call. Data cost is $3,000. If avoidable repeated enrichment adds 10%, that adds $300. Spending $100 on queue/storage and four operator hours at $60 per hour produces a hypothetical $3,340 base total, before avoidable repeats.
If 90,000 jobs satisfy the acceptance rule, base cost is about $0.037 per accepted job. These are declared assumptions, not provider prices or measured yield. Save idempotent stage results and compare their retention cost with repurchasing the data.
Use the ABM Account Scoring Template to define which jobs warrant fresh paid evidence and which can reuse an accepted observation. Avoid reducing cost by reusing stale data for a decision that requires current facts.
Choose capacity from sustained workload and deadline, with a queue/recovery design the team can operate. More worker concurrency is useful only until the bottleneck moves to the shared API. Beyond that point it buys contention and retries rather than throughput.
FAQ
Should every 429 be retried immediately?
No. Identify the limiting window and respect documented delay/reset behaviour. Immediate retries can worsen contention and consume resources without progress. Retain the row as pending rather than classifying it as missing data.
Does adding workers make the pipeline faster?
Only when worker capacity is the bottleneck. Once the shared provider quota is saturated, more workers cannot create allowance. Coordinate them through a shared limiter and prioritise the queue.
Can we cache all enrichment results forever?
No. Reuse depends on freshness requirements and licence terms. Retain accepted results with observation dates and a policy for refresh. Avoid repurchasing identical recent results while keeping stale fields out of decisions that need current evidence.
What belongs in a dead-letter queue?
Work that exceeded a bounded retry policy or needs a different intervention, with safe error details and stage progress. Auth and schema failures often need correction, not more retries. Assign an owner and disposition so dead-letter storage does not become an invisible backlog.
