Nina Classify the failure before responding
Different failures need different actions. A browser crash should not trigger the same response as an expired session or a target-side rate limit. Put each failure into a small, useful set of categories.
- Infrastructure. Browser crashes, worker exits, proxy loss, disk or memory pressure, and network timeouts.
- Identity and session. Expired tokens, invalid cookies, changed device requirements, and new challenge types.
- Target behavior. Rate limits, server errors, changed markup, new payloads, and unexpected workflow states.
- Workflow logic. Invalid data, missing elements, and attempts to repeat a side effect that should happen only once.
Use a specific recovery action for each category
A crashed worker can be replaced and the task resumed from a checkpoint. An expired session needs reauthentication, not a new browser with the same expired state. A proxy failure may need a new route, while a target-side rate limit needs bounded backoff.
Unknown challenge changes should pause the affected work and create an actionable alert. Retrying them indefinitely can increase scoring, consume capacity, and hide the real problem.
Make side effects explicit
Read-only extraction and an action such as submitting an order cannot share the same retry policy. Every step needs to be understood as idempotent, resumable, or unsafe to repeat.
- Assign a task ID and persist checkpoints at meaningful workflow boundaries.
- Store enough state to resume, but do not replay completed side effects.
- Use idempotency keys when the target supports them.
- Send uncertain or non-idempotent outcomes to review instead of guessing.
Design the queue for recovery
A queue should represent the state of the task, not only a list of jobs. Visibility timeouts need to exceed normal processing time, and a task should not become visible to another worker while the first worker is still making progress.
Repeatedly failing tasks need a dead-letter path with the target, checkpoint, failure category, and recent error history attached. That makes the next action clear to an operator or another automated process.
Monitor signals that explain recovery
Overall success rate is not enough. Useful signals include success by target and workflow, challenge frequency, session age, authentication failures, worker crashes, proxy failures, queue depth, retry count, and time to recovery.
These signals help distinguish a target change from a capacity problem. They also show whether recovery is reducing manual work or simply moving failures into a larger retry queue.
Test recovery before you need it
Crash workers during real workflows, expire sessions, remove proxies, return target errors, and change challenge responses. Check both the technical recovery and the final task state. A workflow that retries successfully but performs an action twice is still a failure.