Operations6 min
Jobs and projects
Organize work and safely pause, resume, stop, restart, or duplicate crawls.
Projects
A project is a persisted workspace for related jobs. Create one for a website, client, department, or research objective, then select it when creating a crawl. Deleting a project does not delete its jobs; those jobs become unassigned.
Job lifecycle
- Queued: saved and waiting for its worker to begin.
- Running: actively inspecting pages or downloading files.
- Paused: the worker stops at the next safe control checkpoint.
- Completed: the crawl reached its natural end or configured limits.
- Failed: an unrecoverable error ended the job; inspect its error and Runtime logs.
- Stopped: a user intentionally ended the job.
Controls
- Pause and resume preserve the live in-process crawler state.
- Stop ends the active run at its next safe checkpoint.
- Restart creates and immediately starts a new job from the original immutable configuration.
- Duplicate creates a queued copy that can be reviewed before it is started.
- Delete is allowed only after a job is no longer active.
Application restart behavior
In-process workers cannot survive a backend process restart. Jobs left queued, running, or paused are recovered as stopped so the dashboard never claims nonexistent work is still active.