Skip to content

Architecture Overview

saneless bridges SANE-compatible network scanners and paperless-ngx. It provides a web UI and CLI for triggering scans, assembling multi-page PDFs, and ingesting documents into paperless-ngx with metadata -- all from any device on the local network.

This page explains how saneless is structured and why it works the way it does.

The Scan Pipeline

Every scan job follows the same flow, whether triggered from the web UI or the CLI:

Scanner -> [scan child] -> [SaneBackend] -> [Page Spool] -> spooled PNGs + ordered page records
        -> [Empty Page Filter] -> [img2pdf + qpdf] -> PDF -> [PaperlessClient] -> paperless-ngx
                                                          \-> [Consume Dir] (fallback)

Scanner -> [listing child] -> [SaneBackend] -> scanner list, open check, capabilities

Both children are separate processes: every SANE call runs in one of them, never in saneless's own process.

Scanner Abstraction

The ScannerBackend abstract base class isolates SANE behind a clean interface with three methods: get_devices() for discovery, get_capabilities() for querying available options, and scan_pages() for acquiring images. The SaneBackend implementation never loads python-sane itself: it runs every SANE call in a child process. A listing, an open check or a capability read each get a short-lived child, and a scan job gets one child for the whole job, which opens the device, sets its options, reads each sheet and sends the pages back (see Memory, disk and timeouts and Worker Thread Model).

This abstraction means the core pipeline never touches SANE directly, which is what lets the tests drive the whole pipeline with a stub backend and no scanner. Driverless network scanners need no backend of their own: they are reached through SANE like any other. The container image enables two SANE backends, net for a scanner shared by a saned host and escl for an eSCL (AirScan) scanner reached directly; it does not include sane-airscan.

Pipeline Orchestration

The run_pipeline() function coordinates the full scan flow:

  1. Resolve scanner -- find the configured device or auto-detect the first available one.
  2. Set parameters -- apply the selected profile's source, resolution, and color mode.

  3. Acquire pages -- scan via the scanner backend. Each page is written to the job's page spool as it arrives, as a PNG file with an ordered page record, rather than accumulated in memory: the backend hands one page to the spool and forgets it. Document order comes from the list of records, never from sorting or globbing the spool directory. For a simplex or hardware duplex profile this is a single scan_pages() call. A profile with duplex = "manual" splits this stage into three (see Set Up ADF Duplex Scanning):

    1. Pass A -- scan the front sides through the feeder. A pass A that reaches its sheet cap ends the job here, with the fronts, and the flip is never asked for (see What a failed scan keeps).
    2. Flip wait -- wait, for at most operator_wait_timeout_seconds, for the operator to flip the stack and confirm. An abort cancels the job here (it ends CANCELLED) and a timeout fails it, both before pass B.
    3. Pass B -- scan the back sides, then reverse them and interleave them with the fronts. If the two passes disagree on page count, either pass could not read a sheet, or pass B reached its sheet cap, the fronts and backs are delivered as two separate PDFs instead, because position no longer proves which back belongs to which front. The (backs) PDF is reversed into sheet order. Its page N is the back of page N of (fronts) only when both passes fed every sheet exactly once; from a sheet that was skipped, missed or fed twice on, the two drift apart, which is exactly why they are not interleaved.

    A scan with Multiple pages chosen (the web form's checkbox, or saneless scan --multi-page) runs this stage as a loop of scan passes instead (see Scan a Multi-Page Document). Each pass is one scan_pages() call -- one page on a flatbed, whatever is in the feeder on an ADF -- and after each one the pipeline asks the operator, for at most operator_wait_timeout_seconds, whether there is another page. The pages of every pass accepted so far form the document, in scan order. A re-scan throws the last pass's pages away, and a pass that fails with a scanner fault or an empty feeder once the document holds a page adds nothing and asks again rather than failing the job. With empty-page detection on, a pass that produced blank-looking pages asks whether to skip or keep them, so the operator decides about blanks here and step 4 does not run for this document. The loop ends when the operator finishes, when a question times out (the document is finished with a warning, not failed), when a pass reaches its own sheet cap (the document is finished with a warning naming the sheet that was fed but not kept), or when the document holds MAX_DOCUMENT_PAGES (500) pages or more after a pass. That limit is checked only between passes, so a pass already running is not cut short, and a document can reach 999 pages. Every pass runs inside the same run guard as any other scan, described below. Manual duplex cannot be combined with it, and is refused before the scanner is opened. 4. Filter empty pages -- remove blank pages using the ink-coverage rule (see How Empty Page Detection Works). 5. Assemble PDF -- embed the spooled page files into a PDF document. 6. Upload to paperless-ngx -- send the PDF with metadata via the REST API, or fall back to the consume directory if the API is unavailable.

Each stage emits a PipelineEvent (SCANNING, AWAITING_FLIP, SCANNING_REVERSE, AWAITING_NEXT_PASS, AWAITING_BLANK_DECISION, AWAITING_RETRY, ASSEMBLING, UPLOADING, DONE). AWAITING_FLIP marks the start of the flip wait and SCANNING_REVERSE the start of pass B. In a multi-page scan, AWAITING_NEXT_PASS, AWAITING_BLANK_DECISION and AWAITING_RETRY mark the start of each question -- is there another page, what to do about blank-looking pages, what to do after a failed pass -- and SCANNING the start of each later pass. Each event becomes a job state of the same name. The four waiting states are active but not busy: the job is not finished, so the web UI keeps polling it and the Scan button stays disabled, but the scanner is idle and a person has to act, so the UI shows no busy spinner and says what it is waiting for. The web UI uses these events to update the live status indicator via HTMX polling.

What a failed scan keeps

Once paper has gone through the feeder, a scan either delivers its document or keeps its pages. One guard spans the whole run, from the moment scanning starts until paperless-ngx has accepted the document, and no step falls outside it: whatever fails -- the scanner, the disk, assembly, the upload, the wait for paperless-ngx, or a bug in saneless itself -- the guard keeps the most finished artefact the run had produced, in failed/ inside the data directory. Nothing kept is ever uploaded, and nothing in failed/ is ever deleted by saneless.

The run failed... What is kept in failed/
while acquiring pages Each pass spooled so far, as its own PDF: (partial) for a single-pass scan; (fronts) and (backs) for manual duplex, with (backs) in sheet order. Pass B feeds the flipped stack from its last sheet, so a pass B that stopped part way leaves the backs of the last sheets: with k backs of N fronts, and every sheet fed exactly once, (backs) page 1 is the back of (fronts) page N-k+1, and the error says so on that condition. When a pass reported a sheet it could not read, no page can be paired by number, and the error says to match the halves by content instead. This covers a scanner fault, a page timeout, the spool running out of room, an empty pass B, a flip wait that timed out and a flip prompt that could not be read. In a multi-page scan, the passes already accepted into the document are kept as one PDF, in document order, and a pass in flight when the run failed is kept beside it as its own (partial) PDF.
after acquiring, including when every page looked blank The whole document, unfiltered, as one PDF in document order.
while assembling the PDF, or when assembly was refused for lack of space The spooled page files themselves, in a job-keyed directory.
while uploading, or while waiting for paperless-ngx to consume it The assembled PDF, or both halves of a mismatched manual-duplex run.
after delivery Nothing: the document is in paperless-ngx.

Blank pages are never removed from what is kept. A PDF built for failed/ follows the same twice-the-spool rule as any other (see Memory, disk and timeouts); if it cannot be built, or the rule refuses it, the page files are kept as well.

Reaching a sheet cap is not a failure, so nothing is kept in failed/ for it. One scan through the feeder keeps at most 500 sheets, or 50 sheets for an Auto source that auto_source_mode = "adf" sends through the feeder -- whether the profile named Auto or Auto stood in for a flatbed source the scanner does not have -- because an Auto source may be a glass that the feeder loop scans again and again without ever running out. A feeder can only tell a full hopper from a runaway by feeding one more sheet, so the sheet past the cap is fed but not kept. The pages kept are uploaded, and the job finishes as Uploaded with a warning (exit 7 from saneless scan), with a warning that names that sheet, so the operator knows where to resume. If the job then fails after all -- every page judged blank, or the PDF could not be built or delivered -- the pages are kept in failed/ like any failure's, and the error, after saying where they went, still names that sheet: "The scan had reached its sheet cap before that: one scan stops after 500 sheets, so sheet 501 was fed but not kept. Scan sheet 501 and any remaining pages as a new document." In a multi-page scan the document is finished at that pass. In manual duplex, a pass A that reaches its cap uploads the fronts and ends the job without asking for the flip, because the sheet past the cap is already in the output tray and would pair every back with the wrong front; a pass B that reaches its cap is never interleaved, and the fronts and backs are uploaded as two documents, (fronts) and (backs).

The failure keeps its own identity. The error a job records, and the line a command prints, is the original one -- a scanner fault is still a scanner fault, a paperless-ngx timeout still a timeout, and a bug in saneless still an unexpected error (exit 5) rather than being blamed on the scanner or on paperless-ngx -- followed by one sentence saying what was kept and where. That sentence is composed from what really reached failed/: it says the scan could not be preserved only when nothing was, and when only part of it was kept it names the part.

Whether anything is kept turns on whether anyone chose to stop:

  • An operator's cancel keeps nothing. Aborting at the flip prompt or at a multi-page question, answering no, or Ctrl-C is a decision to stop, and saneless never prunes failed/, so a cancel that preserved pages would leave the operator files to clean up after choosing not to scan. That includes Ctrl-C pressed while a scan that has already failed is still keeping its pages: the keeping stops there, and anything not yet in failed/ is lost.
  • An interruption keeps the pages, like a failure. A saneless command stopped by SIGTERM or SIGHUP -- kill, or a dropped SSH session -- or a server that stops mid-scan made no decision about the scan, so the pages already scanned are kept. That includes a multi-page scan stopped while it waits for the operator's answer: it is not treated like a timeout, so nothing is uploaded, and the document so far is kept in failed/.

Status updates cannot change how a scan ends. The progress the web UI shows, the first page's preview and the page count between manual-duplex passes are reported as the run goes, and a failure to record one of them -- a busy job database, say -- is logged as a warning while the scan carries on to its real outcome.

Workspaces. Each run works in its own directory under tmp_dir, named job-<the first 8 characters of the job id>-<random>, which holds the spooled pages and the assembled PDFs and is removed when the run ends, however it ends -- with one exception. When a failed run's pages could not all be kept in failed/, because the data volume is full or cannot be written, the workspace holds the only copy of the rest: it is left in place, unlocked, the failure's message names where, and the next sweep (below) moves them into failed/. A workspace that cannot be removed when its run is over is renamed leftover-, because nothing in it still needs keeping and a sweep must not take it for a killed scan; delete it by hand. It is created under a hidden staging name, locked, and only then given its job- name, and it stays locked for as long as the run lives, so anything that looks through tmp_dir can tell a live run's workspace from one a killed process left behind. On a filesystem that cannot lock (some network mounts), the workspace is used unlocked, with a warning, and named unlocked- rather than job-: nothing could prove its owner alive, so no sweep ever touches it, and it is never recovered either.

Recovering a killed scan

A process killed outright -- SIGKILL, the out-of-memory killer, a power cut -- runs no guard at all, so its pages are still in its workspace when it dies. Because the workspace is named after its job and its lock dies with the process, the next start can find it. Both saneless serve, as it starts, and every saneless scan, before it opens the scanner, sweep tmp_dir for job- workspaces whose lock is free. The CLI sweeps too, because an install that only ever runs saneless scan has no server to do it. A workspace whose lock is held belongs to a scan that is still running, in this process or another one sharing tmp_dir, and is never touched; nor is a symbolic link, or a directory another user owns.

Each recovered workspace is kept the way a run that failed while acquiring is kept: the spooled pages of each pass become a PDF in failed/ -- (partial) for a single pass, (fronts) and (backs) for manual duplex, with (backs) in sheet order -- under the same twice-the-spool rule. An interrupted write (a .part file) is not a page, and an unreadable last page, the one the process was writing when it died, is left out. If a PDF cannot be built, or the rule refuses it, or a page other than the last cannot be read, the raw page files are moved into failed/ instead. The workspace is then removed, unless some of its pages could not be kept, in which case it stays for the next sweep. One warning per workspace names the job, its title and where the pages went. At serve startup, a job row the workspace belongs to is failed with its restart text followed by that sentence, so the web UI shows its owner where the pages are. A CLI scan has no job row, so the warning is its whole report. A sweep that fails is logged and never stops the server starting or the scan running. The sweep runs before the web server answers, and each page is read back and assembled again, so a large orphan delays startup; one log line says up front how many workspaces and pages it is recovering.

PDF Assembly

PDF assembly uses img2pdf, which embeds each spooled PNG directly into the PDF without re-encoding it. This is lossless: the page is encoded once, as the PNG the spool wrote, and that PNG's image data is what the PDF carries. It is not byte-identical to the raw data the scanner sent over the wire -- that data was never stored -- but no pixel changes between the spooled page and the page in the PDF, and assembly is fast because there is no second compression step.

Assembly converts one page at a time, writing each to its own single-page PDF, and then merges those with qpdf. That is why its memory does not grow with page count: at no point is more than one page's worth of PDF being built. Handing the whole set to img2pdf in one call instead would hold every page in memory until the document was finished.

Memory, disk and timeouts

These are the rules that decide how much machine a long scan needs, and how a scan that stalls ends.

  • Roughly one decoded page is in memory at a time while scanning. A page is handed to the spool and written out as it arrives, so the ceiling is a small constant rather than something that grows with the stack: saneless holds one decoded page, and the scan's child process holds the page it is sending and one strip of its rows. At A4 300 DPI colour a page is about 26 MB, so a few hundred megabytes of RAM is enough for a scan of any length.
  • Disk is checked per page, not just once. Before each page is written, saneless checks there is room for that page plus min_free_space_mb, which is held back so PDF assembly still has somewhere to work. A shortfall fails the scan naming the page number and the spool path, and the pages already spooled are kept.
  • Assembly needs twice the spool free, and is refused up front without it. Assembly writes a single-page PDF for each page and then the merged document, all beside the spooled pages, so at its peak it needs about twice the spooled pages' size on top of the spool itself. Before building any PDF saneless checks for twice the spooled bytes plus min_free_space_mb, and when that is not free it refuses before writing anything, naming the MB needed and the MB free, rather than filling the disk half way. The rule applies to the main document, to the two halves of a mismatched manual-duplex run (counted together), to a preserved partial scan, and to the recovery of an interrupted scan's pages. A refusal keeps the page files in failed/, as any other assembly failure does.
  • The per-page limit scales with the page, and is never below 120 seconds or above an hour. One limit bounds one sheet through the feeder and one sheet on the flatbed alike. It is worked out for each scan from the resolution and page size the scanner agreed to once every option is set: a page gets twice its share, by bytes, of one minute for an A4 colour page at 600 DPI. So A4 colour at 1200 DPI gets 480 seconds and A4 grey at 1200 DPI 160 seconds, while anything at 300 DPI, where a page typically takes 10-15 seconds, gets the 120-second floor. A feeder that does not know the page length in advance is given the time for a legal-length sheet. The hour caps what a scanner can claim: the page size comes from the scanner, over the network for a network scanner, and a nonsense size must not leave a sheet unbounded. The limit is not a setting. A page that times out says what limit it was given and for what page, so a slow USB or Wi-Fi link or a high resolution can be told from a scanner that has stopped answering.
  • Every scanner call runs in the scan's own child process, under a deadline saneless keeps. A scan job makes every SANE call in one child process started for the job, and saneless, not the scanner, decides how long each step may take: 30 seconds each for starting the scanner library, opening the device, setting its options and reading the scan parameters it agreed to, closing it, restarting the library between passes, and the child's exit; the per-page limit above for starting and reading a page. No deadline runs while a job waits for the operator between passes.
  • On a timeout saneless cancels the read, and the child waits for it before closing the device. The SANE rule is that no other operation may run while one is outstanding, so after the cancel the child waits for its read to come back, closes the device and exits. saneless waits up to ten seconds for that; a child that has not exited by then is killed, with its whole process group, and reaped. A page that arrives after the cancel is discarded rather than added to the document, because the failure has already been reported. The pages received before it are kept, as for any scan that fails while acquiring.
  • A scanner that stops answering holds a scan for the ten-second grace at most, on every backend. A cancel, Ctrl-C, a server stop and a page past its limit all end the scanner's use within that ten-second grace plus the time to reap the child, whatever the backend does with the cancel -- including the backends that cancel asynchronously (avision, hp, umax, hp3500), and snapscan and plustek, which fall back to their own cancel only after ten seconds. A page past its limit fails the scan with the page's timeout line, which ends so saneless stopped it when the child had to be killed. Any other step that runs past its deadline kills the child at once, and the scan fails with a line naming the step, such as The scanner stopped answering while opening the scanner; saneless stopped it. A crash in the scanner library kills only the child, and fails the scan the same way.
  • saneless itself never waits inside the scanner library. No scanner call runs in saneless's own process, so a network scanner that stops answering freezes nothing: the web UI, /health and signal handling go on, saneless serve stops on Ctrl-C, and a container stops on docker stop.
  • The next scan starts a fresh child. Once a child has been killed and reaped nothing of it is left, so the next scan starts a new one, with a fresh connection to the scanner; if the scanner is still not answering, that scan fails on its own deadlines. No restart is needed. Listing scanners, opening one for the Scanner check and reading its options run in a separate process too, with its own deadline (see Worker Thread Model).

Paperless Upload

The PaperlessClient sends the PDF via multipart POST to the paperless-ngx REST API (/api/documents/post_document/) with metadata including title, tags and correspondent. It sends no document date: paperless-ngx dates the document itself, from a date it finds in the document or, failing that, the time it received the upload (see Document date). After upload, it polls the task endpoint with exponential backoff until paperless-ngx reports success or failure.

An upload is sent again only when it cannot have reached paperless-ngx -- a connection refused or not made in time, no free connection, a proxy that refused the tunnel, or a send that timed out -- and those failures are retried for about 60 seconds. If a consume directory is configured, the PDF is deposited there as a fallback once that time runs out; saneless never creates the directory, and a missing one fails the scan with the PDF kept in failed/. An upload that may have arrived -- the answer never came or timed out, a 5xx, or a 200 without a task id -- is never resent and never copied: the job ends amber, with the PDF kept in failed/, because a second upload could file the scan twice. saneless speaks paperless-ngx API version 9, or 10 once paperless-ngx says it allows it, so it works with paperless-ngx 2.16 or later; an older one answers 406, which fails the scan with its own message. A problem on saneless's own side never falls back: a paperless.url or paperless.token that could never work is refused when the configuration loads, an unset paperless.url stops a scan from starting at all, and a request that cannot be built from the configured values fails the scan at once as a configuration error, without retrying and without touching the consume directory, with the PDF kept in failed/. An upload paperless-ngx rejects (a 4xx, such as an authentication failure) or redirects (a 3xx, usually a paperless.url pointing at the wrong address) never falls back: the scan fails with Paperless's reason or the redirect target. See How Consume Directory Fallback Works for details.

Worker Thread Model

The web server runs a background worker thread that processes one scan job at a time. Jobs are submitted to a bounded queue.Queue and executed sequentially -- scanning is inherently serial because a physical scanner can only handle one job at a time. The queue holds at most 10 jobs waiting to start. Submitting never blocks: a scan submitted to a full queue is refused with 429, and one submitted while the worker is down or degraded is refused with 503 (see the Web API reference).

The worker communicates progress back to the web layer via state transitions on the Job object stored in SQLite. The UI polls the job status endpoint, and HTMX updates the status indicator when the state changes. The scan itself never runs on a request: it runs on the worker thread. The route handlers are plain (non-async) functions, which FastAPI runs on its threadpool, so a slow paperless-ngx call or a database call in one request does not stall /health, the status poll or any other request.

Startup. Before the worker thread starts, the app first recovers the workspaces a killed process left in tmp_dir (see Recovering a killed scan), then fails every job that a previous process left active (running or still queued) with "The server restarted before this scan finished" -- followed, for a job whose workspace was recovered, by where its pages were kept -- and then prunes job history. A job that was uploading is the exception: it may already be in paperless-ngx, so it is failed with "The server restarted while this scan was being uploaded; it may have reached paperless-ngx" instead, in the amber UNCONFIRMED_SEND category, whose advice is to check the document list before scanning again. A queued job is never picked up again after a restart, so a scan cannot start on whatever paper happens to be in the feeder by then. A recovery that fails is only logged, and the app starts anyway. If the job store cannot be written at this point, the app still starts, but the worker starts degraded and carries out that recovery, with the same texts, once the store accepts writes. The worker thread's first act, before it takes any job, is startup profile generation: the server is already answering requests while it runs, and a scan submitted meanwhile waits in the queue.

Failure handling. A scan that fails -- a scanner error, an upload error, a flip wait that timed out -- fails its job and the worker moves on to the next one. A scan that ends without delivering a document ends in one of three ways: an operator's abort at the flip prompt or at a multi-page question is recorded CANCELLED and logged at INFO, with no traceback, because nothing went wrong and nothing is kept; a failure is recorded ERROR, logged with its traceback, and keeps its pages as What a failed scan keeps describes; and a scan the server stopped is recorded ERROR with "The server restarted before this scan finished" followed by where its pages were kept, logged at INFO with no traceback, and with no error category, because it neither failed nor was cancelled. A scan the server stopped during its upload is recorded with the uploading restart text and the amber UNCONFIRMED_SEND category instead, because the document may already be in paperless-ngx. The terminal job states are DONE, FALLBACK, ERROR and CANCELLED. A failure of the worker's own job-store writes, or of its hourly history prune, is different: three in a row mark the worker degraded. A write that succeeds in between -- a scan whose job records are all written, a prune that succeeds, or an owed job record written -- starts the count again, so failures spread between successes never add up. While degraded, /health returns 503 with job store failing and new scans are refused, because a job whose outcome cannot be recorded should not ask anyone to feed paper. The worker retries any job records it could not write earlier -- a finished scan's outcome, a job failure, or the rejection of a refused scan that the web request could not record -- after each scan that ends cleanly and, whenever it has no job, every 5 seconds, so neither a busy queue nor an idle one leaves them waiting, and a job the store briefly would not let end still ends once writes succeed again. A scan that reached paperless-ngx is still recorded as done, not as an error, even when recording that outcome failed at first. If three of those retries fail in a row, the worker marks itself degraded as well, so a job store that stays broken shows up on /health instead of only in the logs. A scan that ends cleanly starts that count again too, so on an idle worker a store that stays broken degrades it within about 15 seconds, and a store that heals sooner never does. Degraded is not permanent either: while degraded, the worker also probes the job store on those same ticks, and it clears degraded on the first tick on which the probe succeeds and the records it still owes are written.

Shutdown. Stopping sets a stop flag and shuts the queue down, so jobs still waiting are dropped rather than run. A job parked at the flip prompt or at a multi-page question is answered with a shutdown outcome of its own, not with Abort, and a job still scanning has its scan child cancelled, and killed if it has not stopped within ten seconds: nobody chose to throw the scan away, so the run's single guard keeps its most finished artefact -- the pages received so far, pass A's fronts as a (fronts) PDF in failed/, or the multi-page document so far -- and the job is recorded ERROR with "The server restarted before this scan finished" followed by where the pages were kept. The server then waits at most 5 seconds for the worker thread to finish. If the thread is still running then while its job has a scan child, the wait is extended by up to 12 seconds, the ten-second grace and a margin for reaping, so the child is always cancelled or killed, and reaped, before the server exits. If the thread is still running after that because it is keeping those pages -- under Docker, a copy from /tmp to the data volume -- the wait is extended once, by up to 60 seconds, so the process does not exit half way through the copy; a thread busy with anything else gets no extension. Before it exits, the thread tries once more to write any job records it still owes. A job busy with anything else when the wait runs out, such as assembly or the upload, is abandoned: its job stays active, and the next startup recovers the workspace it leaves behind and records the job with its restart text, as it does for the dropped queued jobs. The job store, the paperless-ngx client and the scanner are closed only once the worker thread has confirmed it stopped, so a thread still finishing a scan never writes to a closed database.

Stopping the server. Once asked to stop, the web server stops accepting connections and waits at most 8 seconds for requests still being answered before it cancels them. The status strip's checks, including the probe the Check again button asks for, run on a background refresher thread rather than on a request, and every call a request makes to paperless-ngx has a 2-second connect and a 5-second read timeout. So one such call that was under way when the stop came, such as a connection test or one page of the tag or correspondent list, ends inside those 8 seconds. A list of more than one page is one call per page. The scan form's lazy load of the tag and correspondent lists (GET /api/metadata) makes two such calls in a row, and each may first wait as long again for another request's fetch of the same list. A name lookup takes no timeout at all. Any of these can outlast the wait. The request is then cancelled, but the thread answering it runs on until its call returns, and the server can close its paperless-ngx client while that thread is still using it. The worker and the refresher then share the one 5-second wait above. The refresher is told to stop earlier, as soon as the server is and before the wait for requests begins, so it starts no check after the one it is in. Stopping it aborts a scanner check in flight: a scanner listing, whose child it kills and reaps, and the probe of a saned host, which gives up waiting for an answer at once and on a connection still being made within 2 seconds. Its paperless-ngx check is one request of up to 7 seconds that cannot be cut short, and a name lookup takes no timeout at all. A stop that lands in either waits for it, and has the wait for requests as well as the shared wait to do so. A server with no request to wait for has little more than the shared 5 seconds, so a paperless-ngx check that had only just begun, or a slow name lookup, can still outlast them. The server then exits without closing its job database, its paperless-ngx client or the scanner, leaving that to the process exit. So a server with no scan running and no request to wait for stops well inside the 10 seconds Docker gives a container by default, and it exits 0 on SIGTERM whether or not it is the container's first process. A stop that has to wait for a request to paperless-ngx can take up to 8 seconds longer, a stop during a scan up to 12 seconds longer while its child is cancelled and reaped, and a stop while a stopped scan's pages are being kept up to 60 seconds longer, which is what the compose file's stop_grace_period: 90s and docker run --stop-timeout 90 allow for.

No SANE call happens in the server's own process. Every SANE query outside a scan -- the Scanner check's listing and open check, auto-detecting a device for a scan, reading a scanner's options for the CLI and for generating the start-up profiles the web form offers, and the saneless devices and saneless auto-profiles commands -- runs in a short-lived child process with a 30-second deadline. The child is killed and reaped before the query returns, so a caller holding the scanner gate releases it with nothing still inside the scanner library, and a crash or a hang in the scanner library cannot take the server down: a crash becomes a failed query, and a hang becomes a process that is stopped at the deadline.

A scan job runs in one child process of its own, started for the job's first pass and reaped before the job releases the scanner gate, so a health probe never reaches the scanner while a scan's child is alive. Each child starts SANE afresh, which gives every scan a fresh connection to a network scanner host, including after its saned restarted. Within a job the child restarts SANE before every pass after the first, except the second pass of a manual duplex scan. The pixels of each page come back over a pipe in strips of whole rows, and the child reads no further sheet until saneless has spooled the page. SANE never starts in saneless's own process, so there is nothing to shut down when a command or the server ends.

For manual duplex scanning, the pipeline waits between pass A (fronts) and pass B (backs) through a FlipCoordinator, a small interface with one method, wait_for_flip(timeout), that returns one of three outcomes: continued, aborted or timed out. The pipeline does not know who answers. In the web server, the worker's coordinator is answered by the /api/flip/continue and /api/flip/abort routes, which the flip prompt's Continue and Abort scan buttons call, and only an answer from the browser that started the job reaches it. In the CLI, a coordinator asks the operator a yes/no question at the terminal and reads the answer on the main thread, so Ctrl-C, a hangup or the timeout ends the wait at once and nothing is left reading the terminal after the command ends. Whichever answer arrives first -- including the timeout -- is final, and later answers are dropped. In the web server the worker's coordinator is bound to its job and accepts an answer only once that job is waiting at the flip prompt, and the routes name the job they answer, so a stale or repeated click is dropped instead of reaching another job.

The wait is bounded by operator_wait_timeout_seconds (600 seconds by default), so an abandoned flip prompt cannot hold the worker thread forever: when the timeout elapses the job fails and the next queued job runs. The scanner itself is not what the timeout frees. The job's scan child opens and closes the device inside each pass, so the scanner handle is already released between the two passes; the child waits through the flip holding no device, under no deadline, and what the timeout releases is the worker thread. A pass A that failed never reaches the flip prompt, because the scan has already failed (see Memory, disk and timeouts).

A multi-page scan asks its questions through a separate PassCoordinator, whose one method, ask(prompt), puts a numbered question to the operator and returns an answer from the set that question offers, or a timeout, or a shutdown. It is a sibling of the FlipCoordinator, not a widening of it, so the manual duplex flow is unchanged. In the web server the worker's coordinator is answered by POST /api/multi-page/answer, which names the job and the question number; an answer for a question that is no longer open, or one the question does not offer, is dropped. In the CLI a coordinator asks a one-letter question at the terminal, read on the main thread like the flip question. Every question is bounded by the same operator_wait_timeout_seconds as the flip wait, but where a flip timeout fails the job, a multi-page timeout finishes the document with the pages kept and records a warning, because the pages already scanned are what the operator was building. The scanner is closed between passes, as it is between the two manual duplex passes, and the job's scan child restarts SANE before every pass after the first, so a document that waits a long time for its next page still gets a fresh connection to a network scanner.

Job Storage

Job records live in a SQLite database under the configured data_dir -- durable state, deliberately not the disposable tmp_dir the scan in progress is built in -- and a single JobStore instance owns the only connection to it. Nothing else in the process opens that file. Centralising the connection is what makes the rest of this section possible: there is exactly one place a transaction can begin, one place the schema can change, and one lock to take. The connection enables write-ahead logging before it switches to explicit transaction control, because SQLite refuses a journal-mode change once a transaction is open.

That single connection is driven by two kinds of caller. Web request threads read and write job rows while serving HTTP, and the background worker thread writes state transitions as a scan progresses. A sqlite3 connection is not safe to drive from several threads at once, so every public JobStore method is serialised on a threading.RLock and each method opens its own transaction inside that lock. The lock guards the connection, not merely the data: without it, two threads can interleave statements inside one another's transaction and commit a mixture of both. Public methods deliberately never call other public methods, because sqlite3 transaction contexts do not nest -- an inner one commits the outer -- and that rule is enforced by a test rather than left to convention.

The schema is versioned with PRAGMA user_version and evolved by an ordered ladder of migration steps, where step N produces version N. A fresh database reports version 0 with no jobs table, so the ladder runs from the start and builds the schema from nothing. A database that reports version 0 but already has a jobs table predates the version stamp, so it is taken to be at version 1 and joins the ladder partway. Each step stamps its version and commits before the next begins, so a ladder that fails at step N leaves a valid database at version N-1 rather than a half-applied one.

If a database does not have the shape the next step expects, the store raises StorageError naming the file and the columns it could not find, and closes the connection rather than opening. The loud refusal is the point. An unrecognised database that opened anyway would look like a clean startup and then fail at its first write -- far from the cause, with an error that says nothing about the schema. Failing at open time puts the report where the problem is, and because the check runs before the first ALTER, a rejected database is left exactly as it was found and can still be inspected or restored.

Retention is a single DELETE that removes jobs past an age cutoff together with jobs outside a row cap, and it reports its count from that one statement rather than from reads taken either side of it, so a job inserted mid-prune cannot make the count wrong. Both limits are settings -- see history_retention_days and history_max_rows in the Configuration Reference. Both select on creation time and neither consults whether the job finished, so retention removes in-flight and errored records on exactly the same terms as completed ones.

Six columns -- outcome, pages_scanned, pages_removed, pages_uploaded, warning and owner_token -- record who started a job and how it ended. Each is written by one component:

  • pages_scanned, pages_removed and pages_uploaded are written by the worker on the success path, and rendered as one sentence in the status area and as a second line in the history table's Title cell.
  • outcome and warning are written by the worker when a scan finishes. outcome records how it resolved -- SUCCESS or FALLBACK, distinguishing a document that went through the API from one left in the consume directory. There is no FAILED member and there will not be one: a failure raises, so the column stays NULL on the error path and ERROR carries that fact alone rather than in two columns that could disagree. warning carries the sentence the status area prints beneath the status line, such as what a fallback cost in metadata. saneless jobs --json reports both.
  • owner_token is written by the web layer when a scan is submitted, and identifies the browser that started it, which carries the same token in its saneless_owner cookie. It is what lets the flip prompt and the multi-page questions be shown to the person holding the paper and not to everyone with the page open. It is not a login credential: it grants only the right to see that job's details -- its title, preview and error text -- and to answer its flip prompt and multi-page questions (see Answering from a script). A job with no recorded owner is nobody's: every browser sees its generic title, and anyone may answer it.

Every one of them can read back unset. A job that never reached the success path has no page counts, and most refused submits record no owner. Anything rendering them has to handle NULL, and the UI does: a job with no counts simply shows no counts line.

Configuration Layer

saneless uses pydantic-settings to load configuration from one TOML file, saneless.toml, with environment variables (prefixed SANELESS_, nested with __) overriding it. A variable that is set overrides the file without a word, so a container keeps the paperless-ngx URL and token in its mounted saneless.toml and uses variables only for what differs per deployment. Where saneless looks for the file, and the order in which variables, the file and the built-in defaults apply, are in Where saneless reads settings.

The profile system stores named scanner presets (source, resolution, color mode, default metadata) so users can switch between common scan types without reconfiguring each time. Profiles are defined in the TOML file under [profiles.<name>] sections.

Web Layer

The web layer is built with FastAPI, Jinja2 templates, and HTMX for reactivity. There is no JavaScript framework and no JS build step -- the entire frontend is server-rendered HTML with declarative HTMX attributes for dynamic updates.

Pico CSS provides the styling, with automatic dark/light mode support; the build vendored in the package is Pico's standard, class-based one (pico-2.1.1.min.css). There is no application JavaScript. The server renders the Scan button, and every status response -- a scan submission, the status poll, a flip Continue or Abort -- re-renders it out of band, so its enabled or disabled state always comes from the server's view of the current job. htmx and Pico are served from the package itself, pinned with integrity hashes, so loading the web UI needs no internet access.

Having no application JavaScript has a cost, and it is accepted. htmx gives up on any request after 20 seconds, and a request that fails on the network fails silently, so with the server stopped the page keeps its last rendering and nothing on it says the server has gone. A poll is not cancelled by a failed request, though: htmx schedules the next one regardless, so once the server answers again the status area catches up by itself. A restarted server never matches the token a poll presents (the token is keyed per process), so that first answer is always a full rendering. A job store that cannot be read is a different case, because the server is still answering. The status area shows a retrying line in amber and polls again after 2 seconds, then 5, then every 15 seconds. Polling never stops, so the line is replaced by the real status once the store can be read again. The page's message area is never used for this, because it is where the operator's own failed actions are reported, and an error left there would still be standing above "Done".

The status poll sends as little as it can. Every status rendering carries a token of what it shows in its poll URL. A poll whose token still matches is answered 204 with no body, and htmx leaves the area alone, so a focused button and a held key or pointer press survive the poll.

The status strip's polling

The status strip at the top of the page (see GET /api/checks) is a cache read. The checks themselves run on the background refresher thread, so however many tabs are open, watching the page generates no scanner or paperless-ngx traffic beyond what the cache allows. Until the first results exist, the strip polls for itself, and every way that poll can go is bounded or deliberate.

Two windows, chosen by whether a check is running. With nothing in flight, the strip gives up after about twenty seconds. That is the case where the checks are not going to run at all -- the refresher has stopped, say -- and a tab left open on a hallway tablet in front of a half-broken appliance would otherwise ask forever. Giving up costs nothing that was on the page: the six rows and the Check again button stay, and the line beneath them points at the button, which is the one way forward left. With a check running -- the first one, still working through an unreachable scanner -- the strip keeps asking for about three minutes instead. A single check can legitimately take far longer than twenty seconds: name resolution has no timeout of its own, and a deployment with no configured scanner host goes straight into the SANE enumeration, which on Linux spends around two minutes on a host that is switched off. Telling somebody to press a button that starts the very check already running would not help either: the press collapses into the check in flight and the page waits exactly as long. The longer window is still a window. A check still running after three minutes is one whose thread may well have died holding the lock, and at that point the button really is the only thing that can put an answer on the page.

The poll counter. Each poll carries a counter of how far its chain has gone. A counter above the range is read as the end of the chain: it clamps to the larger cap, which is the give-up body, so the strip comes back with no poll attached. A counter below the range, which nothing on the page sends, is the one input that does not end the chain: it is read as zero, the number a page render starts at, so it comes back as the first body of a fresh chain, bounded like any other.

Errors end the chain. Any failure inside the strip's own rendering comes back as the cold-start body with no poll attached. A genuine error response to the poll's own request -- a counter that is not a number at all -- is not retargeted into the status area the way every other htmx error is, so it replaces the strip, and the poll ends with it. That exemption is for the strip fetching itself and nothing else: the Check again button aims at the same element, but it is a POST, and an error answering it goes to the status area like every other click's, leaving the strip and the button on the page. The strip does not come back on its own once that error body has replaced it: the body carries no checks-body id, so the reload every terminal job state fires at the strip and the out-of-band strip a scan submit carries both find nothing to replace, and a later scan's paused note has nowhere to land. That is accepted because the only request that can reach this response is the poll's own, carrying a counter nothing on the page ever mints, so the tab it empties is the one that crafted it.

No response keeps the poll going. A request that gets no response at all -- the appliance switched off mid-poll, the connection reset -- produces nothing to replace the strip with, so that tab goes on asking every couple of seconds until it is closed. It is asking an origin that is not answering, so there is nothing on the page for it to overwrite and no scanner or paperless-ngx traffic behind it.

Check again. The button (POST /api/checks/refresh) exists so an operator who has just plugged the scanner back in does not have to wait out the cache's freshness window. Three rules keep it from becoming a way to load the scanner or paperless-ngx:

  • A scan is not interrupted. Whether a scan is running is decided from the job the scan worker reports, not from whether some other check happens to hold the scanner, so a busy scanner during the start-up capability read gives a row saying the scanner was busy and naming no scan. That job is read once, before the checks run, and the worker records the job it is about to start before it takes the scanner. A scan that starts inside that window is invisible to the sample, so the scanner check goes ahead, finds the scanner taken and produces the busy row, while the line above the rows, composed after the checks finish, has since seen the job and says the strip is paused during a scan. The two do not contradict each other: nothing was checked either way, which is what both say, and the next refresh replaces the row with the ordinary paused one. A skipped row stored during a real scan stays until the next probe replaces it, so for up to one refresh interval after a scan the strip can still say a scan is running; it was true when it was written, so it is stale rather than wrong, and Check again replaces it at once.
  • Simultaneous refreshes collapse into one. A refresh that arrives while one is under way re-renders the strip as it stands instead of running every check again. The answer the refresh in flight is about to produce is the same answer, seconds away, and repeating the work would mean a second request to paperless-ngx and a second pair of write tests. That answer still arrives without a second click: the collapsed strip asks for itself until the result lands, within the same two windows as the start-up poll, so a check wedged against an unreachable host cannot leave a tab asking forever.
  • A floor under probing. A refresh is honoured at most once every couple of seconds, and a sooner one re-renders the current strip with the same status code and partial as an honoured one, because an early click is not an error. A collapsed refresh does not spend that floor: nothing was probed, so nothing was spent, and the next press is honoured at once. So holding the button down, or scripting the endpoint in a loop, cannot add scanner or paperless-ngx traffic beyond that rate or delay a scan by contending for the scanner -- and neither can a loop that always collides with a running refresh, because a collapsed request performs no check of any kind.

Every error the web layer produces goes through one rendering rule: the web UI receives a message shown in the page's message area, and other clients receive a JSON error body with the same status code (see Errors). A middleware in front of every route rejects cross-site state-changing requests (see Cross-site requests).

Three more rules protect a web UI that has no login. A Host check in front of everything else answers only requests that name saneless's own host names, which is what stops DNS rebinding (see Host check). Every response carries a Content-Security-Policy that allows no inline script or style, and forbids framing (see Security headers). And templates never receive a stored job: every route renders an owner-scoped view of it, so only the browser that started a scan sees its title, preview and error text, and no page carries a host path or the paperless-ngx URL (see What an unauthenticated client can read).

The web UI is a thin layer over the same pipeline that the CLI uses. Both entry points build their request with one request builder and decide a scan's tags and correspondent with one metadata policy -- an untouched or hidden control means the profile's defaults, a cleared one means none -- and both call run_pipeline() with what those produce. The difference is how status updates and the outcome are delivered. The web UI's scans run on the worker and are recorded in the job store, and the status area and the job history render each job's stored outcome: a FALLBACK job as Saved to folder, and a DONE job that carries a warning as Uploaded with a warning, with the warning sentence beneath it in both cases. saneless scan does not write to the job store, so its scans never appear in the job history or in saneless jobs. It reports the outcome run_pipeline() returns on the terminal instead: an outcome line on stdout (Done:, Uploaded with a warning: or Saved to folder:), the warning on stderr, and an exit code of 0, 7 or 6. The wording for each outcome is defined once and shared by every surface, so the CLI and the web UI cannot describe the same result differently.