# Tutorial 4: PDF workflows

This tutorial covers `adsfolio pdf` end-to-end: identify, rename, watch, status, upgrade, and library push.

## 0) Start with `--help`

Before you apply file operations, review the full PDF command surface:

~~~bash
adsfolio pdf --help
adsfolio set --help
adsfolio cache --help
adsfolio pdf ocr --help
adsfolio pdf identify --help
adsfolio pdf refresh-ads --help
adsfolio pdf rename --help
adsfolio pdf watch --help
adsfolio pdf status --help
adsfolio pdf upgrade --help
adsfolio pdf push-library --help
adsfolio pdf export --help
adsfolio pdf obsidian --help
~~~

Key PDF options:

| Option | Meaning | When to use |
| --- | --- | --- |
| `identify --settings-scope global/library` | Load reusable global or per-library settings | Share defaults or keep one library distinct |
| `identify --recursive/--no-recursive` | Override saved recursive scanning for one run | Multi-folder paper archives |
| `identify --strategy` | Select one or more ordered evidence strategies for one run | Limit or reorder local metadata extraction and lookups |
| `identify --debut-year/--end-year` | Override publication-prefetch years for one run | Narrow a cache refresh without changing setup |
| `identify --refresh-cache` | Rebuild the publication cache first | Deliberate cache refresh |
| `identify --output-unresolved` | Move unresolved PDFs to a separate directory | Keep review queue clean |
| `ocr --output` | Route likely scanned/non-OCR PDFs | Separate files needing OCR before identify |
| `ocr --make-ocr-sh` | Generate `ocrmypdf.sh` in current directory | Run OCR in batch with one command |
| `identify --ai-csv` | Apply AI-enriched unresolved CSV directly | Non-interactive AI roundtrip |
| `identify -v` | Print per-file unresolved move logs | Debug detailed unresolved routing |
| `identify --verbose-parser` | Show raw parser warnings from malformed PDFs | Debugging problematic files or corrupt inputs |
| `--apply` | Apply the command's file moves, renames, remote additions, or upgrades | Preview the command first. This flag does not disable all catalog writes or ADS requests |
| `rename --scheme` | Template defining output filename | Enforce your citation/file naming convention |
| `rename --author-format` | Author token style (`surname`, `surname_initials`, etc.) | Match your preferred filename style |
| `watch --use-watchdog` | Event-driven watcher backend | Lower-latency updates on supported systems |
| `watch --auto-upgrade` | Periodic arXiv→published upgrade checks | Keep long-lived PDF archives canonical |
| `watch --upgrade-interval` | Hours between ADS rechecks and auto-upgrade passes | Balance freshness and API activity |
| `upgrade --overwrite-arxiv-with-official` | Replace tracked arXiv file contents | Preserve stable path/filename while updating content |
| `upgrade --arxiv-backup-if-overwritten` | Save backup before overwrite | Safety rollback path |
| `upgrade --skip-annotation-port` | Download official PDF without transferring annotations | Fast upgrades or no `pymupdf` environment |
| `upgrade --force` | Re-check entries already marked as published | Force re-scan after manual registry edits |
| `push-library --apply` | Push resolved bibcodes to ADS library | Synchronize local PDF collection with ADS library |
| `obsidian --notes-dir` | Write `.md` notes to a directory other than the PDFs | Keep notes separate from the PDF folder |
| `obsidian --force` | Regenerate existing notes and replace manual edits | Refresh metadata after re-running `identify` |
| `obsidian --apply` | Fetch/cache missing abstracts and write notes | Always run after dry-run verification |

## Prerequisite

Install optional features:

```bash
pip install 'adsfolio[full]'
```

Run `adsfolio set` once before your first `pdf identify` run. This step lets the tool cache
these items:

- your publication list (ORCID + optional names + year window)
- optional library papers
- optionally, references/citations of that library set

This cache reduces per-file ADS queries.
Run `adsfolio set` again only when token/profile values change.

## How identify works

`adsfolio pdf identify` uses the central user catalog `pdf-library.sqlite` as its
incremental state. The tool imports legacy per-root registry and JSON files. It does
not rename or remove them.
The user-level publication cache is separate. `adsfolio cache` manages it.

For each new, changed, or unresolved catalog location, `identify` does these steps.

Find new and changed files:

1. Scans the selected root top-level or recursively. When `--skip-unresolved-directory`
   is enabled, it skips the configured unresolved destination before hashing.
2. Compares stored size, modification time, and filesystem identity first.
3. Computes SHA-256 only for new or changed PDFs, then reuses matching catalog attachments.

Run the evidence strategies:

4. Runs the enabled evidence strategies in order. Each stage handles only PDFs that
   are still unresolved. Before it makes any ADS network request, it checks the
   root's cached ADS documents and publication cache.
5. With the default strategy order, extracts and looks up:
   `filename_bibcode`, `filename_doi`, `pdf_metadata_bibcode`,
   `pdf_metadata_doi`, `first_page_bibcode`, `first_page_doi`,
   `structured_filename`, `pdf_metadata_title`, and `first_page_title`.
   It batches candidate lookups by identifier where possible. Repeat
   `--strategy` to select an ordered subset for one run, for example:

   ```bash
   adsfolio pdf identify ~/Papers \
     --strategy filename_doi \
     --strategy first_page_title
   ```

Handle the files that remain unresolved:

6. If files remain unresolved, writes `unresolved.csv` and prints an AI prompt.
7. In interactive mode, waits for an AI-enriched CSV path and applies batched
   ADS bibcode/DOI lookups from that CSV.

With `--recursive`, the same central catalog still holds all tracked files.

First-page title identification collects typography candidates and checks them
against ADS titles. A larger journal masthead does not automatically become the
title. Conflicting records remain unresolved. Uniform-font OCR and titles with
mixed font sizes can still require a verified DOI or Bibcode.

Resolved PDFs with unchanged content reuse their catalog metadata without
repeat identification queries. Unresolved or changed PDFs remain eligible for
identification.

## 1) Optional: route non-OCR PDFs before identify

Dry-run:

```bash
adsfolio pdf ocr ~/Papers --recursive -o ~/Papers/ocr_required
```

Apply move:

```bash
adsfolio pdf ocr ~/Papers --recursive -o ~/Papers/ocr_required --apply

# Generate ocrmypdf.sh (uses the same --output directory as OCR input)
adsfolio pdf ocr ~/Papers --recursive -o ~/Papers/ocr_required --make-ocr-sh
bash ocrmypdf.sh
```

The command prints an `ocrmypdf` script template. If these files have annotations,
check OCR outputs carefully. You can manually move selected files out of
`ocr_required` to skip OCR for those files.

## 2) Identify PDFs and update catalog metadata

```bash
adsfolio pdf identify ~/Papers --recursive
```

This command matches the publication cache, reuses cached fallback ADS documents, and
writes the central catalog outside the library root.

Reusable Identify settings include these values:

- ordered evidence strategies
- recursive scanning
- unresolved destination
- unresolved-directory skipping
- verbosity

Personal identity and publication-year bounds are global profile data. You configure
them with `adsfolio set` or **Tools → My Publication Profile…**. They are not
library-scoped Identify behavior.

For a paper that ADS does not index, choose DOI providers and their order. This
example does not use ADS or need an ADS token:

```bash
adsfolio pdf identify /path/to/papers --provider crossref --provider datacite
adsfolio pdf rename /path/to/papers
```

DOI lookup fills missing metadata and records its provider and file digest.
Existing metadata, manual edits, and catalog history remain intact. See
[My paper is not on ADS](../local-references.md) for the desktop steps.

The default Identify scope is `global`. For values stored with the current library,
use `--settings-scope library`. Explicit options override the selected settings for
one run without mutating them:

```bash
adsfolio pdf identify ~/Papers --settings-scope library --no-recursive
adsfolio pdf identify ~/Papers --clear-output-unresolved
```

Use `--debut-year`, `--end-year`, `--clear-debut-year`, or `--clear-end-year` to
override the global profile's prefetch window for one run. `--refresh-cache`
and `--apply` are never inherited. Supply them each time.

ADS fallback follows the saved Identify setting. Use `--allow-ads` to permit
fallback without confirmation for one run. Use `--no-allow-ads` to restore
confirmation for one run. Neither flag changes the saved value.

Library exports now include the canonical
`identify_settings` payload. Older adsfolio releases ignore that new payload but
can still read the retained `identify_profile` provenance.

To inspect the state or make a backup, run:

```bash
adsfolio cache status
adsfolio pdf export ~/Papers -o pdfs-backup.json
```

If many files remain unmatched after cache/profile matching,
`identify` asks you to approve the fallback query batches before it starts them.
It also prints an estimated count of DOI batch queries.
In non-interactive runs, `identify` skips fallback ADS queries unless the saved
setting or `--allow-ads` permits them. Query-count limits still apply.

Use `adsfolio rate-limit` to inspect the limits that ADS returns for your account.
Request allowances and reset times depend on the endpoint and server response.

If unresolved files remain at the end, `identify` writes `unresolved.csv` in the scan
root. The file contains `filename`, `relative_path`, and a `first_page_text` preview.

Existing CSV files remain unchanged. New reports receive a numeric suffix when
the default filename exists. Use the report path printed by the command.
An AI CSV selected for import is never replaced by a generated report.
`identify` also prints an LLM prompt template so you can parse metadata with your
preferred AI.

See unresolved routing (dry-run, no moves):

```bash
adsfolio pdf identify ~/Papers --recursive \
  --output-unresolved ~/Papers/unresolved
```

Move unresolved files:

```bash
adsfolio pdf identify ~/Papers --recursive \
  --output-unresolved ~/Papers/unresolved --apply

# Optional: print per-file "Moved unresolved" lines
adsfolio pdf identify ~/Papers --recursive \
  --output-unresolved ~/Papers/unresolved --apply \
  -v
```

Skip files already in the unresolved destination on later recursive runs:

```bash
adsfolio pdf identify ~/Papers --recursive \
  --output-unresolved ~/Papers/unresolved \
  --skip-unresolved-directory
```

Identify updates catalog records even without `--apply`. It may also make ADS
requests or export an unresolved CSV without `--apply`. That flag controls file
moves, including identified PDFs returning from the unresolved subtree.

Identify counts verified saved metadata as **Already identified PDFs (verified)**.
These files do not enter the **Resolved PDFs** or **Unresolved PDFs** counts
for this run. The summary also counts unreadable PDFs separately.
**No file moves planned** means there are no moves to apply. Identify suggests
`--apply` only when it previews moves.

The skip is path-based and happens before hashing or ADS work. It does not skip
unresolved-status records elsewhere. If you omit `--output-unresolved`, the
effective skipped path is `~/Papers/unresolved`. The option defaults off.

If you already generated AI hints, pass them directly:

```bash
adsfolio pdf identify ~/Papers --recursive --ai-csv ~/Papers/unresolved_ai.csv
```

## 3) Rename safely: dry-run then apply

`adsfolio pdf refresh-ads` also renames successfully updated PDFs immediately.
An update succeeds only when the record passes the
[PDF identity check](../gui.md#pdf-identity-checks). A conflict between the
record and the PDF's own arXiv stamp or metadata DOI always stops the update.
When the PDF states no identifier that can confirm the record, pass
`--confirm-identity` with `--relative-path` after you check the paper.
The flag applies to that one PDF. The confirmation is
stored with the file's SHA-256 hash and applies only to that file content and
record.
It uses the library's saved Rename settings, or global settings when no library
override exists. To use global settings explicitly, pass `--rename-scope global`.

To supply a reference manually, use `--identifier` with `--relative-path`.
Use either format:

* `{authorLastName}-{year}-{bibstem}`: an author's last name, four-digit publication
  year, and ADS journal abbreviation.
* `{bibstem}-{volume}-{page}`: ADS journal abbreviation, journal volume number, and
  starting page or article number.

You can replace `{bibstem}` with the full journal name in either format.
For example, `Last-2020-Natur` means author Last, year 2020, and journal
*Nature*. `Sci-123-45` means journal *Science*, volume 123, and starting
page 45. These illustrate the input format.

Reference hints search ADS. Candidates must match identifiers from the current
PDF. A coauthor match alone does not establish identity.
If none match, recovery uses the PDF's own DOI or arXiv ID with enabled
providers. Explicit identifier conflicts still stop the update.

`--identify-scope global` selects global provider settings for an update.
The default `library` scope uses library overrides with global fallback.
The GUI forwards its selected Identify scope automatically.
Quote references that contain spaces:

```bash
adsfolio pdf refresh-ads ~/Papers --relative-path paper.pdf --identifier 'Last-2020-Natur'
adsfolio pdf refresh-ads ~/Papers --relative-path paper.pdf --identifier 'Sci-123-45'
adsfolio pdf refresh-ads ~/Papers --relative-path paper.pdf --identifier 'The Astrophysical Journal-900-100'
```

If several ADS records match, the command lists candidates without changing
the PDF. The GUI lets you select the correct candidate.

Files under the configured unresolved directory return to the library root
with their new name. Other PDFs stay in their current directory. The
`--output-unresolved` option overrides the saved unresolved directory.

`refresh-ads` does not overwrite existing destinations. The log reports skipped
renames. Those files keep the refreshed ADS metadata. Optional citation-metrics
requests run after these renames. A metrics failure does not undo completed file
updates.

To rename only selected files, repeat `--file` with paths inside the library:

```bash
adsfolio pdf rename ~/Papers --file ~/Papers/first.pdf \
  --file ~/Papers/subfolder/second.pdf --apply
```

Omit `--apply` to preview those filenames. With `--allow-ads`, identification
also targets only those files. Failed matches keep their current paths.
PDF identity checks and destination-conflict checks still apply.
If a saved PDF changed, Rename can verify it against the cached ADS record.
It preserves the existing metadata and history when verification succeeds.
Preview does not save the recovered record. If verification fails, the log
states the reason. Review the PDF and retry with Identify or Update by ID.

Rename uses the exact selected template, including when metadata has no ADS bibcode.
It does not replace `{bibcode}` with `{year}`. If a selected field lacks metadata,
Rename skips the file and names the field. Add the metadata or choose another
template. Then retry the rename.
**Already matches rename rule** means the current filename already equals its target.
The log reports other skipped files individually.
Identification alone does not establish an ADS bibcode.

Available template variables: `{bibcode}`, `{year}`, `{title}`,
`{first_author}`, `{first_author_full}`, `{authors}`, `{authors_N}`, `{journal}`,
`{volume}`, `{page}`

The template becomes the filename. `{authors}` adds a trailing `+` when
`--max-authors` truncates the list. `{authors_N}` uses the same limited list
without that marker. Use `{{` and `}}` for literal braces.

Filename rendering follows these rules:

- Unknown fields stop the rename.
- The tool sanitizes unsafe filename characters.
- The tool appends `.pdf` automatically when it is absent.
- The tool limits results to 240 UTF-8 bytes.
- The tool skips a rename when its target already exists.

Dry-run:

```bash
adsfolio pdf rename ~/Papers \
  --scheme "{authors}_{bibcode}-{title}" \
  --author-format surname_initials \
  --max-authors 3
```

Apply:

```bash
adsfolio pdf rename ~/Papers \
  --scheme "{authors}_{bibcode}-{title}" \
  --author-format surname_initials \
  --max-authors 3 \
  --apply
```

Before it renders names, `rename --allow-ads` identifies pending PDFs recursively.
It uses the same ADS workflow as `identify`.

If the scan root contains
`unresolved/`, applying the rename also moves resolved records from that
directory into the scan root. Unresolved files stay in `unresolved/`.

To route unresolved PDFs during this identify step, pass
`--output-unresolved unresolved`. Relative paths resolve below the scan root.

Author format options:

| Value | Example output |
| --- | --- |
| `surname` | `Smith` |
| `surname_initials` (default) | `SmithJ` |
| `last_comma_first` | `Smith,_J.` |
| `ads_custom:FORMAT` | delegates to ADS export API |

## 4) Watch a folder for new PDFs

Polling (default):

```bash
adsfolio pdf watch ~/Papers --recursive --interval 10 --apply
```

Watchdog backend (requires `adsfolio[full]`):

```bash
adsfolio pdf watch ~/Papers --recursive --use-watchdog --interval 60 \
  --output-unresolved unresolved --apply
```

At startup, CLI watch reconciles the filesystem. It can process untracked or
changed PDFs that are already present. It skips catalog entries whose filesystem
facts still match. Later `--interval` passes reconcile filesystem facts to catch
missed events. These passes do not hash or re-identify unchanged PDFs.

Watch also rechecks existing catalog records with ADS at startup and every 24
hours. This check runs independently of `--auto-upgrade`. Set
`--upgrade-interval` to change the interval. The check uses the effective
Identify scope and requires ADS to be enabled with a token.

The check looks up identifiers in ADS in batches. It does not open a PDF when
ADS returns its stored bibcode. With `--apply`, it remembers failed identity
checks and unidentified PDFs without a metadata DOI for each library. It skips
those PDFs until the PDF, its saved identifiers, or the ADS record changes.

When ADS returns a verified new match or bibcode, Watch updates the record and
applies the saved Rename settings with `--apply`. Without `--apply`, Watch
previews the changes. With `filename_bibcode` enabled, a unique exact ADS match
can associate a PDF when its readable identifiers do not conflict. The filename
alone does not independently verify PDF contents. This check leaves the filename
unchanged.
A failed identity check or existing destination keeps the affected files in
place. Review the reported issue and retry.
If separate attachments in different roots share one catalog record, the
check leaves them unchanged and reports the issue for review.

This initial reconciliation differs from the GUI watcher. The GUI watcher's startup
scan establishes a baseline and leaves existing PDFs alone. The GUI only offers
later arrivals for watch processing.

With `--use-watchdog`, event callbacks only queue paths. One worker performs
PDF parsing, ADS queries, renames, and catalog writes.

Watch mode defers incomplete, encrypted, or concurrently changing PDFs until a
later event or reconciliation. Watch mode matches the `.pdf` suffix in any letter
case. It ignores symbolic links to PDFs outside the watched root.
Watch-mode unresolved destinations must be inside the watched root.

If a pass cannot read a folder or file (for example, permission denied), it logs
the error and skips that pass. It does not mark tracked files missing. The next
pass retries.

Filename rendering preserves Unicode by default, for example `Wesołowski`.
Use `--char-policy ascii` to request `Wesolowski`.

Without `--apply`, watch mode
reports proposed path changes but does not rename or move files.

The author separator and truncation suffix are independent settings.
`--max-authors` is inclusive. The tool shows a list at or below that size in full.
It truncates a longer list. With `--max-authors 2`, the tool truncates three or more
authors.

`--authors-when-truncated` controls how many leading
authors remain in a truncated result. It defaults to one. For example,
`--max-authors 2 --authors-when-truncated 1 --author-separator "+"
--author-truncation-suffix "+"` renders three authors as `SmithJ+`.
Setting `--authors-when-truncated 2` renders `SmithJ+DoeA+` instead.
`{authors_N}` uses the configured retained authors and separator but never adds
the truncation suffix.

The Rename dialog persists these values globally or per
library.
Use **Save Defaults** there to update the standard without renaming existing
PDFs.

### Run continuously on macOS

1. Create `~/Library/LaunchAgents/org.example.adsfolio-pdf-watch.plist` with the content below.
2. Replace every `/Users/YOU/...` path with an absolute path valid for your account.
   The executable is the output of `which adsfolio`.

```xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
  "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key><string>org.example.adsfolio-pdf-watch</string>
  <key>ProgramArguments</key>
  <array>
    <string>/Users/YOU/path/to/venv/bin/adsfolio</string>
    <string>pdf</string><string>watch</string>
    <string>/Users/YOU/Papers</string>
    <string>--recursive</string><string>--use-watchdog</string>
    <string>--interval</string><string>60</string>
    <string>--output-unresolved</string><string>unresolved</string>
    <string>--apply</string>
  </array>
  <key>RunAtLoad</key><true/>
  <key>KeepAlive</key><true/>
  <key>StandardOutPath</key><string>/Users/YOU/Library/Logs/adsfolio-pdf-watch.log</string>
  <key>StandardErrorPath</key><string>/Users/YOU/Library/Logs/adsfolio-pdf-watch.err</string>
</dict>
</plist>
```

Load or remove it explicitly:

```bash
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/org.example.adsfolio-pdf-watch.plist
launchctl bootout gui/$(id -u) ~/Library/LaunchAgents/org.example.adsfolio-pdf-watch.plist
```

The package does not install or start this agent automatically because the
executable path, library path, naming policy, and filesystem permissions are
user-specific.

## 5) Check current state

```bash
adsfolio pdf status ~/Papers --recursive
```

This command shows identified catalog locations, unresolved files on disk, and duplicate hash
groups. With `--recursive`, all locations share the same central catalog.

Do not run identify/watch from two machines against the same library concurrently.
For portable backups, use the export command.

## 6) Upgrade arXiv PDFs to published versions

Dry-run:

```bash
adsfolio pdf upgrade ~/Papers
```

Apply:

```bash
adsfolio pdf upgrade ~/Papers --apply
```

Overwrite tracked arXiv file with official PDF and keep backups:

```bash
adsfolio pdf upgrade ~/Papers --apply \
  --overwrite-arxiv-with-official \
  --arxiv-backup-if-overwritten ~/Papers/arxiv-backup
```

Force re-check entries already marked as published:

```bash
adsfolio pdf upgrade ~/Papers --apply --force
```

## 7) Auto-upgrade while watching

```bash
adsfolio pdf watch ~/Papers --recursive --apply \
  --auto-upgrade --upgrade-interval 24
```

## 8) Push resolved bibcodes to an ADS library

Dry-run:

```bash
adsfolio pdf push-library ~/Papers <library_id>
```

Apply:

```bash
adsfolio pdf push-library ~/Papers <library_id> --apply
```

## 9) Generate Obsidian-friendly notes

If the PDFs already live inside an Obsidian vault, `pdf obsidian` writes one Markdown note per
tracked PDF next to each file. Each note has YAML frontmatter with
title/authors/year/journal/bibcode, plus an `## Abstract` section. Notes link to the
PDF with an absolute `file://` link. `pdf obsidian` never copies, moves, or embeds PDFs.

Dry-run (no ADS queries, no files written):

```bash
adsfolio pdf obsidian ~/Papers
```

Apply — fetches missing abstracts from ADS in POST batches of up to 2,000
bibcodes, caches them in the central catalog, and writes the notes. Bibcodes
travel in the request body, so large libraries do not need long query URLs:

```bash
adsfolio pdf obsidian ~/Papers --apply
```

The export reads PDF highlights, underlines, strikeouts, squiggles, and text
comments. It places them in a marked `PDF annotations` section. The export
updates that section when annotations change. It preserves other note text.
It does not rewrite a note when the output is unchanged. Image-only highlights
may have no selectable text. Drawings and ink are not converted to text.

The export requires PyMuPDF from the `full` extra. An unreadable PDF or a note
with invalid section markers produces a skip. Check the warning before retrying.

ADSfolio stages the complete note before replacing the original. A write failure
keeps the original note intact. A note changed during export produces a skip.
Review the job log before retrying.

To regenerate the entire note, pass `--force`. This replaces manual edits
outside the annotation section:

```bash
adsfolio pdf obsidian ~/Papers --apply --force
```

Write notes to a separate folder instead of colocating them with the PDFs:

```bash
adsfolio pdf obsidian ~/Papers --apply --notes-dir ~/Papers/notes
```

For an Obsidian vault elsewhere, choose a folder such as
`--notes-dir ~/MyVault/papers`. The `papers` subfolder keeps generated notes
separate from notes that you write. It is optional and does not change ADS
requests.

## Next

Continue with [Tutorial 5: Operations and troubleshooting](./05-operations-and-troubleshooting.md).