# Maintaining ADSfolio ## Keep documentation in sync Apply the ADHD and ASD-STE100 skills to documentation, docstrings, and release notes. Follow `AGENTS.md` for the required style and source-preservation rules. When code changes, update docs in the same commit. 1. Update the relevant docs pages. 2. If CLI behavior changes, update `docs/cli.rst`. 3. If you add or delete public API modules, update `docs/api/index.rst`. ## Lookup and document invariants 1. Treat bibliographic references as search hints. Require current PDF evidence before applying a candidate. Never replace an explicit conflicting ID automatically. 2. Keep provider order consistent across manual updates, Identify, and the watcher. A rejected candidate does not prevent a later provider from supplying verified metadata. A failed service request does not establish that a record is missing. 3. Read complete identifier results before deciding uniqueness. Stop changed or incomplete pages. Cache provider documents independently of PDF associations. 4. Use exclusive file moves and downloads. Roll moves back after catalog failure. Mark an external move's old location missing while retaining attachment history. 5. Stage notes and private settings before replacement. Recheck existing note contents before publication. Create CSV reports exclusively and preserve selected inputs. `pdf.reference_lookup` owns reference recovery and manual provider ordering. The identity module retains the shared mutation checks. Neither a cached document nor a filename supplies proof of PDF identity. An exact unique ADS filename match has separate filename-association provenance. Reject conflicting PDF IDs. Private settings saves use independent temporary files. Final replacements run in sequence within one process to avoid concurrent Windows move failures. A failed replacement preserves the existing file and removes the temporary file. The caller can report the error and retry the save. All rename paths use the exact selected template. Do not replace missing fields with other fields or select an implicit alternate template. ## Release checks and API boundaries For changed Python files, run these commands in this order: ```bash uv run --no-sync isort PATHS uv run --no-sync ruff format PATHS uv run --no-sync ruff check PATHS uv run --no-sync pytest tests/RELEVANT_TEST_FILE.py ``` The blocking lint gate explicitly selects `E4,E7,E9,F`. Import order belongs to isort, and formatting belongs to Ruff. Full Ruff checks remain useful during development. But their existing findings need individual review before you expand the gate. To set the release version: 1. Set the version in `src/adsfolio/__init__.py`. 2. Run `uv lock`. Setuptools reads that same version for build metadata. The package-check workflow runs tests on Python 3.10–3.13, builds documentation and distributions, and installs the wheel without optional extras. The desktop workflow checks native bundles separately. It publishes tagged releases after all required checks pass. The Python client, documented domain functions, and CLI commands are for direct use. GUI widgets, job objects, and command implementation modules are implementation interfaces. Their API pages aid maintenance. They do not promise stable constructor or callback signatures. Existing registry aliases remain available for compatibility. `REGISTRY_FILENAME` names the legacy per-directory database. `CATALOG_FILENAME` names the central catalog. ## Notification layout Keep queued notices and view guidance outside the central pane layout. `NotificationPanel` displays them in a scrollable overlay. The permanent toolbar button displays the queue count and severity. Queue changes must not open the panel automatically. Closing it must not dismiss messages. Keep the panel open across action dialogs and focus changes. Render every queued notice with controls bound to that notice's payload. Use explicit close controls rather than the automatic dismissal of `Qt.Popup`. Measure sidebar, table, and detail geometry across view changes, long warnings, and queue dismissal. Preserve table selection and scroll position when notices change. Keep candidate actions bound to their notice after moving the controls. **Verify This ID** fetches and validates provider metadata without manual confirmation. Bind that check to the same PDF digest and candidate fingerprint used by reviewed candidates. Reserve **Use This ID** for candidates that permit manual confirmation. For stored ADS candidates, verify locally first. If that fails without a conflict, fetch the exact candidate through the enabled ADS provider and repeat the existing identity checks. Do not request manual confirmation for a successful automatic check. Persist repaired proof even when an obsolete proof key already exists. Reconcile a relocated catalog path before saving its repaired proof. Keep exact-record verification separate from broader fallback-search permission. ## Automatic recheck process Run the automatic identifier recheck in a child process. PDF parsing in a GUI thread holds the Python interpreter lock. Each table paint then waits for the lock, and scrolling lags. 1. `AdsRecheckJob` writes a request file with `build_request`. 2. It starts the hidden `adsfolio pdf recheck-worker REQUEST` command. 3. The worker sets `Config.config_dir_override` to the app's directory. 4. The worker writes the result file with `write_result`. 5. The job reads the result and removes its temporary directory. Cancel creates the request's cancel file. The worker checks it between steps. It stores each remembered failure when it finds it, and it prunes old entries only after a complete, uncancelled scan. Do not kill the worker during a catalog write. The window never starts a cycle by itself. `_offer_ads_recheck` asks at startup and on the 24-hour timer. **Tools → Check for ADS Updates** starts a cycle directly. A library added during a running cycle joins it. Otherwise, adding a library starts nothing. A normal cycle skips a resolved PDF when ADS returns its stored bibcode. Only `retry_remembered`, from **Recheck PDFs That Need Review…**, reads such a PDF to add a missing identity proof. Tests run `run_request` in a thread through the same files. Mark a test with `recheck_process` to start the real child process. The `no_recheck_offer` fixture hides the start question. Mark a test with `recheck_offer` to show it. ## Rename preview process Build the Rename preview in a child process, so the window never waits for PDF reads. `batch_rename` renders each target name from stored metadata first. It hashes and checks a PDF only when the name would change and the target name is free. Keep that order: the identity check guards a rename, not a listing. 1. `RenamePreviewJob` writes a request with `rename_preview.build_request`. 2. It starts the hidden `adsfolio pdf rename-preview REQUEST` command. 3. The worker writes plans, skipped files, and refused identities as JSON. `JsonWorkerJob` in `gui/jobs.py` holds the shared process, file, and result logic for both workers. A preview changes no files, so `RenamePreviewJob.cancel` kills the process. Tests build previews in-process through the same files. Mark a test with `rename_preview_process` to start the real child process. ## Documentation Check Run this before committing: ```bash uv sync --extra gui --group docs bash scripts/docs_check.sh ``` This installs the docs group and PySide6 in the project's `.venv`. The API reference imports GUI modules without launching the application. To build without a separate setup step: ```bash uv run --extra gui --group docs sphinx-build -n -W --keep-going -b html docs docs/_build/html ``` For an already activated shared environment, avoid project synchronization. Layer the documentation requirements temporarily instead: ```bash uv run --active --no-sync --with-requirements docs/requirements.txt sphinx-build -n -W --keep-going -b html docs docs/_build/html ``` The build treats warnings as errors. Fix documentation warnings before you commit. The build refreshes the API stubs with `docs/_templates/autosummary/module.rst`. Edit that template or `docs/api/index.rst`, not the generated stubs. Sphinx also fetches Python, Click, and Qt cross-reference inventories. Network failures can prevent a strict build even when local pages are valid. Report such failures separately from documentation errors. Do not suppress all reference warnings to pass a build. Read the [Sphinx autodoc options](https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html) before you change API member selection. Re-export packages link to implementation pages. The registry page documents the central classes and explains legacy aliases. Keep internal executable entry points out of recursive discovery. ## Current Limitations and Follow-ups **Citation and text formatting** - Citation formatting converts title and publication HTML to plain text but leaves `bibstem` unchanged. Defensive normalization remains a follow-up. - `html_to_latex` converts nested ``, ``, ``, ``, ``, and `` elements. `format_title_latex` also uses it for titles. Existing `$...$` math stays unchanged outside subscript and superscript elements. Directly nested subscripts and superscripts share one math group. Unsupported tags and unmatched tags remain literal text. It decodes HTML entities once. - Markdown CV output reuses the LaTeX title conversion, so titles can contain LaTeX math (`$_{2}$`) and escapes (`\&`). It is not plain text. - In `format_author_list_cv`, `use_and` only decides whether the final conjunction appears (`None` omits it). The conjunction itself is always the word "and". The function does not insert the text of `use_and`, so the default `\&` does not appear in the output. `me_fmt` and LaTeX escaping of author names still apply. **Catalog and file identity** - File identifiers outside SQLite's signed 64-bit range use hexadecimal text internally and return as Python integers. Existing integer records remain valid. Windows discovery uses `os.stat` because `DirEntry.stat` omits device/inode IDs. - Catalog matching by filesystem identity is limited because operating systems reuse inode numbers. An observation matches an active path first, then a device and inode, then a content digest. - Some inode matches have a stored location with a known digest. Matching accepts such a match only if the incoming digest is known and equal. Otherwise, the file can still rejoin through the digest lookup. - `discover_root` hashes a file only when both of these conditions are true. The file sits at a path with no active location. A stored location with the same inode has a known digest. It then hashes the file once, before any catalog write. Unchanged files are never hashed. A read error other than a vanished file aborts the scan before the catalog changes. - Matching cannot check a stored location without a digest. Such a location matches by inode alone, and a reused inode can inherit that attachment. - Change checks compare size, modification time, and inode. They do not compare the device number, because macOS can give a volume a new device number after a restart. `file_facts_changed` holds this rule for `discover_root`, `reconcile_file_facts`, and the command-line watcher. - `discover_root` marks a known path for the watcher only when the path is new or its SHA-256 changed. A row without a stored digest uses the fact check. - The watcher's startup pass takes only watch-marked or changed rows. A metadata-stale row alone does not qualify, because a retry gives the same result. After a deterministic failure (an identity refusal or a move conflict), `PdfWatcher` clears the mark. Network and file errors keep it. - The watch worker queues each request and processes one PDF per event-loop turn. A pause request from a foreground job runs before the next PDF. - Earlier versions marked every PDF after a new device number. A one-time repair (`repair:watch-pending-device-number` in `catalog_migrations`) clears a mark when the current content version is older than the mark. **GUI file table** - `pdf.classification` derives current identification groups without changing the stored ADS lookup status or rename permissions. Non-ADS identification requires accepted proof bound to the current content hash and identifier fingerprint. Registry queries apply the `identification` filter before sorting and pagination. Counts use the same classifier, including in combined libraries. Changed records need review. Usable metadata alone does not establish identity. - `FileTableModel` runs the registry query once per reset (`set_query`, `set_registry`, reload) and keeps the full sorted result in memory. It shows `PAGE_SIZE` (500) rows at a time. `fetchMore` slices that snapshot without querying again, so rows added after a reset appear only after the next reset. Memory grows with the number of matching records. If a query fails because the library is locked, the table becomes empty and the status bar reports it. - When you omit `parent`, `FileTableModel` recognizes only lists and tuples as the `columns` argument. It interprets other sequences as the legacy positional parent. Pass columns as a list or tuple. ## Deferred design Annotation review saves a separate PDF. For library PDFs, saving the copy inside an open library folder registers it with the canonical PDF's metadata. After a complete, verified merge, the checked default moves unchanged inputs to Trash. Users can clear that option before saving to keep inputs in place. Partial merges keep all inputs. Cleanup failures retain failed inputs and open the retry dialog. Catalog history remains available. See the [annotation workflow](gui.md) for supported types and limits. Catalog consolidation and unified TOML configuration remain future work. They must preserve these requirements: - Establish work identity from exact bibliographic identifiers. Title similarity and ADS result order must not authorize a merge. Preserve all identifiers. - Let the user choose the canonical PDF. Transfer text markup automatically only when exact normalized text and its target geometry are unique. Require manual review for ambiguous text and geometry-based annotations. Preserve subtype, appearance, and comments. Each annotation needs an outcome or explicit waiver. - Check a staged PDF before replacement. Journal filesystem and catalog updates together. Retain originals. Provide rollback and explicit recovery paths. Never delete obsolete inputs automatically. - Use provenance to prevent duplicate transfers. A dual-pane review should show one annotation at a time. - Use one typed configuration service for GUI and CLI, with global and per-library scope, explicit precedence, checked atomic writes, and recoverable migration. Keep session restoration global. Additional limits: - Uniform-font OCR or mixed-size titles can require a verified identifier. - Title candidates require ADS verification and the PDF identity check below. - Downloads serialize session calls while they stream bodies concurrently. Callers must not mutate or close the client during active downloads. ## PDF identity checks `adsfolio.pdf.identity` decides whether a PDF may be associated with a record. Every code path that writes publication metadata for a PDF or renames it must call `assert_mutation_allowed` on the current file. Store its returned provenance under `IDENTITY_KEY`. ### Accepted evidence - Direct evidence includes the page-1 arXiv stamp and DOI properties in DocInfo and XMP. A complete labeled DOI or DOI URL in DocInfo Title also counts. A DOI mentioned within an ordinary title does not count. - Subject evidence requires three agreements. Subject must state one labeled DOI. The first page must repeat it with exact DOI boundaries. DocInfo Title must equal the provider title after formatting normalization. This includes case, whitespace, canonical Unicode, quotation marks, and spaced subtitle separators. Words, numbers, and other punctuation remain significant. The provider must explicitly list the same DOI. A new verified result needs `provider_document`, a fresh provider record. It must list every identifier that the proposed record and the PDF claim. Aliases that appear only in the proposed metadata establish nothing. Without a provider record, ADSfolio requires stored verified provenance. That provenance must match the same file hash, identifiers, and PDF evidence. ### Refused changes - A PDF identifier that contradicts the record raises `IdentityConflictError`. No flag or stored confirmation overrides it. - Missing or unconfirmable evidence raises `UnverifiedIdentityError`. Automatic actions must stop there. A warning from optional XMP does not invalidate readable matching identifiers or corroborated Subject evidence. `check_document_doi_evidence` reports which corroboration check failed. Include that reason in the refusal. Do not report missing identifiers when the PDF contains a DOI that failed a different check. `UnverifiedIdentityError.candidate_bibcode` carries the proposed ADS record. The watcher passes it through `WatchOutcome` to the notice's **View ADS Entry** action. Keep this candidate separate from the resolved bibcode. Do not extract identifiers from error prose or use the currently selected row for this link. **Use This ID** is explicit consent for a manually confirmable candidate. Pass the inspected PDF hash and candidate identifier fingerprint to the refresh command. Recheck both before applying publication metadata. Keep the notice until the requested association succeeds. Hover and keyboard focus display icon tooltips without the platform delay. - An unreadable file, a file that changes while it is read, or a hash that differs from the caller's expected hash raises `IdentityValidationError`. ### Manual confirmation and stored provenance Explicit manual confirmation admits a PDF without a readable own identifier. It also admits hand-entered metadata without a stable identifier. It cannot override an own identifier that the provider does not corroborate. Its stored stamp binds the file's SHA-256 hash and the record fingerprint. A changed file or different record invalidates that stamp. Records without stored provenance receive another check before updates or renames. Recovered XMP identifiers must use explicit namespace declarations. An unrelated undeclared prefix can produce a warning without hiding valid identifier fields. Readable identifier conflicts still block the change. File names, hashes, result order, and title or author similarity alone do not establish identity. Renames change a path. Metadata updates change the catalog. Neither operation rewrites PDF contents. The check confirms stated identifiers only. It does not guarantee that every PDF can be identified or that its content is the paper. ### Repairing wrong associations `scripts/repair_pdf_identities.py` reattaches PDFs that a catalog links to the wrong paper. It reads a JSON manifest. Each entry names the PDF path, its SHA-256 hash, the location and attachment IDs, optionally the wrong document ID, the metadata the catalog must still hold, and the full ADS record of the correct paper. A verified DOI-provider record can supply the correct metadata when ADS has no record. Its ADS link remains empty. Without `--apply` or `--undo`, the script only prints the plan. ```sh uv run python scripts/repair_pdf_identities.py --catalog CATALOG --root ROOT \ --manifest manifest.json --rename-settings rename.json uv run python scripts/repair_pdf_identities.py --catalog CATALOG --root ROOT \ --manifest manifest.json --rename-settings rename.json --apply --journal-dir DIR uv run python scripts/repair_pdf_identities.py --catalog CATALOG --undo DIR/repair-ID.json ``` Close ADSfolio before apply or undo. Stop every watcher. The repair requires exclusive catalog access. Its writer check does not hold a lock across the whole operation. A separate operating-system lock prevents concurrent repairs in the same journal directory. ADSfolio flushes journal data before atomic replacement. POSIX also flushes the parent directory. Windows has no equivalent directory flush through Python's file API. Pause file sync for the library as well. The manifest's `proposed_ads_record` serves as the provider record for the identity check, so build it from a fresh ADS response and do not edit it by hand. Before it changes any PDF or association, the plan checks every entry. It refuses an entry if the hash, IDs, or metadata differ from the manifest, or if the attachment has several active locations. It also refuses stale metadata and managed local references. The PDF's own identifiers must verify the correct record. For a PDF with parser errors, a reviewed manifest can instead list the exact `reviewed_parser_errors` and an `audit_reference`. Every readable own identifier must still agree with the provider. The result receives `reviewed_repair` status. That status alone does not permit automatic updates or renames. Optional XMP warnings with independently confirming identifiers need no manual audit. No PDF bytes are changed to remove parser errors. Locked automatic fields block an entry unless the manifest lists them in `authorized_locked_fields`. Apply backs up the catalog and each PDF, writes a journal, and never overwrites a file or changes PDF bytes. A failure reverts the recorded steps. Undo refuses to run if a repaired file or catalog row changed after the repair. ## Publishing releases Remote-changing commands print a preview unless you supply `--apply`. The repository scripts prepare GitHub repositories, native application releases, a Homebrew tap, and a static website. Installing the package publishes nothing. ### Keep documentation and media current Sphinx reads `release` and `version` from `adsfolio.__version__`. In Markdown, use ``{sub-ref}`release` `` for a current version. In reStructuredText, use `|release|`. The website builder replaces `__ADSFOLIO_VERSION__` from the same package version. Release builds update these labels without editing page text. Keep migration boundaries, OS requirements, and changelog versions fixed. Use `VERSION` for commands that require the next release number as input. After changing visible controls, regenerate the affected screenshots and videos: ```sh uv run --no-sync python scripts/capture_docs_media.py ``` The renderer uses temporary configuration and fictional papers. It does not capture the user's desktop or library. It preserves files when their contents are unchanged. Inspect each changed screenshot and each video before publication. ## Repository setup 1. Install Git, [uv](https://docs.astral.sh/uv/), and the [GitHub CLI](https://cli.github.com/). 2. Authenticate with `gh auth login`. 3. Preview the repository setup, then apply it: ```sh uv run --no-sync python scripts/release.py repository uv run --no-sync python scripts/release.py repository --apply ``` The default repository is `ysBach/adsfolio`. For a fork, use `--repository OWNER/NAME`. The script initializes Git only when needed. It keeps existing history. An existing `origin` becomes `previous-origin`, then the new repository becomes `origin`. If `origin` already points to the requested repository, rerunning setup verifies that repository without changing remotes or creating it again. Otherwise, creation stops if the remote repository already exists. For an existing repository, inspect it and configure `origin` manually instead of creating it again. The script does not stage files, commit changes, or push history. A newly created GitHub repository therefore remains empty until the first push. Before the first commit: 1. Review the working tree, including untracked files. 2. Keep credentials, catalogs, personal PDFs, build outputs, and local reports out of Git. 3. After review, commit the intended files. 4. Push `main` to the new remote. After you verify the new repository, you can delete the old Git remote. ## Prepare and publish a version **Prepare the version** 1. Set the version. Replace `VERSION` with the next release version: ```sh uv run --no-sync python scripts/release.py version VERSION --apply ``` 2. Update `CHANGELOG.md`. 3. Check the package: ```sh uv run --no-sync python scripts/release.py check ``` **Test the bundles** 1. Run the **Desktop bundles** workflow manually. 2. Inspect all four platform results. 3. Test the downloads on clean target systems. Complete code signing and dependency-license review before you claim production readiness. **Tag the release** 1. Commit the reviewed version, lockfile, and documentation. The release script requires a clean working tree on `main` with the intended `origin`. 2. Preview and create the release tag: ```sh uv run --no-sync python scripts/release.py tag uv run --no-sync python scripts/release.py tag --apply ``` The script repeats local checks, creates an annotated tag, and pushes `main` and that tag atomically. It uses no force push. If the push fails, inspect the local tag before you retry. The script never deletes a tag automatically. Tag workflows check exact version equality, run package checks, and build macOS ARM64, macOS Intel, Windows x86-64, and Linux x86-64 archives. They run frozen application smoke tests, verify checksums, and create a GitHub Release with Python distributions and desktop downloads. They mark development and prerelease versions as prereleases. Stable releases also include `adsfolio.rb` for Homebrew. The workflows do not overwrite existing release assets. GitHub-hosted runners cannot establish every supported desktop configuration. See [Building the desktop app](desktop.md) for acceptance checks. This repository does not include signing and notarization credentials. ## Optional PyPI publication 1. Configure a PyPI trusted publisher for the project `adsfolio`, the repository, workflow `desktop.yml`, and environment `pypi`. 2. Create that GitHub environment with your chosen reviewer rules. 3. Set the repository variable `PUBLISH_PYPI=true`. The workflow publishes the already-built wheel and source archive only after the GitHub Release job succeeds. It uses short-lived OIDC credentials, not a stored API token. Check project-name availability and publisher setup on PyPI. Repository scripts do not reserve a name. ## Homebrew tap Run this after a stable GitHub Release exists. Ruby must be available for the cask syntax check. Git commits use your configured Git identity. ```sh uv run --no-sync python scripts/publish_tap.py VERSION --create uv run --no-sync python scripts/publish_tap.py VERSION --create --apply ``` This creates `ysBach/homebrew-adsfolio`, downloads the release checksums, generates `Casks/adsfolio.rb`, checks its Ruby syntax, and commits and pushes the cask. For later releases, omit `--create`. For other owners, use `--repository` and `--tap`. A cask requires checksums for both native macOS archives. The script uses no placeholder checksum or `:no_check`. Do not publish a cask before native acceptance. Users can then run: ```sh brew install --cask ysBach/adsfolio/adsfolio brew upgrade --cask adsfolio ``` ## Website and documentation The static landing page lives in `website/`. It has no build-time JavaScript framework, tracking scripts, or external fonts. The build includes strict Sphinx documentation under `docs/` and works beneath a repository URL prefix. ```sh uv sync --locked --extra gui --group docs uv run --no-sync python scripts/build_site.py uv run --no-sync python -m http.server 8000 --directory _site ``` Open `http://localhost:8000`. Before you deploy publicly, update the first-release notice and download status to match the artifacts actually available. To publish the website: 1. In GitHub **Settings → Pages**, select **GitHub Actions** as the source. 2. Set `PUBLISH_PAGES=true` in repository variables. 3. Run **Documentation website**. The expected project URL is `https://ysbach.github.io/adsfolio/`. Pull requests build the site but cannot deploy it. Until you configure Pages and the variable, website builds remain available as workflow artifacts without publishing. ## Data directories and the full rename Update Python imports and scripts to the new name. The public package namespace, CLI, application identifiers, and new storage paths use `adsfolio`. The package does not install earlier command aliases. `ADS_API_TOKEN` still names the credential for the NASA ADS service, not the application. On first use, the application copies earlier configuration and cache directories into the ADSfolio directories if the destination does not exist. SQLite uses the backup API, including committed WAL data. Directory migration never merges or overwrites an existing destination. Current versions no longer convert reserved metadata keys from the earlier application. Copying an unconverted catalog retains its data, but manual field locks, author text, and local references under those keys are not honoured. Provider refreshes can overwrite fields whose earlier locks are not recognised. Before upgrading from the earlier application: 1. Close the earlier application. 2. Open ADSfolio **0.6.0** once to convert the catalog and import its per-library files. Verify the metadata, manual locks, and local references there. 3. Install ADSfolio {sub-ref}`release`. Keep the original directories until you verify the converted library. If you already used ADSfolio 0.6.0, the catalog conversion is complete. Current versions do not read per-library files under the earlier application name. ADSfolio-named per-library sidecars and registries remain readable. The rename keeps old Git history. It does not rewrite commits or change bibliographic user text.