Maintaining ADSfolio

Keep documentation in sync 

Apply the ADHD and ASD-STE100 skills to documentation, docstrings, and release notes. Follow AGENTS.md for the required style and source-preservation rules.

When code changes, update docs in the same commit.

Update the relevant docs pages.
If CLI behavior changes, update docs/cli.rst.
If you add or delete public API modules, update docs/api/index.rst.

Lookup and document invariants 

Treat bibliographic references as search hints. Require current PDF evidence before applying a candidate. Never replace an explicit conflicting ID automatically.
Keep provider order consistent across manual updates, Identify, and the watcher. A rejected candidate does not prevent a later provider from supplying verified metadata. A failed service request does not establish that a record is missing.
Read complete identifier results before deciding uniqueness. Stop changed or incomplete pages. Cache provider documents independently of PDF associations.
Use exclusive file moves and downloads. Roll moves back after catalog failure. Mark an external move’s old location missing while retaining attachment history.
Stage notes and private settings before replacement. Recheck existing note contents before publication. Create CSV reports exclusively and preserve selected inputs.

pdf.reference_lookup owns reference recovery and manual provider ordering. The identity module retains the shared mutation checks. Neither a cached document nor a filename supplies proof of PDF identity. An exact unique ADS filename match has separate filename-association provenance. Reject conflicting PDF IDs.

Private settings saves use independent temporary files. Final replacements run in sequence within one process to avoid concurrent Windows move failures. A failed replacement preserves the existing file and removes the temporary file. The caller can report the error and retry the save.

All rename paths use the exact selected template. Do not replace missing fields with other fields or select an implicit alternate template.

Release checks and API boundaries 

For changed Python files, run these commands in this order:

uv run --no-sync isort PATHS
uv run --no-sync ruff format PATHS
uv run --no-sync ruff check PATHS
uv run --no-sync pytest tests/RELEVANT_TEST_FILE.py

The blocking lint gate explicitly selects E4,E7,E9,F. Import order belongs to isort, and formatting belongs to Ruff. Full Ruff checks remain useful during development. But their existing findings need individual review before you expand the gate.

To set the release version:

Set the version in src/adsfolio/__init__.py.
Run uv lock.

Setuptools reads that same version for build metadata.

The package-check workflow runs tests on Python 3.10–3.13, builds documentation and distributions, and installs the wheel without optional extras. The desktop workflow checks native bundles separately. It publishes tagged releases after all required checks pass.

The Python client, documented domain functions, and CLI commands are for direct use. GUI widgets, job objects, and command implementation modules are implementation interfaces. Their API pages aid maintenance. They do not promise stable constructor or callback signatures.

Existing registry aliases remain available for compatibility. REGISTRY_FILENAME names the legacy per-directory database. CATALOG_FILENAME names the central catalog.

Notification layout 

Keep queued notices and view guidance outside the central pane layout. NotificationPanel displays them in a scrollable overlay. The permanent toolbar button displays the queue count and severity. Queue changes must not open the panel automatically. Closing it must not dismiss messages. Keep the panel open across action dialogs and focus changes. Render every queued notice with controls bound to that notice’s payload. Use explicit close controls rather than the automatic dismissal of Qt.Popup.

Measure sidebar, table, and detail geometry across view changes, long warnings, and queue dismissal. Preserve table selection and scroll position when notices change. Keep candidate actions bound to their notice after moving the controls.

Verify This ID fetches and validates provider metadata without manual confirmation. Bind that check to the same PDF digest and candidate fingerprint used by reviewed candidates. Reserve Use This ID for candidates that permit manual confirmation.

For stored ADS candidates, verify locally first. If that fails without a conflict, fetch the exact candidate through the enabled ADS provider and repeat the existing identity checks. Do not request manual confirmation for a successful automatic check. Persist repaired proof even when an obsolete proof key already exists. Reconcile a relocated catalog path before saving its repaired proof. Keep exact-record verification separate from broader fallback-search permission.

Automatic recheck process 

Run the automatic identifier recheck in a child process. PDF parsing in a GUI thread holds the Python interpreter lock. Each table paint then waits for the lock, and scrolling lags.

AdsRecheckJob writes a request file with build_request.
It starts the hidden adsfolio pdf recheck-worker REQUEST command.
The worker sets Config.config_dir_override to the app’s directory.
The worker writes the result file with write_result.
The job reads the result and removes its temporary directory.

Cancel creates the request’s cancel file. The worker checks it between steps. It stores each remembered failure when it finds it, and it prunes old entries only after a complete, uncancelled scan. Do not kill the worker during a catalog write.

The window never starts a cycle by itself. _offer_ads_recheck asks at startup and on the 24-hour timer. Tools → Check for ADS Updates starts a cycle directly. A library added during a running cycle joins it. Otherwise, adding a library starts nothing.

A normal cycle skips a resolved PDF when ADS returns its stored bibcode. Only retry_remembered, from Recheck PDFs That Need Review…, reads such a PDF to add a missing identity proof.

Tests run run_request in a thread through the same files. Mark a test with recheck_process to start the real child process. The no_recheck_offer fixture hides the start question. Mark a test with recheck_offer to show it.

Rename preview process 

Build the Rename preview in a child process, so the window never waits for PDF reads. batch_rename renders each target name from stored metadata first. It hashes and checks a PDF only when the name would change and the target name is free. Keep that order: the identity check guards a rename, not a listing.

RenamePreviewJob writes a request with rename_preview.build_request.
It starts the hidden adsfolio pdf rename-preview REQUEST command.
The worker writes plans, skipped files, and refused identities as JSON.

JsonWorkerJob in gui/jobs.py holds the shared process, file, and result logic for both workers. A preview changes no files, so RenamePreviewJob.cancel kills the process. Tests build previews in-process through the same files. Mark a test with rename_preview_process to start the real child process.

Documentation Check 

Run this before committing:

uv sync --extra gui --group docs
bash scripts/docs_check.sh

This installs the docs group and PySide6 in the project’s .venv. The API reference imports GUI modules without launching the application. To build without a separate setup step:

uv run --extra gui --group docs sphinx-build -n -W --keep-going -b html docs docs/_build/html

For an already activated shared environment, avoid project synchronization. Layer the documentation requirements temporarily instead:

uv run --active --no-sync --with-requirements docs/requirements.txt sphinx-build -n -W --keep-going -b html docs docs/_build/html

The build treats warnings as errors. Fix documentation warnings before you commit. The build refreshes the API stubs with docs/_templates/autosummary/module.rst. Edit that template or docs/api/index.rst, not the generated stubs.

Sphinx also fetches Python, Click, and Qt cross-reference inventories. Network failures can prevent a strict build even when local pages are valid. Report such failures separately from documentation errors. Do not suppress all reference warnings to pass a build.

Read the Sphinx autodoc options before you change API member selection. Re-export packages link to implementation pages. The registry page documents the central classes and explains legacy aliases. Keep internal executable entry points out of recursive discovery.

Current Limitations and Follow-ups 

Citation and text formatting

Citation formatting converts title and publication HTML to plain text but leaves bibstem unchanged. Defensive normalization remains a follow-up.
html_to_latex converts nested , , , , , and  elements. format_title_latex also uses it for titles. Existing $...$ math stays unchanged outside subscript and superscript elements. Directly nested subscripts and superscripts share one math group. Unsupported tags and unmatched tags remain literal text. It decodes HTML entities once.
Markdown CV output reuses the LaTeX title conversion, so titles can contain LaTeX math ( $_{2}$ ) and escapes (\&). It is not plain text.
In format_author_list_cv, use_and only decides whether the final conjunction appears (None omits it). The conjunction itself is always the word “and”. The function does not insert the text of use_and, so the default \& does not appear in the output. me_fmt and LaTeX escaping of author names still apply.

Catalog and file identity

File identifiers outside SQLite’s signed 64-bit range use hexadecimal text internally and return as Python integers. Existing integer records remain valid. Windows discovery uses os.stat because DirEntry.stat omits device/inode IDs.
Catalog matching by filesystem identity is limited because operating systems reuse inode numbers. An observation matches an active path first, then a device and inode, then a content digest.
Some inode matches have a stored location with a known digest. Matching accepts such a match only if the incoming digest is known and equal. Otherwise, the file can still rejoin through the digest lookup.
discover_root hashes a file only when both of these conditions are true. The file sits at a path with no active location. A stored location with the same inode has a known digest. It then hashes the file once, before any catalog write. Unchanged files are never hashed. A read error other than a vanished file aborts the scan before the catalog changes.
Matching cannot check a stored location without a digest. Such a location matches by inode alone, and a reused inode can inherit that attachment.
Change checks compare size, modification time, and inode. They do not compare the device number, because macOS can give a volume a new device number after a restart. file_facts_changed holds this rule for discover_root, reconcile_file_facts, and the command-line watcher.
discover_root marks a known path for the watcher only when the path is new or its SHA-256 changed. A row without a stored digest uses the fact check.
The watcher’s startup pass takes only watch-marked or changed rows. A metadata-stale row alone does not qualify, because a retry gives the same result. After a deterministic failure (an identity refusal or a move conflict), PdfWatcher clears the mark. Network and file errors keep it.
The watch worker queues each request and processes one PDF per event-loop turn. A pause request from a foreground job runs before the next PDF.
Earlier versions marked every PDF after a new device number. A one-time repair (repair:watch-pending-device-number in catalog_migrations) clears a mark when the current content version is older than the mark.

GUI file table

pdf.classification derives current identification groups without changing the stored ADS lookup status or rename permissions. Non-ADS identification requires accepted proof bound to the current content hash and identifier fingerprint. Registry queries apply the identification filter before sorting and pagination. Counts use the same classifier, including in combined libraries. Changed records need review. Usable metadata alone does not establish identity.
FileTableModel runs the registry query once per reset (set_query, set_registry, reload) and keeps the full sorted result in memory. It shows PAGE_SIZE (500) rows at a time. fetchMore slices that snapshot without querying again, so rows added after a reset appear only after the next reset. Memory grows with the number of matching records. If a query fails because the library is locked, the table becomes empty and the status bar reports it.
When you omit parent, FileTableModel recognizes only lists and tuples as the columns argument. It interprets other sequences as the legacy positional parent. Pass columns as a list or tuple.

Deferred design 

Annotation review saves a separate PDF. For library PDFs, saving the copy inside an open library folder registers it with the canonical PDF’s metadata. After a complete, verified merge, the checked default moves unchanged inputs to Trash. Users can clear that option before saving to keep inputs in place. Partial merges keep all inputs. Cleanup failures retain failed inputs and open the retry dialog. Catalog history remains available. See the annotation workflow for supported types and limits.

Catalog consolidation and unified TOML configuration remain future work. They must preserve these requirements:

Establish work identity from exact bibliographic identifiers. Title similarity and ADS result order must not authorize a merge. Preserve all identifiers.
Let the user choose the canonical PDF. Transfer text markup automatically only when exact normalized text and its target geometry are unique. Require manual review for ambiguous text and geometry-based annotations. Preserve subtype, appearance, and comments. Each annotation needs an outcome or explicit waiver.
Check a staged PDF before replacement. Journal filesystem and catalog updates together. Retain originals. Provide rollback and explicit recovery paths. Never delete obsolete inputs automatically.
Use provenance to prevent duplicate transfers. A dual-pane review should show one annotation at a time.
Use one typed configuration service for GUI and CLI, with global and per-library scope, explicit precedence, checked atomic writes, and recoverable migration. Keep session restoration global.

Additional limits:

Uniform-font OCR or mixed-size titles can require a verified identifier.
Title candidates require ADS verification and the PDF identity check below.
Downloads serialize session calls while they stream bodies concurrently. Callers must not mutate or close the client during active downloads.

PDF identity checks 

adsfolio.pdf.identity decides whether a PDF may be associated with a record. Every code path that writes publication metadata for a PDF or renames it must call assert_mutation_allowed on the current file. Store its returned provenance under IDENTITY_KEY.

Accepted evidence

Direct evidence includes the page-1 arXiv stamp and DOI properties in DocInfo and XMP. A complete labeled DOI or DOI URL in DocInfo Title also counts. A DOI mentioned within an ordinary title does not count.
Subject evidence requires three agreements. Subject must state one labeled DOI. The first page must repeat it with exact DOI boundaries. DocInfo Title must equal the provider title after formatting normalization. This includes case, whitespace, canonical Unicode, quotation marks, and spaced subtitle separators. Words, numbers, and other punctuation remain significant. The provider must explicitly list the same DOI.

A new verified result needs provider_document, a fresh provider record. It must list every identifier that the proposed record and the PDF claim. Aliases that appear only in the proposed metadata establish nothing. Without a provider record, ADSfolio requires stored verified provenance. That provenance must match the same file hash, identifiers, and PDF evidence.

Refused changes

A PDF identifier that contradicts the record raises IdentityConflictError. No flag or stored confirmation overrides it.
Missing or unconfirmable evidence raises UnverifiedIdentityError. Automatic actions must stop there. A warning from optional XMP does not invalidate readable matching identifiers or corroborated Subject evidence. check_document_doi_evidence reports which corroboration check failed. Include that reason in the refusal. Do not report missing identifiers when the PDF contains a DOI that failed a different check. UnverifiedIdentityError.candidate_bibcode carries the proposed ADS record. The watcher passes it through WatchOutcome to the notice’s View ADS Entry action. Keep this candidate separate from the resolved bibcode. Do not extract identifiers from error prose or use the currently selected row for this link. Use This ID is explicit consent for a manually confirmable candidate. Pass the inspected PDF hash and candidate identifier fingerprint to the refresh command. Recheck both before applying publication metadata. Keep the notice until the requested association succeeds. Hover and keyboard focus display icon tooltips without the platform delay.
An unreadable file, a file that changes while it is read, or a hash that differs from the caller’s expected hash raises IdentityValidationError.

Manual confirmation and stored provenance

Explicit manual confirmation admits a PDF without a readable own identifier. It also admits hand-entered metadata without a stable identifier. It cannot override an own identifier that the provider does not corroborate. Its stored stamp binds the file’s SHA-256 hash and the record fingerprint. A changed file or different record invalidates that stamp.

Records without stored provenance receive another check before updates or renames. Recovered XMP identifiers must use explicit namespace declarations. An unrelated undeclared prefix can produce a warning without hiding valid identifier fields. Readable identifier conflicts still block the change.

File names, hashes, result order, and title or author similarity alone do not establish identity. Renames change a path. Metadata updates change the catalog. Neither operation rewrites PDF contents.

The check confirms stated identifiers only. It does not guarantee that every PDF can be identified or that its content is the paper.

Repairing wrong associations

scripts/repair_pdf_identities.py reattaches PDFs that a catalog links to the wrong paper. It reads a JSON manifest. Each entry names the PDF path, its SHA-256 hash, the location and attachment IDs, optionally the wrong document ID, the metadata the catalog must still hold, and the full ADS record of the correct paper. A verified DOI-provider record can supply the correct metadata when ADS has no record. Its ADS link remains empty. Without --apply or --undo, the script only prints the plan.

uv run python scripts/repair_pdf_identities.py --catalog CATALOG --root ROOT \
  --manifest manifest.json --rename-settings rename.json
uv run python scripts/repair_pdf_identities.py --catalog CATALOG --root ROOT \
  --manifest manifest.json --rename-settings rename.json --apply --journal-dir DIR
uv run python scripts/repair_pdf_identities.py --catalog CATALOG --undo DIR/repair-ID.json

Close ADSfolio before apply or undo. Stop every watcher. The repair requires exclusive catalog access. Its writer check does not hold a lock across the whole operation. A separate operating-system lock prevents concurrent repairs in the same journal directory. ADSfolio flushes journal data before atomic replacement. POSIX also flushes the parent directory. Windows has no equivalent directory flush through Python’s file API.

Pause file sync for the library as well. The manifest’s proposed_ads_record serves as the provider record for the identity check, so build it from a fresh ADS response and do not edit it by hand.

Before it changes any PDF or association, the plan checks every entry. It refuses an entry if the hash, IDs, or metadata differ from the manifest, or if the attachment has several active locations. It also refuses stale metadata and managed local references. The PDF’s own identifiers must verify the correct record. For a PDF with parser errors, a reviewed manifest can instead list the exact reviewed_parser_errors and an audit_reference. Every readable own identifier must still agree with the provider. The result receives reviewed_repair status. That status alone does not permit automatic updates or renames. Optional XMP warnings with independently confirming identifiers need no manual audit. No PDF bytes are changed to remove parser errors. Locked automatic fields block an entry unless the manifest lists them in authorized_locked_fields. Apply backs up the catalog and each PDF, writes a journal, and never overwrites a file or changes PDF bytes. A failure reverts the recorded steps. Undo refuses to run if a repaired file or catalog row changed after the repair.

Publishing releases 

Remote-changing commands print a preview unless you supply --apply. The repository scripts prepare GitHub repositories, native application releases, a Homebrew tap, and a static website. Installing the package publishes nothing.

Keep documentation and media current

Sphinx reads release and version from adsfolio.__version__. In Markdown, use {sub-ref}`release` for a current version. In reStructuredText, use |release|. The website builder replaces __ADSFOLIO_VERSION__ from the same package version. Release builds update these labels without editing page text.

Keep migration boundaries, OS requirements, and changelog versions fixed. Use VERSION for commands that require the next release number as input.

After changing visible controls, regenerate the affected screenshots and videos:

uv run --no-sync python scripts/capture_docs_media.py

The renderer uses temporary configuration and fictional papers. It does not capture the user’s desktop or library. It preserves files when their contents are unchanged. Inspect each changed screenshot and each video before publication.

Repository setup 

Install Git, uv, and the GitHub CLI.
Authenticate with gh auth login.
Preview the repository setup, then apply it:

uv run --no-sync python scripts/release.py repository
uv run --no-sync python scripts/release.py repository --apply

The default repository is ysBach/adsfolio. For a fork, use --repository OWNER/NAME.

The script initializes Git only when needed. It keeps existing history. An existing origin becomes previous-origin, then the new repository becomes origin.

If origin already points to the requested repository, rerunning setup verifies that repository without changing remotes or creating it again. Otherwise, creation stops if the remote repository already exists. For an existing repository, inspect it and configure origin manually instead of creating it again.

The script does not stage files, commit changes, or push history. A newly created GitHub repository therefore remains empty until the first push.

Before the first commit:

Review the working tree, including untracked files.
Keep credentials, catalogs, personal PDFs, build outputs, and local reports out of Git.
After review, commit the intended files.
Push main to the new remote.

After you verify the new repository, you can delete the old Git remote.

Prepare and publish a version 

Prepare the version

Set the version. Replace VERSION with the next release version:

uv run --no-sync python scripts/release.py version VERSION --apply

Update CHANGELOG.md.

Check the package:

uv run --no-sync python scripts/release.py check

Test the bundles

Run the Desktop bundles workflow manually.
Inspect all four platform results.
Test the downloads on clean target systems.

Complete code signing and dependency-license review before you claim production readiness.

Tag the release

Commit the reviewed version, lockfile, and documentation. The release script requires a clean working tree on main with the intended origin.

Preview and create the release tag:

uv run --no-sync python scripts/release.py tag
uv run --no-sync python scripts/release.py tag --apply

The script repeats local checks, creates an annotated tag, and pushes main and that tag atomically. It uses no force push. If the push fails, inspect the local tag before you retry. The script never deletes a tag automatically.

Tag workflows check exact version equality, run package checks, and build macOS ARM64, macOS Intel, Windows x86-64, and Linux x86-64 archives. They run frozen application smoke tests, verify checksums, and create a GitHub Release with Python distributions and desktop downloads. They mark development and prerelease versions as prereleases. Stable releases also include adsfolio.rb for Homebrew. The workflows do not overwrite existing release assets.

GitHub-hosted runners cannot establish every supported desktop configuration. See Building the desktop app for acceptance checks. This repository does not include signing and notarization credentials.

Optional PyPI publication 

Configure a PyPI trusted publisher for the project adsfolio, the repository, workflow desktop.yml, and environment pypi.
Create that GitHub environment with your chosen reviewer rules.
Set the repository variable PUBLISH_PYPI=true.

The workflow publishes the already-built wheel and source archive only after the GitHub Release job succeeds. It uses short-lived OIDC credentials, not a stored API token. Check project-name availability and publisher setup on PyPI. Repository scripts do not reserve a name.

Homebrew tap 

Run this after a stable GitHub Release exists. Ruby must be available for the cask syntax check. Git commits use your configured Git identity.

uv run --no-sync python scripts/publish_tap.py VERSION --create
uv run --no-sync python scripts/publish_tap.py VERSION --create --apply

This creates ysBach/homebrew-adsfolio, downloads the release checksums, generates Casks/adsfolio.rb, checks its Ruby syntax, and commits and pushes the cask. For later releases, omit --create. For other owners, use --repository and --tap.

A cask requires checksums for both native macOS archives. The script uses no placeholder checksum or :no_check. Do not publish a cask before native acceptance.

Users can then run:

brew install --cask ysBach/adsfolio/adsfolio
brew upgrade --cask adsfolio

Website and documentation 

The static landing page lives in website/. It has no build-time JavaScript framework, tracking scripts, or external fonts. The build includes strict Sphinx documentation under docs/ and works beneath a repository URL prefix.

uv sync --locked --extra gui --group docs
uv run --no-sync python scripts/build_site.py
uv run --no-sync python -m http.server 8000 --directory _site

Open http://localhost:8000. Before you deploy publicly, update the first-release notice and download status to match the artifacts actually available.

To publish the website:

In GitHub Settings → Pages, select GitHub Actions as the source.
Set PUBLISH_PAGES=true in repository variables.
Run Documentation website.

The expected project URL is https://ysbach.github.io/adsfolio/. Pull requests build the site but cannot deploy it. Until you configure Pages and the variable, website builds remain available as workflow artifacts without publishing.

Data directories and the full rename 

Update Python imports and scripts to the new name. The public package namespace, CLI, application identifiers, and new storage paths use adsfolio. The package does not install earlier command aliases. ADS_API_TOKEN still names the credential for the NASA ADS service, not the application.

On first use, the application copies earlier configuration and cache directories into the ADSfolio directories if the destination does not exist. SQLite uses the backup API, including committed WAL data. Directory migration never merges or overwrites an existing destination.

Current versions no longer convert reserved metadata keys from the earlier application. Copying an unconverted catalog retains its data, but manual field locks, author text, and local references under those keys are not honoured. Provider refreshes can overwrite fields whose earlier locks are not recognised.

Before upgrading from the earlier application:

Close the earlier application.
Open ADSfolio 0.6.0 once to convert the catalog and import its per-library files. Verify the metadata, manual locks, and local references there.
Install ADSfolio 0.10.0. Keep the original directories until you verify the converted library.

If you already used ADSfolio 0.6.0, the catalog conversion is complete. Current versions do not read per-library files under the earlier application name.

ADSfolio-named per-library sidecars and registries remain readable. The rename keeps old Git history. It does not rewrite commits or change bibliographic user text.