Harden initial URL validation in HTTP helpers

design/catalogs.md

@@ -188,11 +188,13 @@ Current behavior differs by primitive:
   redirect handler also applies its strict scheme, host, and remote-to-local
   policy on every hop. Integrations validate cached payload shape, discard
   invalid cache entries when possible, and treat cache writes as best-effort.
-- Workflows use one-hour per-URL caches, validate every redirect hop, fall back
-  to any readable stale cache after errors while opening, redirecting, bounded
-  reading, UTF-8 decoding, or JSON parsing, and currently require only a mapping
-  at the payload root before merge. A freshly parsed non-object payload is
-  rejected after that fallback boundary.
+- Workflows use one-hour per-URL caches, validate every redirect hop, and fall
+  back to any readable stale cache after transport, bounded-read, UTF-8 decoding,
+  or JSON parsing errors. Initial URL, redirect, and final URL validation
+  rejections bypass stale-cache fallback, whether raised by a catalog validator
+  or the shared HTTP handler, and retain their cause in a workflow catalog
+  error. A freshly parsed non-object payload is rejected after the fallback
+  boundary; the payload root must be a mapping before merge.
 - Workflow steps follow the workflow cache/fallback model, validate every
   redirect hop, and disable cache use when the cache path contains a symlink.
   Duplicate JSON fields are re-raised as fatal validation errors before stale
@@ -208,6 +210,17 @@ A future shared fetch boundary may share URL validation, redirect validation,
 bounded reads, cache records, and error types. It must preserve each
 primitive's fallback policy until a separate behavior change is approved.
 
+The workflow and step URL-policy behavior above is an intentional transport
+hardening change: stale metadata must not mask a rejected URL or redirect during
+an online fetch, particularly when bundles rely on the result to validate a
+reference. Previously, a catalog callback could reject a redirect or final URL
+and still return expired metadata. Both catalog and shared HTTP validators now
+preserve these rejections as definitive catalog errors; ordinary fetch failures
+retain their existing stale-cache fallback. The shared redirect handler checks
+raw `Location` and legacy `URI` headers before urllib can normalize malformed
+absolute targets or raise an unclassified parsing error. Valid relative and
+scheme-relative redirects retain urllib's normal resolution behavior.
+
 ## Payload validation ownership
 
 The shared source layer should own source configuration, fetching, cache
@@ -249,9 +262,11 @@ usable. Validation errors are not uniformly recoverable:
   fails. Workflow-step duplicate JSON fields and duplicate IDs are fatal
   validation errors rather than skippable source-fetch errors.
 - Workflow and step fetches may use readable stale cache data after broad
-  fetch-path failures, including rejected redirects, bounded-read failures,
-  decoding failures, and malformed JSON. Fresh non-object payloads and step
-  duplicate-field errors remain outside that fallback.
+  fetch-path failures, including transport errors, bounded-read failures,
+  decoding failures, and malformed JSON. All URL-policy rejections bypass
+  stale-cache fallback, including during forced refresh, independently of
+  which validator rejects the URL first. Fresh non-object payloads and step
+  duplicate-field errors also remain outside that fallback.
 
 Bundle lookup is fail-fast in precedence order: loading a source error stops
 the operation. Built-in fetch fallback is handled inside the bundle fetcher,

docs/reference/authentication.md

@@ -329,6 +329,16 @@ You can configure multiple entries for different hosts or organizations:
 
 ## How it works
 
+The shared HTTP helpers require initial request URLs to use HTTPS with a
+hostname. They allow HTTP only for unambiguous loopback hosts, such as
+`localhost`, `127.0.0.1`, and `::1`. Catalog sources have a stricter policy:
+they require HTTPS even on loopback hosts.
+Unsafe URLs are rejected before credentials are resolved or any request is
+made, including when no authentication is configured.
+Online bundle catalog fetching and reference validation treat URL-policy
+rejections as definitive errors, not as availability fallbacks to packaged
+snapshots or unchecked references.
+
 1. For each outbound HTTP request, the URL hostname is matched against
    the `hosts` patterns in `auth.json`.
 2. If a match is found, the corresponding provider resolves the token

src/specify_cli/authentication/http.py

@@ -3,7 +3,8 @@
 No credentials are sent unless the user has created ``auth.json``.
 For each outbound URL the helper matches the hostname against
 configured entries, resolves the token via the appropriate provider
-class, and attaches auth headers.  Redirect safety is enforced:
+class, and attaches auth headers. Initial URLs must use HTTPS, except
+for HTTP to unambiguous loopback hosts. Redirect safety is enforced:
 the ``Authorization`` header is stripped when a redirect leaves the
 entry's declared hosts.  On 401/403 the next matching entry is tried,
 then unauthenticated.
@@ -16,7 +17,7 @@
 from typing import Callable
 from urllib.parse import urlparse
 
-from .._download_security import is_safe_download_redirect
+from .._download_security import is_https_or_localhost_http, is_safe_download_redirect
 from . import get_provider
 from .config import (
     AuthConfigEntry,
@@ -65,10 +66,22 @@ def _hostname_in_hosts(hostname: str, hosts: tuple[str, ...]) -> bool:
 RedirectValidator = Callable[[str, str], None]
 
 
-class RedirectPolicyError(urllib.error.URLError):
+class URLPolicyError(urllib.error.URLError):
+    """A URL rejected because it violates the client's transport policy."""
+
+
+class RedirectPolicyError(URLPolicyError):
     """A redirect rejected because it violates the client's security policy."""
 
 
+def _validate_initial_url(url: str) -> None:
+    if not is_https_or_localhost_http(url):
+        raise URLPolicyError(
+            "unsafe request URL: target must use HTTPS with a hostname, "
+            "or HTTP to loopback (for example localhost, 127.0.0.1, ::1)"
+        )
+
+
 def _validate_strict_redirect(old_url: str, new_url: str) -> None:
     if not is_safe_download_redirect(old_url, new_url):
         raise RedirectPolicyError(
@@ -81,10 +94,11 @@ def _validate_strict_redirect(old_url: str, new_url: str) -> None:
 class _StripAuthOnRedirect(urllib.request.HTTPRedirectHandler):
     """Redirect handler that guards every redirect it is installed for.
 
-    1. Run any caller-provided redirect validator.
-    2. Reject redirects that are not HTTPS with a hostname. HTTP loopback is
+    1. Reject malformed absolute targets before urllib normalizes their headers.
+    2. Run any caller-provided redirect validator.
+    3. Reject redirects that are not HTTPS with a hostname. HTTP loopback is
        allowed only when the previous hop is also loopback.
-    3. Drop ``Authorization`` when a redirect leaves trusted hosts or downgrades.
+    4. Drop ``Authorization`` when a redirect leaves trusted hosts or downgrades.
     """
 
     def __init__(
@@ -96,6 +110,31 @@ def __init__(
         self._hosts = hosts
         self._redirect_validator = redirect_validator
 
+    def http_error_302(self, req, fp, code, msg, headers):
+        if "location" in headers:
+            target = headers["location"]
+        elif "uri" in headers:
+            target = headers["uri"]
+        else:
+            return super().http_error_302(req, fp, code, msg, headers)
+
+        try:
+            parsed = urlparse(target)
+            hostname = parsed.hostname
+            parsed.port
+        except ValueError as exc:
+            raise RedirectPolicyError(f"malformed redirect URL: {exc}") from exc
+
+        # urllib can otherwise normalize a hostless absolute URL onto the
+        # current host before redirect_request sees the original target.
+        if parsed.scheme and (parsed.scheme not in ("http", "https") or not hostname):
+            raise RedirectPolicyError(
+                "unsafe redirect URL: an absolute target must use HTTP(S) with a hostname"
+            )
+        return super().http_error_302(req, fp, code, msg, headers)
+
+    http_error_301 = http_error_303 = http_error_307 = http_error_308 = http_error_302
+
     def redirect_request(self, req, fp, code, msg, headers, newurl):
         try:
             new_parsed = urlparse(newurl)
@@ -135,7 +174,9 @@ def build_request(url: str, extra_headers: dict[str, str] | None = None) -> urll
     Returns a plain request when no entry matches or the file doesn't exist.
     A ``github-cli`` acquisition failure raises immediately, without trying
     later matching entries or returning an unauthenticated request.
+    Rejects URLs that are not HTTPS or loopback HTTP before resolving tokens.
     """
+    _validate_initial_url(url)
     headers: dict[str, str] = {}
     if extra_headers:
         # Strip Authorization from extra_headers to prevent bypass
@@ -193,7 +234,10 @@ def open_url(
     Redirect scheme safety: every attempt goes through
     ``_StripAuthOnRedirect``, which rejects redirects to non-HTTPS URLs except
     HTTP between loopback URLs, and rejects remote-to-local redirects.
+    Initial URLs must also use HTTPS or loopback HTTP; unsafe URLs are
+    rejected before resolving tokens or making any request.
     """
+    _validate_initial_url(url)
     entries = find_entries_for_url(url, _load_config())
 
     def _make_req(auth_headers: dict[str, str]) -> urllib.request.Request:

src/specify_cli/bundles/adapters.py

@@ -227,7 +227,7 @@ def _http_get_json(source_id: str, url: str) -> dict:
     rather than only on the initial URL.
     """
     from ..authentication.github import GitHubCliAuthError
-    from ..authentication.http import RedirectPolicyError, open_url
+    from ..authentication.http import URLPolicyError, open_url
 
     def _validate_redirect(_old_url: str, new_url: str) -> None:
         _validate_remote_url(source_id, new_url)
@@ -257,8 +257,8 @@ def _validate_redirect(_old_url: str, new_url: str) -> None:
         raise BundlerError(
             f"Failed to fetch catalog from {url}: {exc}"
         ) from exc
-    except (GitHubCliAuthError, RedirectPolicyError) as exc:
-        # Authentication and redirect-policy failures are definitive, not
+    except (GitHubCliAuthError, URLPolicyError) as exc:
+        # Authentication and URL-policy failures are definitive, not
         # transient catalog unavailability.
         raise BundlerError(
             f"Failed to fetch catalog from {url}: {exc}"

src/specify_cli/bundles/component_catalog.py

@@ -7,7 +7,7 @@
 from urllib.error import HTTPError, URLError
 
 from ..authentication.github import GitHubCliAuthError
-from ..authentication.http import RedirectPolicyError
+from ..authentication.http import URLPolicyError
 from . import BundlerError
 from .manifest import ComponentRef
 
@@ -20,7 +20,7 @@ def _is_unavailable(exc: BaseException) -> bool:
     seen: set[int] = set()
     while exc is not None and id(exc) not in seen:
         seen.add(id(exc))
-        if isinstance(exc, (GitHubCliAuthError, RedirectPolicyError)):
+        if isinstance(exc, (GitHubCliAuthError, URLPolicyError)):
             return False
         if isinstance(exc, HTTPError):
             return exc.code in (408, 429) or 500 <= exc.code <= 599

src/specify_cli/workflows/catalog/_domain.py

@@ -559,7 +559,7 @@ def _fetch_single_catalog(
 
         # Fetch from URL — validate scheme before opening and after redirects
         from urllib.parse import urlparse
-        from specify_cli.authentication.http import open_url as _open_url
+        from specify_cli.authentication.http import URLPolicyError, open_url as _open_url
 
         def _validate_catalog_url(url: str) -> None:
             # A malformed authority (e.g. "https://[::1") makes urlparse /
@@ -572,20 +572,18 @@ def _validate_catalog_url(url: str) -> None:
                 hostname = parsed.hostname
                 _ = parsed.port
             except (TypeError, ValueError):
-                raise WorkflowCatalogError(
+                raise URLPolicyError(
                     f"Refusing to fetch catalog from malformed URL: {url}"
                 ) from None
             if parsed.scheme != "https":
-                raise WorkflowCatalogError(
+                raise URLPolicyError(
                     f"Refusing to fetch catalog from non-HTTPS URL: {url}"
                 )
             if not hostname:
-                raise WorkflowCatalogError(
+                raise URLPolicyError(
                     f"Refusing to fetch catalog from URL with no hostname: {url}"
                 )
 
-        _validate_catalog_url(entry.url)
-
         # Validate EVERY redirect hop, not just the final URL: _open_url follows
         # redirects, so an https:// entry that 30x-redirects through http:// (or
         # to a non-HTTPS host mid-chain) could otherwise let a network attacker
@@ -597,6 +595,7 @@ def _validate_redirect(_old_url: str, new_url: str) -> None:
             _validate_catalog_url(new_url)
 
         try:
+            _validate_catalog_url(entry.url)
             with _open_url(
                 entry.url, timeout=30, redirect_validator=_validate_redirect
             ) as resp:
@@ -609,6 +608,10 @@ def _validate_redirect(_old_url: str, new_url: str) -> None:
                         label="workflow catalog",
                     ).decode("utf-8")
                 )
+        except URLPolicyError as exc:
+            raise WorkflowCatalogError(
+                f"Failed to fetch catalog from {entry.url}: {exc.reason}"
+            ) from exc
         except Exception as exc:
             # Fall back to cache if available
             try:

src/specify_cli/workflows/step/catalog/_domain.py

@@ -350,7 +350,7 @@ def _fetch_single_catalog(
 
         from urllib.parse import urlparse
 
-        from specify_cli.authentication.http import open_url as _open_url
+        from specify_cli.authentication.http import URLPolicyError, open_url as _open_url
 
         def _validate_url(url: str) -> None:
             # A malformed authority (e.g. "https://[::1") makes urlparse /
@@ -363,20 +363,18 @@ def _validate_url(url: str) -> None:
                 hostname = parsed.hostname
                 _ = parsed.port
             except (TypeError, ValueError):
-                raise StepCatalogError(
+                raise URLPolicyError(
                     f"Refusing to fetch catalog from malformed URL: {url}"
                 ) from None
             if parsed.scheme != "https":
-                raise StepCatalogError(
+                raise URLPolicyError(
                     f"Refusing to fetch catalog from non-HTTPS URL: {url}"
                 )
             if not hostname:
-                raise StepCatalogError(
+                raise URLPolicyError(
                     f"Refusing to fetch catalog from URL with no hostname: {url}"
                 )
 
-        _validate_url(entry.url)
-
         # Validate EVERY redirect hop, not just the final URL: _open_url follows
         # redirects, so an https:// entry that 30x-redirects through http:// (or
         # to a non-HTTPS host mid-chain) could otherwise let a network attacker
@@ -388,6 +386,7 @@ def _validate_redirect(_old_url: str, new_url: str) -> None:
             _validate_url(new_url)
 
         try:
+            _validate_url(entry.url)
             with _open_url(
                 entry.url, timeout=30, redirect_validator=_validate_redirect
             ) as resp:
@@ -403,6 +402,10 @@ def _validate_redirect(_old_url: str, new_url: str) -> None:
                 )
         except _DuplicateCatalogField:
             raise
+        except URLPolicyError as exc:
+            raise StepCatalogError(
+                f"Failed to fetch catalog from {entry.url}: {exc.reason}"
+            ) from exc
         except Exception as exc:
             try:
                 cached = read_cache_json(