✨ feat(mq-lang): add HTTP module resolver with GitHub shorthand and CLI flags#1856
Merged
Conversation
Merging this PR will improve performance by 44.06%
Performance Changes
Tip Curious why this is faster? Comment Comparing 
Footnotes |
…and cache selectivity - Replace reqwest::blocking with ureq to avoid Tokio runtime drop panic inside async test contexts (#[tokio::test] / mq-lsp server tests) - Fix is_allowed_domain starts_with bypass: require domain boundary after prefix match (/, :, ?, # or end-of-string) so example.com.evil.com no longer passes an "example.com" allowlist entry - Split HTTP module cache into versioned/ and mutable/ subdirectories so --refresh-modules only removes mutable-ref (HEAD/branch) entries while preserving tagged versions indefinitely - Add comprehensive tests: is_versioned_url, cache_subdir routing, selective clear_cache, resolve cache-hit per subdir, to_fetch_url allowlist blocking, github_to_raw_url edge cases, multi-entry allowlist
github.com/owner/json5.mq was incorrectly stripping the .mq suffix, resolving to repo json5 instead of the actual repo json5.mq. The name component is now used verbatim as the repository name so that repos whose names end in .mq (e.g. json5.mq) are fetched correctly.
Strip URL prefix, version suffix, and .mq extension from HTTP/GitHub module paths so that e.g. `include "github.com/alice/mymod.mq@v1.0"` registers the module under the short name `mymod` instead of the full URL string. - Add `ModuleResolver::canonical_name` with a default pass-through impl - Add `HttpModuleResolver::extract_module_name` and override canonical_name - Update `ModuleLoader::load_from_file` to use canonical name for registration - Fix `eval_import` AlreadyLoaded path to look up by canonical name
Replace if-let-Ok chains in DefaultModuleResolver::resolve and get_path with match arms that fall through only on ModuleError::NotFound, and immediately return IOError, SyntaxError, and other variants.
…resolution Cache resolved module source in ModuleLoader so repeated lookups by module name skip the resolver entirely. Also fix the doc comment on Hir::new to correctly describe the module_loader parameter.
… flag in docs - Correct resolved URL for .mq-suffixed shorthand: repo name retains .mq suffix - Fix --refresh → --refresh-modules to match actual CLI flag - Update http_resolver.rs doc comment table to match code behavior
- Reject non-HTTPS URLs (http:// and other schemes) in fetch_url - Disable redirect following (max_redirects=0) to prevent allowlist bypass - Cap response body at 1 MiB to prevent memory exhaustion - Change default domain policy: empty allowlist now permits only raw.githubusercontent.com/harehare instead of all domains; --allowed-domain adds extra domains on top of this default - Add tests for fetch_url HTTPS enforcement and empty-allowlist blocking
…list permits all URLs An empty allowed_domains list only permits the built-in default domain (raw.githubusercontent.com/harehare), not all URLs. Corrects four doc comments across HttpModuleResolver::new, DefaultModuleResolver::with_http, DefaultModuleResolver::set_allowed_domains, and Engine::set_http_allowed_domains.
…llowlist, and Content-Type validation
- Add SHA-256 sidecar (`.mq.sha256`) for each cached module; re-fetch on
hash mismatch or missing sidecar to detect tampering or partial writes
- Normalize `github.com/{user}/{repo}` in `--allowed-domain` to
`raw.githubusercontent.com/{user}/{repo}` so users can allow a specific
repo without opening all of GitHub
- Reject responses whose Content-Type is `text/html` with a clear error
message instead of a confusing parse failure
- Add comprehensive tests for all three changes including normalization via
`with_http()` / `set_allowed_domains()`, Content-Type charset variants,
tampered/missing sidecar cache behaviour, and `https://github.com/` URL form
…file locking Add double-checked locking to HttpModuleResolver::resolve() using std::fs::File::lock() (stable since Rust 1.82) so that concurrent engines resolve the same URL only once. Also move the loaded_modules check before resolve() in load_from_file to skip unnecessary I/O when a module is imported more than once.
…ache Add clear_all_cache() to HttpModuleResolver that removes both mutable/ and versioned/ subdirectories including lock files. Expose it through the resolver/module/engine chain and wire it to a new --clear-cache CLI flag. Unlike --refresh-modules (mutable only), --clear-cache resets everything including versioned (tagged) modules.
a7277cc to
5b89ad2
Compare
Empty list restricts to the default domain only (raw.githubusercontent.com/harehare), not permits all domains.
0bf17fd to
62d0ed0
Compare
…disable in LSP - Add `HttpImportNotAllowed` error variant (cfg-gated to http-import feature) with a user-friendly message directing users to move the import to top level - Add `http_depth` counter to `ModuleLoader`; `resolve()` returns `HttpImportNotAllowed` for HTTP/GitHub URLs when depth > 0 - Push/pop the depth boundary in `eval_include` and `eval_import` around `load_module_with_env` so transitive HTTP imports inside modules are blocked - Remove `http-import` feature from `mq-lsp` entirely (Cargo.toml, main.rs, server.rs) so the LSP can never issue HTTP requests regardless of build flags - Add rstest parametrized tests covering blocked/unblocked cases, nesting, and underflow protection
…propagation Add eval-level integration tests verifying that HTTP/GitHub imports inside a locally-loaded module return HttpImportNotAllowed through the full evaluator pipeline.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.