#29360

vpm: version ranges in v.mod, a resolver, and v why

Openmetif12opened 27d ago

Describe the feature

ROADMAP.md:83 has carried an unchecked "Package versioning" line for years. This proposal is about finishing it.

Today v install vsl@v0.1.47 splits on the last @ (parse.v:89) and hands the right-hand side to git clone --branch= (vcs.v:69-84). A "version" in vpm is a git ref: it is never parsed as a version, never compared, never resolved. Everything downstream works around the absence of a version model:

  • dependencies is []string with no constraint (vlib/v/vmod/parser.v:32)
  • asking what is installed is a string comparison of git ls-remote output, and the code says so itself: // NOTE: can be refined for branch installations (parse.v:301)
  • conflicts are resolved by whoever was parsed first — Parser.modules is a flat map keyed by name@version, and the second writer is dropped silently (parse.v:22-29)
  • v outdated compares rev-parse @ against rev-parse @{u}, i.e. branch heads, so tagged releases are invisible to it (outdated.v:50-69)
  • the on-disk layout is $VMODULES/<publisher>/<name>/, which has room for exactly one version

This proposes three things on top of that:

  1. Version ranges in v.mod, using the @ separator that already exists. 'vsl@^0.1.47', 'nedpals.args@>=0.4.0 <0.6.0', 'markdown@*'. A bare name keeps meaning "any version", so every existing v.mod stays valid.
  2. A resolver that picks the highest tagged version satisfying every constraint, backtracks when that dead-ends, and prints the requirement chain when nothing fits. The lockfile from #29250 gets priority over "highest", so an unchanged project does not drift.
  3. v why and v mod graph, plus a version-aware v outdated with four columns (Current / Upgradable / Resolvable / Latest), each answering a different reason a version might be stale.

Plus dev_dependencies, dependency_overrides, retracted, and min_v.

None of that changes a compiler behaviour or an on-disk layout — phase 1 is entirely inside cmd/tools/vpm/ plus using vlib/semver properly.

The RFC also contains a complete design for letting two major versions coexist via import-path suffixes (import nedpals.args and import nedpals.args.v2 resolving to sibling directories, Go-style). I am proposing that as a separate RFC: it is only correct if the compiler enforces the suffix, that is a language-visible change, and bundling it makes this one easier to reject whole. The design is written out in full so that RFC starts from something.

The full design, with the reasoning and the things I think are wrong with it, is in the RFC:

https://github.com/vlang/rfcs/pull/32

Use Case

Four things are impossible today:

  • Saying what you need. 'vsl@v0.1.47' means "that one". You cannot write "anything in the 0.1 series" or "at least 0.2, below 0.3". So you either pin exactly and lose every bugfix, or you leave the constraint off and take whatever master happens to be.
  • Depending on two incompatible majors at once. If a needs args@0.4 and b needs args@0.5, one of them is wrong and nothing tells you which.
  • Reproducing a build. Two runs of v install on an unchanged project can resolve differently, because every dependency is re-resolved against a moving branch tip.
  • Explaining your own tree. There is no v why. Every comparable tool has one — go mod why, npm explain, cargo tree --invert, pnpm why, bun why.

Proposed Solution

Full design in the RFC. The shape of it:

Module {
	name: 'myapp'
	version: '0.3.1'
	min_v: '0.5.0'
	dependencies: [
		'vsl@^0.1.47'
		'nedpals.args@>=0.4.0 <0.6.0'
	]
	dev_dependencies: [
		'assert@~0.2.1'
	]
	dependency_overrides: [
		'somepkg@>=2: 2.4.1'
	]
}
v install                  # resolve from v.mod, honour the lock
v install vsl              # add 'vsl@^<latest>', resolve
v install 'vsl@^1.2'       # a range — quote it, the shell splits on space
v install --locked         # fail if resolution would differ from the lock
v install --frozen         # never write the lock
v update                   # re-resolve inside existing constraints
v update --latest          # widen constraints across majors
v update -p vsl --precise 0.1.45
v why vsl
v outdated

The new v.mod keys are all readable through the parser's existing unknown map (vlib/v/vmod/parser.v:254-262), so an older vpm ignores them rather than failing to parse the file.

Relationship to #29250

This builds on #29250 and does not overlap it.

#29250 is the recording layer: after resolution, write down what was chosen so it can be reproduced. It has an implementation at whiter001/vlang#3, and I assume that work should continue there.

This is the decision layer underneath it: what should be chosen. Without a constraint language and a resolver, the lockfile faithfully records an arbitrary pick — a branch tip, chosen by whatever the remote said that morning. --locked from #29250 also only becomes meaningful once there is a resolution for it to disagree with.

Concretely, this RFC changes one thing about the lockfile: the set of versions a run may choose from. Under #29250 alone it records a branch tip; under this it records a tag picked from a range, plus that tag's SHA.

I do not propose changing the lockfile format. v why needs the requirement edges, and it can get them by walking each installed module's own v.mod — which #29250's format is already sufficient for.

Other Information

Phased, and phase 1 needs no compiler change. Phase 1 is ranges + resolver + v why + v mod graph + v outdated + --precise, all inside cmd/tools/vpm/. Phase 2 is the manifest additions (dev_dependencies, dependency_overrides, min_v). Phase 3 is retracted, deliberately last because it is the only feature whose cost scales with the number of versions you didn't pick. Phase 1 alone still replaces the ROADMAP line.

Much of phase 1 already exists. vlib/semver/range.v is a complete node-semver range engine — x-ranges, ~, ^, hyphen, || — reachable through one public function, semver.Version.satisfies() (vlib/semver/semver.v:67). vpm imports semver (vcs.v:4) and uses it only to compare the git version against 2.36.0 for shallow submodules (vcs.v:29-43). The range engine is never called. Phase 1 is mostly wiring.

There is precedent to follow and precedent to avoid. The design takes Go's /vN suffix and go mod why, Cargo's --precise and [patch], pnpm's convergence overrides, pub's dev-dependency rule and its four-column outdated, and Zig's hash-as-identity. It deliberately avoids vpkg's lockfile (a flat list with no edges and no hashes) and adding a second content hash on top of the commit SHA #29250 already records. The Prior art section goes through all of it.

The parts I think are weakest, spelled out in the RFC's Drawbacks: requiring version to be tag-checked breaks existing published modules; a backtracking resolver is a permanent commitment; retraction costs a metadata fetch per candidate version; and vlib/semver becomes load-bearing while carrying TODOs at semver.v:55 and parse.v:16 and never having been fuzzed against real-world version strings.

There are decisions in there I would like challenged. Nine open items, each with a recommendation rather than a punt — the ones I am least sure of are adopting the full node-semver grammar instead of a stricter subset (I argue a subset means writing a second range parser, which is worse), and letting overrides force a version rather than only redirect a source. If either looks wrong to a maintainer, that is the part to say so about.

Acknowledgements

  • I may be able to implement this feature request
  • This feature might incur a breaking change (making version real, and /vN at major >= 2)

Version used

V 0.5.2 (master @ 665874bfd3)

Environment details (OS name and version, etc.)

Not OS-specific in principle — the resolver, the range grammar, and /vN are all pure path and string logic. Development on Windows 11 x86_64.

Activity

5
metif1227d ago

The RFC is up: https://github.com/vlang/rfcs/pull/32

Per the rfcs process the file is numbered after this issue's PR — rfcs/32-vpm-versioning.md.

Short version of what's in it, and what I am actually asking for:

The design takes Go's /vN suffix and go mod why, Cargo's --precise and [patch], pnpm's convergence overrides, pub's dev-dependency rule and its four-column outdated, and Zig's hash-as-identity. It avoids vpkg's lockfile (a flat list with no edges and no hashes) and does not add a second content hash on top of the commit SHA #29250 already records.

Phase 1 is entirely inside cmd/tools/vpm/. The range engine already exists in vlib/semver/range.v behind semver.Version.satisfies(); vpm never calls it.

Nine decisions are listed under Unresolved questions, each with a recommendation and its alternative rather than a punt. The two I would most like a maintainer to weigh in on:

  1. Whether to adopt the full node-semver grammar (||, hyphen ranges) instead of a stricter subset. My argument is that a subset means writing a second range parser to reject syntax range.v already accepts.
  2. Whether dependency_overrides should be able to force a version, or only redirect a source.

If either looks wrong, that is the part to say so about.

metif1227d ago

Update, because the thing I said was nearly free turned out not to be.

I claimed the range engine already existed so phase 1 was "mostly wiring, not writing". I wrote a 100-case corpus against the node-semver ranges grammar and ran it: 20 divergences, several in the core expansion logic rather than at the edges.

  • a partial version is read as an exact pin, so 1.2 means =1.2.0, not 1.2.x
  • ^0.0.3 admits the whole 0.0.x series
  • 0.x has no ceiling at all
  • 2.2 - 2 does not expand, so it matches nothing
  • prereleases have no ordering: 1.0.0-alpha is neither < nor > 1.0.0, yet both <= and >= hold, because Version overloads only == and <
  • no prerelease admission check, so 1.0.0-alpha satisfies ^1.0.0 and *
  • more than two comparators is a parse failure, and an unparseable range reports "does not satisfy" — indistinguishable from a genuine miss

The RFC now has a phase 0 for repairing vlib/semver, described as a prerequisite rather than an improvement, and the "mostly wiring" sentence is replaced with the measurement. The recommendation in the RFC is unchanged — reusing one repaired parser still beats writing a second — I just cannot call it nearly free.

Two things that make the repair harder than it looks:

  • compare_gt, compare_ge and compare_le are unreachable, so the ordering fix has to go through < and those three have to go or be rewired
  • vlib/semver/semver_test.v:41 pins the ^0.0.1 divergence as expected behaviour, so a repair has to change an existing assertion — a discussion about intent, not a bugfix

Confirmed by breaking expand_caret and watching both corpus_test.v and semver_test.v go red together.

vlang/v#29394 adds the corpus test. No behaviour change and no repair — it exists so the divergences are visible and so a fix fails loudly.

metif1227d ago

Correction, found by writing a fixture rather than by reading the spec.

Every v.mod example above was missing the separator between array elements. v.mod requires a comma there, so those examples did not parse — they failed with invalid separator, which reads as a complaint about the array syntax rather than about what the element was going to say. The issue body is fixed.

While checking, I confirmed the rest of the syntax claims by parsing all of the examples:

rfc main example:        ok
overrides example:       ok
legacy map form:         ok  -> reduces to ['markdown']
retracted example:       ok
min_v example:           ok

and that dev_dependencies, dependency_overrides, retracted and min_v all land in Manifest.unknown, which is what the backward-compatibility argument depends on: an older vpm ignores them rather than failing to parse the file.

Also now open: vlang/v#29405, adding v why. It is the one part of phase 1 that depends on none of the open questions here — it reads each installed module's own v.mod and prints the routes from the project to a module, so its input is the same data a resolver would read. When the resolver lands it gains a version label rather than being replaced.

metif1224d ago

Phase 0 is now paid for rather than outstanding, and a first slice of phase 1 landed.

Phase 0. The 194 divergences grew to 302 once the shape matrix was run over 6000 ranges against node-semver 7.8.5. Those reduced to six root causes, and #29569 takes the differential result to 0 divergences across four seeds — about 16000 ranges — with false accepts down from 79 to 11-13.

The corpus lesson is the part that survives: it sat at 136 conforming cases and agreed with node-semver on all of them before and after, so it could not have found any of the six causes.

The reporting half is still open, and #29569 did not take it: version_satisfies still collapses parse_range failure to return false (vlib/semver/compare.v), so a caller still cannot tell a broken constraint from a genuine miss.

Phase 1. #29600 landed the first piece: v install vsl@v0.1.47 resolves to a concrete tag via git ls-remote --tags. It is a selection rather than a constraint — resolved once, at install time — and validate_range_destinations refuses two ranges for the same module outright, with "joint version-range resolution is not yet supported" as the message. The resolver and lock integration this issue asks for are still unwritten.

metif1224d ago

The reporting half is closed too.

Version.satisfies_or_error and is_valid_range now report an unparseable range as an error rather than as a miss, and select_version_tag rejects a broken constraint before its loop. Without that, a typo in a range made every tag get skipped and the user got "no semantic-version tag satisfies" with no idea why.

Version.satisfies itself still returns false for the same case, because changing it would break every caller at once.