vfstool_lib

Breaking changes

• The library is the Jess package org.dreamweave.vfs, module @vfs, installed with jess rather than Cargo. The Rust crate vfstool_lib stays at 1.0.0 on crates.io and on this repository's rust branch, and is no longer developed on main.

Breaking changes

• The lua feature makes the crate an l3i extension, lua::VfsExtension, providing the module @dream/vfs. lua::open, lua::register and mlua are gone.
• In Luau: key lists are sequence views and trees are streams (:toTable() gives the old tables), indices are 1-based, sizes are integers, unknown option keys are errors, and paths are bytes.
• Building with lua needs l3i's toolchain: clang, lld and cross-language thin LTO.
• dream-path 1.0.
• The library writes nothing to standard output or error. collapse_into and dump_to_directory return a MaterializationReport, the files written and each winner left out as a MaterializationSkip, instead of () and a count, and DirectoryDiff gains unreadable, what diff_directory's walk could not read. In Luau, collapseInto returns the skipped files and dumpToDirectory returns them after the count.

Added

• VFS::len and VFS::is_empty, in constant time, and get_file_normalized, contains_normalized and paths_matching_bytes for keys already normalized.
• VfsKeyInput::vfs_key_bytes, and VfsKeyInput for [u8]; VfsFile::path_bytes.
• In Luau: file:readInto(buffer, offset?), tree:count(), snapshot:len(), and a type for every function, method and view element in the plan's definitions.
• File I/O in Luau: file:size(), positional file:readAt(buffer, fileOffset, length?, bufferOffset?) through a memory map, file:readRange, file:readAllBuffer, and file:open() giving a Reader with read, readInto, seek, skip, tell and close.
• The lua-write feature: a write root on the VFS (writeRoot option, setWriteRoot), writeFile, openWrite giving a Writer, mkdir, remove and rename, behind the filesystem.write runtime capability; the root is kept canonical and refuses keys that pass through a symbolic link beneath it.
• The lua-host feature: a host table of host-path functions (readFile, readAt, writeFile, open, openWrite, stat, exists, list, mkdir, remove, rename, copy, canonicalize), behind the filesystem.host runtime capability.
• lua::VfsRequireNavigator, l3i's require over a VFS with .luaurc aliases and relative paths, and lua::TemplateCache, OpenMW's script template cache: one compiled template per module per VM, cloned per instance, invalidated explicitly. The jit feature forwards l3i's and reports each template's native code status.
• Zip archives: .jpk files are read as zip archives, and Zstandard entries are listed and decoded through a pure-Rust decoder.

Changed

• Looking up a key that is already normalized borrows it instead of allocating, and searches compare bytes.
• The sorted key list is built once per change to the VFS and shared by every keys() call.
• lua::Vfs is shared and Clone, and lua::VfsFileHandle is built with VfsFileHandle::new. dream_archive 1.0.0 and l3i 1.0.0 come from crates.io; Rust 1.92.

Fixed

• On Unix, a key that is not UTF-8 keeps its exact bytes in provider records, lock manifests and semantic conflicts, instead of a lossy spelling that looks up nothing.
• collapse_into and dump_to_directory into one of the VFS's own data directories leave a loose file that already is its destination where it is. They deleted it before linking or copying it.
• extract_file into the folder a loose file already lives in leaves the file there. It deleted the file, then failed to copy it.
• run_finalize and run_finalize_tracked leave a destination that already is the changed file, rewritten in place through a hard link from output_dir, as it is. They copied it onto itself, which truncated it to nothing.
• VfsFile::file_name and file_stem end an archive entry's folders at \ as well as / on every platform: a Morrowind BSA's meshes\x\ex_door.nif is ex_door.nif on Linux and macOS too, in trees, Display and Luau's file:fileName().
• ConflictIndex::diff_report compares the keys the index holds for each source, found by exact spelling or with case and separators folded, so archives, indexes built from file lists and sources spelled in another case compare correctly; it read both paths from disk as typed. A path the index does not have is still read from disk.
• conflicts_report and shadowed_report without use_relative join a loose source's path with each file's own spelling, instead of the lowercase key, so every path names a file that exists. ConflictIndex::from_directories records those spellings.
• simulate, simulate_with_opts, simulate_impact and plan_candidate_directory find each key's current winner by its place in the provider stack. They compared the VFS's number for the source with the index's, which differ after a source that provides nothing, and counted unchanged keys as changed and left current_winner_source empty.
• The layer index of a VFS keeps a source that never provided a key, such as an empty data directory, so its positions are the source_index of the VFS's provider records and ArchiveInfo; it dropped such sources and numbered later ones lower. A source whose providers were all removed still leaves it.
• A simulated reorder never lifts an archive above a loose file: simulate, simulate_with_opts and simulate_impact ranked every source by the new order alone, so moving an archive after a data directory reported the archive winning.
• set_winner_file and set_winner_loose_file record a new key's directories, so set_winner_file("a/b.txt", …) followed by set_winner_file("a", …) refuses the second instead of keeping a file and a directory named a, which no VFS can write out.
• remove_resolved_file, remove_resolved_prefix and remove_resolved_matching_glob release the removed keys' directories: after remove_resolved_file("a/b.txt"), push_provider("a", …) returned false.
• push_provider_batch checks each entry against the ones before it, so push_directory and push_archive no longer add a file and a directory with one name; push_directory keeps the same one from_directories keeps.
• Display for VFS and display_filtered list a file whose path has no file name, such as VfsFile::from("/"), by its whole path instead of panicking, and serialized trees list it too instead of leaving it out.
• Display for VFS, display_filtered and Luau's vfs:display() nest each directory under its parent, one │ per level, and list directories that hold only directories. Every directory was printed by its last name at one level, so meshes/x and textures/x both showed as x/, and an archive's folders looked like loose ones.
• Luau's serialize writes Entries and Providers views, tree streams and VfsFile handles, each file as its file row's fields; it refused every userdata but Keys and ProviderRecords views, although the documentation said views serialize.
• collapse_into with use_symlinks links to each file's absolute path. A VFS built from relative directories made links that resolved from the output folder instead, and dangled.
• In Luau, an unknown source kind in VfsProvider.new reads VfsProvider.new.kind: unknown source kind 'zip' (...), without a second dream.vfs: inside it.
• lua::KeyBlob is exported, so a host can fill the lua::Keys(pub KeyBlob) view it could only build empty.
• Every PathBuf in a report serializes as a string, with U+FFFD in place of bytes that are not UTF-8, instead of failing with path contains invalid UTF-8 characters; diff_against_lock compares keys and sources as that text, so a lock read back from a file does not drift.

Changed

• vfstool_lib is licensed MIT OR Apache-2.0, at your option, instead of GPL-3.0-only.

Fixed

• A prefix with a trailing separator matches like one without: paths_with("meshes/"), remove_provider_prefix and remove_resolved_prefix, and their Luau forms.
• vfs:keys() returns the keys sorted, as documented, instead of in hash map order.

Fixed

• VfsFile:readAll() returns a Luau string, as documented, instead of a table of bytes.

Changed

• Rust 1.88.

Fixed

• Fallout 3's voice archives are read, through dream_archive 0.2.2.

Changed

• dream-path 0.2.

Fixed

• Oblivion's BSAs are no longer mistaken for Xbox archives, through dream_archive 0.2.

Breaking changes

• The lua feature builds mlua 0.12's vendored Luau; standalone-lua and LuaJIT are gone. Functions, methods, table fields and string enum values are camelCase.

Fixed

• Fallout 3 BSAs whose entries store only a folder and a file name are enumerated.
• Dependencies updated past the crossbeam-epoch and memmap2 security advisories.

Breaking changes

• Keys are byte-first NormalizedPaths from dream-path, taken through VfsKeyInput; host paths stay Path and PathBuf. normalize_path is gone; normalize_host_path normalizes a host path for comparison.
• The bsa feature is beth-archives, and reads BSA and BA2 through dream_archive instead of ba2.
• Rust 1.87.

Added

• VFS keeps every provider of a key, low to high priority, with the winner cached: explain, provider_records_for, duplicates, archives, archive_entries, source_contributions, and mutation that either replaces a key's stack or preserves it.
• LayerIndex for provenance, provider chains, lock manifests (VfsLock) and drift reports; ConflictIndex for conflicts, shadowed sources and source-to-source diffs.
• analyze_pair and semantic conflict reports that tell cosmetic changes from behavior changes in INI, TOML, JSON and script files.
• collapse_into, materialization_plan, dump_to_directory and extract_file, and run_setup and run_finalize for running a tool on the merged VFS.
• ZIP and PK3 archives behind the zip feature.
• With serialize, serde, serde_json, serde_yaml and toml are re-exported.
• Lua bindings through mlua behind lua; standalone-lua builds a vendored LuaJIT.
• Simulation, candidate planning, policies, a load-order solver and a knowledge base, the last three in the unstable experimental module.

Changed

• vfstool_lib is licensed GPL-3.0-only instead of GPL-3.0-or-later.

Changed

• Updated dependencies, with serde_yaml in place of serde_yaml_with_quirks.

Fixed

• Files reached through symbolic links in a data directory are part of the VFS.

Breaking changes

• Archive support is behind the bsa feature; without it, a VFS is built from loose files only.

Changed

• Updated serde_yaml_with_quirks and toml.

Added

• With zip, .jpk files are read as ZIP archives beside .zip and .pk3, and Zstandard entries decode through ruzstd, in pure Rust. Every ZIP entry is listed whatever its compression; one compressed another way fails when it is opened.
• VFS::from_directories builds the VFS from data directories in load order, later ones winning, and from the archives it is given, which loose files override.
• get_file, paths_matching and paths_with look files up by their lowercase, /-separated path; tree and tree_filtered sort them into folders.
• VfsFile: a loose file or an archive entry, opened for reading either way.
• Morrowind, Oblivion, Skyrim and Fallout 4 archives through ba2, and JSON, YAML and TOML trees behind serialize.