Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,28 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/),
and this project adheres to [Semantic Versioning](https://semver.org/).

## [Unreleased]
### Added
- Added extraction modes to `-x`/`--extract`: `-x f` extracts every archive into its own folder named after it, and
`-x p` extracts everything next to the archives.
- `-p`/`--pack` packs the folders of disc-based games as a whole, into a single archive named after the folder, with
just the files of the folder inside: the opposite of `-x`. A folder is taken for a game when it has a disc sheet
(`.cue`, `.gdi`, `.ccd`, `.mds`, `.toc` or `.m3u`) or a file with a generic name, and only the files of one game, or
when it's an extracted disc (PS1/PS2, PS3, PSP, Xbox, Xbox 360).

### Changed
- `-x` without a mode now gives an archive its own folder when it holds more than one file, a file with a generic
name like `game.iso`, `Disc 1.cue` or `track01.bin`, or a file that would overwrite a file already in the folder or
one from another archive, instead of always extracting next to the archive. Use `-x p` for the old behavior.
`--dry-run` reads the archives to show where they would be extracted.
- With `-x` and `-x f`, archives of the same name in different formats with the same files inside get folders with
the format in the name, like `Game (USA).zip/` and `Game (USA).7z/`.
- The cleanup after processing now also removes `__MACOSX` folders, left by zips made on a Mac.
- Python 3.10+ is now required, as Python 3.9 has reached its end of life.

### Fixed
- Fixed `-r` treating the names that differ only in case (like `Pac-Man (USA)` and `PAC-MAN (Japan)`) as different
games, including the discs of one multi-disc set.

## [1.2.0] - 2026-10-16
### Added
- Added `-n`/`--dry-run` to preview what would be done without changing anything: duplicates are decided but not
Expand Down
27 changes: 26 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,18 @@ feel free to open an issue on the [GitHub issues](https://github.com/ManeFunctio
- **Extract (`-x, --extract`)**
Automatically unpacks all `.7z` or `.zip` files in the current directory (or a specified folder) into subfolders.
Supports nested subfolders, keeping all the structure.
- parameterless call is a "smart" mode, it extracts the files right next to the archive, but gives an archive
its own folder (named after the archive) when the files would make a mess otherwise: when there is more than
one file inside (like `.cue` and `.bin` tracks), or when the file has a generic name, telling nothing about the game,
which is common for disc-based systems: `game.iso`, `Disc 1.cue`, `CD2.bin`, `track01.bin`, `disk1.adf`, `Disk 1 Side A.st`,
`EBOOT.BIN`, `default.xbe`, `main.dol` and so on, or when its files would overwrite the files already
in the folder, or the files of another archive (all such archives get own folders then). An archive that
already keeps everything in one properly named folder is extracted as it is.
- `f` extracts every archive into its own folder.
- With `-x` and `-x f`, archives of the same name but different formats (like `Game (USA).zip` and
`Game (USA).7z`) that have the same files inside get folders with the format in the name: `Game (USA).zip/`
and `Game (USA).7z/`. The same happens when the own folder already has these files.
- `p` is a plain extraction: all the files go right next to the archives.

![](https://raw.githubusercontent.com/wiki/ManeFunction/romlm/extract.png)

Expand All @@ -23,6 +35,19 @@ feel free to open an issue on the [GitHub issues](https://github.com/ManeFunctio
Compresses all uncompressed files in the current directory (or a specified folder) into `.7z` or `.zip` archives.
Supports nested subfolders, keeping all the structure.

Folders of disc-based games are packed as a whole: all the files of such a folder go into a single archive named
after it (`Sonic CD (USA)/` becomes `Sonic CD (USA).7z`), without the folder itself, so it's the opposite of `-x`.
A folder is taken for a game when it holds:
- a disc sheet with its tracks: `.cue` (with `.bin`, `.iso` or `.img`), `.gdi`, `.ccd` (with `.img` and
`.sub`), `.mds` (with `.mdf`), `.toc`, or an `.m3u` playlist of a multi-disc game,
- or a file with a generic name, like `game.iso` or `track01.bin` (see `-x` above),
- and nothing but the files of one game, besides the files with generic names: either the game it is named
after (`Sonic CD (USA)/Sonic CD (USA).cue`), or any single game (`FF7/Final Fantasy VII (USA).cue`) if the
folder has no subfolders and isn't a `--sort` one (`A`-`Z`, `1-9`, `!...`). So the folders with many games
are never packed together.
- An extracted disc is a game folder whatever the names are: it has `SYSTEM.CNF` (PS1, PS2), `PS3_DISC.SFB` or
`PS3_GAME/` (PS3), `UMD_DATA.BIN` or `PSP_GAME/` (PSP), `default.xbe` (Xbox) or `default.xex` (Xbox 360).

![](https://raw.githubusercontent.com/wiki/ManeFunction/romlm/pack.png)


Expand Down Expand Up @@ -101,7 +126,7 @@ feel free to open an issue on the [GitHub issues](https://github.com/ManeFunctio


- **Other Utilities**
- Cleans out unwanted system meta-files (e.g., `desktop.ini`, `.DS_Store`).
- Cleans out unwanted system meta-files (e.g., `desktop.ini`, `.DS_Store`, `__MACOSX` folders).
- Removes empty subdirectories after sorting.
- Provides an in-script help guide (`-h, --help`).

Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ romlm = "romlm:mane"
version = { attr = "romlm.version" }

[tool.setuptools]
py-modules = ["romlm", "tags", "duplicates", "archives", "operations"]
py-modules = ["romlm", "tags", "duplicates", "archives", "operations", "discs"]

[tool.setuptools.package-dir]
"" = "src"
211 changes: 200 additions & 11 deletions src/archives.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,10 @@
import os
import re
import shutil
import subprocess
import tempfile
import zipfile
from enum import Enum
from functools import lru_cache

# Candidate binary names for a 7-Zip CLI, in preference order: "7zz" is what
Expand All @@ -11,6 +14,28 @@

ARCHIVE_EXTENSIONS = (".7z", ".zip")

# System junk that comes along with files: it is cleaned after processing, and doesn't count as archive contents
META_FILES = ("desktop.ini", "thumbs.db", ".ds_store")
META_FOLDERS = ("__macosx",)

# Names that tell nothing about the game, as disc-based dumps often have: "game.iso", "Disc 1.cue",
# "CD2.bin", "track01.bin" (Dreamcast GDI, cue/bin), "disk1.adf", "Disk 1 Side A.st" (floppies),
# "EBOOT.BIN" (PS3), "default.xbe" (Xbox), "boot.dol"/"main.dol" (GameCube, Wii).
# Such a name is made of these words only, each one can be numbered ("disc2", "Disk 1A", "Side B", "Disc 1 of 2").
# A bare number counts too, if it's short ("01.bin"), so "1942.nes" is still a game.
GENERIC_WORDS = ("game", "rom", "image", "img", "iso", "disc", "disk", "cd", "cdrom", "dvd", "gd", "gdrom", "umd",
"floppy", "tape", "side", "track", "data", "default", "boot", "eboot", "main", "of")
GENERIC_NUMBER_LENGTH = 2

# A short extension, without spaces, so "Vol. 1 (USA)" is not taken for one
_EXTENSION = re.compile(r"\.[a-z0-9]{1,5}", re.IGNORECASE)

class ExtractMode(Enum):
PLAIN = 1 # the contents go next to the archive
FOLDERS = 2 # every archive gets its own folder
AUTO = 3 # own folder only when the plain extraction would be a mess (see needs_own_folder),
# or would overwrite some files

@lru_cache(maxsize=None)
def find_seven_zip_binary():
"""Locate a usable 7-Zip CLI binary on PATH, or None if none is installed."""
Expand All @@ -20,11 +45,113 @@ def find_seven_zip_binary():
return path
return None

def exists_ignoring_case(path, listdir=os.listdir) -> bool:
"""Is there a file or folder at the path, with its names in any case, as case-insensitive file systems see it."""
return _actual_path(os.path.normpath(path), listdir) is not None

def _actual_path(path, listdir):
"""The path as it is on disk, with the names in any case, or None if there is nothing."""
if os.path.exists(path):
return path
parent, name = os.path.split(path)
if not name:
return None
actual_parent = _actual_path(parent or ".", listdir)
if actual_parent is None or not os.path.isdir(actual_parent):
return None
match = next((entry for entry in listdir(actual_parent) if entry.lower() == name.lower()), None)
return os.path.join(actual_parent, match) if match is not None else None

def is_archive(file_name) -> bool:
return file_name.lower().endswith(ARCHIVE_EXTENSIONS)

def unpack(file_name, target_folder):
"""Extracts the archive into the folder, and removes the archive."""
def strip_extensions(name) -> str:
"""The name without its extensions: "game.nkit.iso" -> "game"."""
stem, extension = os.path.splitext(name)
while stem and _EXTENSION.fullmatch(extension):
name = stem
stem, extension = os.path.splitext(name)
return name

def is_generic_name(name) -> bool:
"""Is the file (or folder) name telling nothing about the game, like "Disc 1.iso" or "track01.bin"."""
stem = strip_extensions(name).lower()
if not re.fullmatch(r"[a-z0-9\s._-]+", stem):
return False
previous = None
for token in re.findall(r"[a-z]+|\d+", stem):
is_number = token.isdigit() and (previous is not None or len(token) <= GENERIC_NUMBER_LENGTH)
is_letter = len(token) == 1 and previous is not None
if not (is_number or is_letter or token in GENERIC_WORDS):
return False
previous = token
return previous is not None

def _list_paths(file_name):
"""Paths of the files (not folders) inside the archive, as the archive keeps them."""
if file_name.lower().endswith(".zip"):
with zipfile.ZipFile(file_name, 'r') as archive:
return [info.filename for info in archive.infolist() if not info.is_dir()]
# The technical listing has a "Key = Value" block per entry, after the archive's own block ending with dashes
# -sccUTF-8 since 7-Zip prints in the OEM code page on Windows otherwise
result = subprocess.run([find_seven_zip_binary(), "l", "-slt", "-sccUTF-8", os.path.abspath(file_name)],
check=True, capture_output=True)
_, _, listing = result.stdout.decode("utf-8", errors="replace").partition("\n----------")
paths = []
for block in re.split(r"\n\s*\n", listing):
fields = dict(line.split(" = ", 1) for line in block.splitlines() if " = " in line)
is_folder = fields.get("Folder") == "+" or fields.get("Attributes", "").startswith("D")
if "Path" in fields and not is_folder:
paths.append(fields["Path"].replace("\\", "/"))
return paths

def list_files(file_name) -> list[str]:
"""Paths of the files inside the archive, with '/' separators, without the system junk."""
files = []
for path in _list_paths(file_name):
parts = [part for part in path.split("/") if part not in ("", ".")]
if parts and not any(part.lower() in META_FILES + META_FOLDERS for part in parts):
files.append("/".join(parts))
return files

def top_level_entries(files) -> set[str]:
"""Names of the files and folders that land right in the target folder."""
return {path.split("/")[0] for path in files}

def needs_own_folder(top_level) -> bool:
"""An archive with a few things inside, or one with a generic name, would clutter the folder it's extracted to."""
return len(top_level) > 1 or any(is_generic_name(name) for name in top_level)

def own_folder_name(file_name, top_level) -> str:
"""The archive name without its extension, and without the inner file's extension too, if romlm packed it."""
name = os.path.splitext(os.path.basename(file_name))[0]
if name.lower() in {entry.lower() for entry in top_level}:
name = os.path.splitext(name)[0] or name
return name

def extraction_folders(file_name, target_folder, mode, files) -> list[str]:
"""
Where the archive's `files` can go, from the most to the least preferred, according to the mode: the target
folder itself (if AUTO mode finds it fine), an own folder ("Game (USA)"), and an own folder with the archive format
("Game (USA).zip"), for the case another archive of the same name has the same files.
"""
if mode == ExtractMode.PLAIN:
return [target_folder]
top_level = top_level_entries(files)
own_folder = own_folder_name(file_name, top_level)
folders = [os.path.join(target_folder, own_folder),
os.path.join(target_folder, own_folder + os.path.splitext(file_name)[1])]
if mode == ExtractMode.AUTO and not needs_own_folder(top_level):
folders.insert(0, target_folder)
return folders

def extraction_folder(file_name, target_folder, mode) -> str:
"""Where the archive's contents should go, if nothing is in the way. Reads the archive, unless in PLAIN mode."""
if mode == ExtractMode.PLAIN:
return target_folder
return extraction_folders(file_name, target_folder, mode, list_files(file_name))[0]

def _extract(file_name, target_folder):
if file_name.lower().endswith(".7z"):
subprocess.run(
[find_seven_zip_binary(), "x", os.path.abspath(file_name), f"-o{os.path.abspath(target_folder)}", "-y"],
Expand All @@ -33,30 +160,92 @@ def unpack(file_name, target_folder):
elif file_name.lower().endswith(".zip"):
with zipfile.ZipFile(file_name, 'r') as archive:
archive.extractall(target_folder)

def unpack(file_name, target_folder):
"""Extracts the archive into the folder, and removes the archive. The folder can be named as the archive itself."""
# A folder can't take the archive's place before the archive is gone, so it is extracted next to it first
is_in_place = os.path.isfile(target_folder) and os.path.samefile(target_folder, file_name)
if is_in_place:
folder = tempfile.mkdtemp(prefix=".romlm-", dir=os.path.dirname(os.path.abspath(file_name)))
else:
folder = target_folder
os.makedirs(folder or ".", exist_ok=True)
try:
_extract(file_name, folder)
except Exception:
if is_in_place:
shutil.rmtree(folder, ignore_errors=True)
raise
os.remove(file_name)
if is_in_place:
os.rename(folder, target_folder)

def _clean_and_list(folder) -> list[str]:
"""Removes the system junk from the folder, and lists the files left, relative to it."""
files = []
for dirpath, dirnames, filenames in os.walk(folder):
for d in [d for d in dirnames if d.lower() in META_FOLDERS]:
shutil.rmtree(os.path.join(dirpath, d))
dirnames.remove(d)
for f in filenames:
if f.lower() in META_FILES:
os.remove(os.path.join(dirpath, f))
else:
files.append(os.path.relpath(os.path.join(dirpath, f), folder))
return sorted(files)

def _remove_packed(folder, files):
"""Removes the packed files, and the folders left empty after them, up to the packed folder itself."""
for name in files:
os.remove(os.path.join(folder, name))
for dirpath, _, _ in os.walk(folder, topdown=False):
if not os.listdir(dirpath):
os.rmdir(dirpath)

def pack(file_name, target_folder, packing_format) -> str:
"""Packs the file into an archive in the folder, and removes the file. Returns the archive path."""
"""
Packs the file into an archive in the folder, and removes the file. Returns the archive path.
A folder is packed the same way, as a single archive with the files of the folder, but not the folder itself.
"""
archive_file = f"{os.path.join(target_folder, os.path.basename(file_name))}.{packing_format}"
if os.path.isdir(file_name):
# Adding to an existing archive would mix two games in one
if exists_ignoring_case(archive_file):
raise FileExistsError(f"'{archive_file}' already exists")
source_folder = os.path.abspath(file_name)
files = _clean_and_list(file_name)
else:
source_folder = os.path.dirname(os.path.abspath(file_name))
files = [os.path.basename(file_name)]
list_file = None
try:
if packing_format == "7z":
# Run with cwd set to the source file's folder, passing only its basename,
# so the archive stores just that name (matching the old arcname behavior)
# instead of the full source path.
# Run with cwd set to the source folder, passing the names relative to it, so the archive stores
# just them, instead of the full source paths. A list file keeps a big folder within the
# command line length limit.
with tempfile.NamedTemporaryFile("w", encoding="utf-8", suffix=".txt", delete=False) as listing:
listing.write("\n".join(files))
list_file = listing.name
subprocess.run(
[find_seven_zip_binary(), "a", os.path.abspath(archive_file), os.path.basename(file_name)],
cwd=os.path.dirname(os.path.abspath(file_name)),
check=True, capture_output=True,
[find_seven_zip_binary(), "a", os.path.abspath(archive_file), "-scsUTF-8", f"@{list_file}"],
cwd=source_folder, check=True, capture_output=True,
)
elif packing_format == "zip":
with zipfile.ZipFile(archive_file, 'w', zipfile.ZIP_DEFLATED) as archive:
archive.write(file_name, arcname=os.path.basename(file_name))
for name in files:
archive.write(os.path.join(source_folder, name), arcname=name)
except Exception:
# Don't leave a half-written archive next to the source file
if os.path.exists(archive_file):
os.remove(archive_file)
raise
os.remove(file_name)
finally:
if list_file is not None:
os.remove(list_file)
if os.path.isdir(file_name):
_remove_packed(file_name, files)
else:
os.remove(file_name)
return archive_file

def describe_error(error) -> str:
Expand Down
Loading
Loading