mirror of
https://github.com/Dvorinka/swingmusic-extended.git
synced 2026-06-04 20:43:04 +00:00
merge refactors pr #364 from @michilyy
* Save to DB only unique trackhashes * Add check if track already exists in playlist * replace all paths with `pathlib.Path` * `architecture.md`: * add config folder layout `config.py`: * fix bug where `pathlib.Path` cannot be serialized `files.py`: * remove unused imports * update path concatenation to `pathlib.Path` * add config-folder creation `imgserver.py`: * fix serialisation bug `playlistlib.py`: * update path concatenation to `pathlib.Path` * update all `settings.Paths` usages to new singleton `Paths` class. * update all usages of `settings.Paths` * `files.py`: * rework assets copy function. * remove unused loop and unused `shutil.copy2` function `settings.py` * fix recursion exception in `Paths` * `settings.py`: * remove Singleton and `@property` todos from `Paths` * `__init__.py`: * remove now unused function `create_config_dir()` `setup.files`: * remove because merged into `settings.Paths()` for more central and clear flow how the base path gets decided `settings.py`: * add `copy_assets` function `start_swingmusic.py`: * add configurable settings.Paths class `__main__.py`: * update click to used correct default path * remove wrong commited egg files * remove change in the wrong branch * add forgotten `property` decorator update `get_files_and_dirs` to use pathlib where possible `config.py`: * update type annotation `folders.py`: * convert `pathlib` to posix path where needed for sub-functions `folderlib.py`: * rework `get_files_and_dirs` to use `pathlib` where possible `settings.py`: * add forgotten `@property` `start_swingmusic.py`: * remove second `log_startup_info()` * `artistlib.py`: * fix calling property `tagger.py`: * fix comparing elements in `pathlib.Path` * add support for repeating lyrics. * rework lyrics api and lib * update most path functions. add type-hint pathlib where needed * for serialization paths are converted to posix path * use `open` instead of `os.open` update `metaclass` with constant * fix initial config exception if empty file existed * update `userConfig` with `InitVar` to be excluded from `asdict` * remove `is_windows_slash()` rework path function to use pathlib * convert `pathlib.Path` to `str` for serialization * fixing bug with str + pathlib * `__main__.py`: * update click to use package version * remove now unused function `print_version` `filesystem.py`: * rework `CWD` to use importlib `pyproject.toml`: * disable namespace for `importlib.resources` to work correctly * update `lyrics.py`: * remove unused functions * simplify functions * fix bug where assets get created on root * remove unused code * update lyrics for clearer structure. * add support for unsynced lyrics * fix wrong return type in unsynced lyrics * update `/check` to use `send_lyrics` * prefer tags to duplicates * `lyrics.py`: * add docs to a function group * `logger.py`: * add logging config dict. * combine Logging into one file * add socket logger * add debug mode to logger * add JSONL formater * `logger.py`: * update config to directly use the formater. resolves circular import exception `__main__.py`: * add logger setup to main `start_swingmusic.py`: * add debug option to cli * `lyrics.py`: * add offset support * add `setuptools-scm` to get version from git * add support for docker build with scm * add support for docker build with scm need someone who can test the changes workflow * update all usage of `version.txt` to `metadata.version()` * 2x update all usage of `version.txt` to `metadata.version()` * update to no local_scheme version * provide fix for #331. convert `sql.Row` and `TrackTable` to dict before converting to dataclass. * fix `__main__.py`: * wrong import and uncommited changes * add debug and base_path parameter * fix logger pathlib * add client build workflow * set name * split client from build * try fixing builds * try another fix * try also another fix * try again something new * try again something new * change runner * fix failed run because of malformed runner * add wheel builds * remove systems from pure python build * add isolated pyinstaller build * artifacts with names * wrong wheel path * try fetch-depth for tag fetch * disable fail-fast. add wheel installation * add install system packages * add debug * fix wheel install fix pyinstaller spec file * try fix for pyinstaller * try another fix * build on release * add concrete release types * only run on released or pre-released * try release upload * reformat upload * fix needs tag * identifiable pyinstaller builds * compress client folder before uploading * update to src build * remove no more needed aarch64 build script rename pyinstaller assets to lowercase * remove unneeded code * fix: save to DB only unique track hashes * replace click with argparse * set concrete types in argparse * replace manuall path usages with pathlib * remove unused `configs.py` file * reformat `start_swingmusic.py` * fix empty set startup exception * optimizing static files serve function * fixing bug in optimisation of static files serve function * fix folder view bug * colorlib.py: * fix wrong type exception * remove singe use Index_everything class * update logging of populate.py * cleanup files * fix settings.py Paths copy function. Created folder on file. * add exist check to folder * remove unused `INFO` class * fix multiprocessing bug on windows * potential icon fix for pyinstaller fix multiple logging bug * fix argparse config path bug add jobs file * cleanup code fragments fix logging issue add notes to function * note that concurrent creates own sys.modules * refactor some lyrics plugin condition remove unused import from hashing * refactor taglib.py * update import statements to be static * playlistlib.py: * refactoring and more doc strings populate.py: * add poc bugfix settings.py: * add typehint * possible bugfix for multitreading globals * folder.py: * add check if provided path is absolute populate.py: * add bug note settings.py: * add possible error from Singleton implementation start_swingmusic.py: * correct spelling * pass resolved path to Paths tagger.py: * add logging * trying out fixes for multithreading * only upload results not metadata * fix build action again * folder.py: * strictly use pathlib where possible folderlib.py: * add missing docstring to function, who really need it. track.py: * refactor some code folder.py: * refactor some more code * Merge DBPath class and Paths class. Update all usages of DBPath folderslib.py: * fix bug with logging taglib.py: * add missing docstring settings.py: * merge classes * refactor * network.py: * add more docstring config.py: * update pathlib usage tools.py: * refactor * add docstrings * colorlib.py: * add docstring Refactor App builder into grouped config settings. * update assets access for migration * Update FUNDING.yml * Update FUNDING.yml * upgrade tinytag in requirements.txt * update readme * update license * update readme * Update README.md * Update README.md * cleanup requirements.txt remove unused import in audio_segment.py add entrypoint.sh for appimage support update pyproject.toml for optional dependencies add appimage to github workflow * fix invalid workflow file * AppImage build needs more research. Commenting for now * testing a new build workflow * add libev installation * update workflow to new optional dependencies * trying again another fix * finally fix all optional deps installation correctly * remove AppImage poc * albumslib.py: * add docstring folder.py: * add unix path fix update logger name to `__name__` * update build with docker update Dockerfile with git fix typo in lyrics.py add dynamic deps back * add log for static folder * add missing import * add some more todos * add support for AppImages even when it's not perfect. * quick bugfix for wrong appimage config path * fix uploading not finding AppImages builds aka wrong pattern * optimise docker build by using artifacts. Add client path option. change docstring to sphinx format * add todos * Now support AppImages for real: manually build AppImage as we are building a complex project. * fix missing dep in AppImage build * add full AppImage metadata * add missing image file. * only update swingmusic appimage not tool * add todo and fix AppImage build again. * Try fixing some path mixup in AppImage build * add debug tag to action * correct path to appimage folder * do not download tool before checkout * Another fix for path in appimage build * extend config files with more information * default client dir is now inside the config dir. TODOs updated. * default client dir is now inside the config dir. TODOs updated. Add priority todos. * Auto download client when client not found. Respects user provided dir. * rename `requests` submodule to `request` * poc for arm AppImage builds * try out another fix * fix typo in build.yml * add missing arch tag * fix uploading double names * unique naming * enable fallback version for project. * do not download client into readonly dir. * fix relative client download path. Client was resolved into parent of config. * remove client backup path as client is now downloadable * `Paths` checks if config folder exists and creates it if necessary. logger no more creates the config folder. `app_builder.py`: static route no more with '/client' * path are only created in MainProcess. fix gz file not found. * move assets into src and update usages accordingly * remove solved todos * Only upload artefacts if not draft/master aka only on tag * wrong type in assets copy * update log with correct priority * add debug statements and logging to Paths * remove debugging statement * remove double version tag from docker build * fork save release protection * fix typo * add fallback client dir for static builds. * update argparse to new param * add missing import pathlib * add sparse checkout as we do not need everything downloaded * add assets copy check * init logger bevor Paths * remove unused import * check if logdir exists and create if not * only add exec info to file * remove exception log from cli * move logging into main. Allows tools support again. * UserConfig now correctly uses _finished key. Bug where _finished was never written * double save serverId. update root_dir to trow no exception on init. remove debug param * clean up TODOs --------- Co-authored-by: skilletfun <skilletfun.laptew.sergey@yandex.ru> Co-authored-by: Mungai Njoroge <geoffreymungai45@gmail.com>
This commit is contained in:
michily
@@ -0,0 +1,330 @@
|
||||
"""
|
||||
Contains all the track routes.
|
||||
"""
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
import time
|
||||
from typing import Literal
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
from flask_openapi3 import APIBlueprint, Tag
|
||||
from swingmusic.api.apischemas import TrackHashSchema
|
||||
from swingmusic.lib.transcoder import start_transcoding
|
||||
from flask import request, Response, send_from_directory
|
||||
from swingmusic.lib.trackslib import get_silence_paddings
|
||||
|
||||
from swingmusic.store.tracks import TrackStore
|
||||
from swingmusic.utils.files import guess_mime_type
|
||||
|
||||
bp_tag = Tag(name="File", description="Audio files")
|
||||
api = APIBlueprint("track", __name__, url_prefix="/file", abp_tags=[bp_tag])
|
||||
|
||||
|
||||
class TransCodeStore:
|
||||
map: dict[str, str] = {}
|
||||
|
||||
@classmethod
|
||||
def add_file(cls, trackhash: str, filepath: str):
|
||||
cls.map[trackhash] = filepath
|
||||
|
||||
@classmethod
|
||||
def remove_file(cls, trackhash: str):
|
||||
del cls.map[trackhash]
|
||||
|
||||
@classmethod
|
||||
def find(cls, trackhash: str):
|
||||
return cls.map.get(trackhash)
|
||||
|
||||
|
||||
class SendTrackFileQuery(BaseModel):
|
||||
filepath: str = Field(description="The filepath to play (if available)")
|
||||
quality: str = Field(
|
||||
"original",
|
||||
description="The quality of the audio file. Options: original, 1411, 1024, 512, 320, 256, 128, 96",
|
||||
)
|
||||
container: Literal["mp3", "aac", "flac", "webm", "ogg"] = Field(
|
||||
"mp3",
|
||||
description="The container format of the audio file. Options: mp3, aac, flac, webm, ogg",
|
||||
)
|
||||
|
||||
|
||||
@api.get("/<trackhash>/legacy")
|
||||
def send_track_file_legacy(path: TrackHashSchema, query: SendTrackFileQuery):
|
||||
"""
|
||||
Get a playable audio file without Range support
|
||||
|
||||
Returns a playable audio file that corresponds to the given filepath. Falls back to track hash if filepath is not found.
|
||||
|
||||
NOTE: Does not support range requests or transcoding.
|
||||
"""
|
||||
trackhash = path.trackhash
|
||||
filepath = query.filepath
|
||||
msg = {"msg": "File Not Found"}
|
||||
|
||||
track = None
|
||||
tracks = TrackStore.get_tracks_by_filepaths([filepath])
|
||||
|
||||
if len(tracks) > 0 and os.path.exists(filepath):
|
||||
track = tracks[0]
|
||||
else:
|
||||
res = TrackStore.trackhashmap.get(trackhash)
|
||||
|
||||
# When finding by trackhash, sort by bitrate
|
||||
# and get the first track that exists
|
||||
if res is not None:
|
||||
tracks = sorted(res.tracks, key=lambda x: x.bitrate, reverse=True)
|
||||
|
||||
for t in tracks:
|
||||
if os.path.exists(t.filepath):
|
||||
track = t
|
||||
break
|
||||
|
||||
if track is not None:
|
||||
audio_type = guess_mime_type(filepath)
|
||||
return send_from_directory(
|
||||
Path(filepath).parent,
|
||||
Path(filepath).name,
|
||||
mimetype=audio_type,
|
||||
conditional=True,
|
||||
as_attachment=True,
|
||||
)
|
||||
|
||||
return msg, 404
|
||||
|
||||
|
||||
@api.get("/<trackhash>")
|
||||
def send_track_file(path: TrackHashSchema, query: SendTrackFileQuery):
|
||||
"""
|
||||
Get a playable audio file with Range headers support
|
||||
|
||||
Returns a playable audio file that corresponds to the given filepath. Falls back to track hash if filepath is not found.
|
||||
|
||||
Transcoding can be done by sending the quality and container query parameters.
|
||||
|
||||
**NOTES:**
|
||||
- Transcoded streams report incorrect duration during playback (idk why! FFMPEG gurus we need your help here).
|
||||
- The quality parameter is the desired bitrate in kbps.
|
||||
- The mp3 container is the best container for upto 320kbps (and has better duration reporting). The flac container allows for higher bitrates but it produces dramatically larger files (when transcoding from lossy formats).
|
||||
- You can get the transcoded bitrate by checking the X-Transcoded-Bitrate header on the first request's response.
|
||||
"""
|
||||
trackhash = path.trackhash
|
||||
filepath = query.filepath
|
||||
|
||||
# If filepath is provided, try to send that
|
||||
track = None
|
||||
tracks = TrackStore.get_tracks_by_filepaths([filepath])
|
||||
|
||||
if len(tracks) > 0 and os.path.exists(filepath):
|
||||
track = tracks[0]
|
||||
else:
|
||||
res = TrackStore.trackhashmap.get(trackhash)
|
||||
|
||||
# When finding by trackhash, sort by bitrate
|
||||
# and get the first track that exists
|
||||
if res is not None:
|
||||
tracks = sorted(res.tracks, key=lambda x: x.bitrate, reverse=True)
|
||||
|
||||
for t in tracks:
|
||||
if os.path.exists(t.filepath):
|
||||
track = t
|
||||
break
|
||||
|
||||
if track is not None:
|
||||
if query.quality == "original":
|
||||
return send_file_as_chunks(track.filepath)
|
||||
|
||||
# prevent requesting over transcoding
|
||||
max_bitrate = track.bitrate
|
||||
requested_bitrate = int(query.quality)
|
||||
|
||||
if query.container != "flac":
|
||||
# drop to 320 for non-flac containers
|
||||
requested_bitrate = min(320, requested_bitrate)
|
||||
|
||||
quality = f"{min(max_bitrate, requested_bitrate)}k"
|
||||
return transcode_and_stream(trackhash, track.filepath, quality, query.container)
|
||||
|
||||
return {"msg": "File Not Found"}, 404
|
||||
|
||||
|
||||
def transcode_and_stream(trackhash: str, filepath: str, bitrate: str, container: str):
|
||||
"""
|
||||
Initiates transcoding and returns the first chunk of the transcoded file.
|
||||
|
||||
The other chunks are streamed on subsequent requests and are rerouted to `send_file_as_chunks`.
|
||||
"""
|
||||
temp_file = TransCodeStore.find(trackhash)
|
||||
if temp_file is not None:
|
||||
return send_file_as_chunks(temp_file)
|
||||
|
||||
format_params = {
|
||||
"mp3": ["-c:a", "libmp3lame"],
|
||||
"aac": ["-c:a", "aac"],
|
||||
"webm": ["-c:a", "libopus"],
|
||||
"ogg": ["-c:a", "libvorbis"],
|
||||
"flac": ["-c:a", "flac"],
|
||||
"wav": ["-c:a", "pcm_s16le"],
|
||||
}
|
||||
|
||||
# Create a temporary file
|
||||
format = f".{container}" if container in format_params.keys() else ".flac"
|
||||
container_args = (
|
||||
format_params[container]
|
||||
if container in format_params.keys()
|
||||
else format_params["flac"]
|
||||
)
|
||||
temp_file = tempfile.NamedTemporaryFile(delete=False, suffix=format)
|
||||
temp_filename = temp_file.name
|
||||
temp_file.close()
|
||||
|
||||
TransCodeStore.add_file(trackhash, temp_filename)
|
||||
start_transcoding(filepath, temp_filename, bitrate, container_args)
|
||||
|
||||
chunk_size = 1024 * 512 # 0.5MB
|
||||
file_size = os.path.getsize(filepath)
|
||||
|
||||
def generate():
|
||||
# Poll for the output file
|
||||
while (
|
||||
not os.path.exists(temp_filename)
|
||||
or os.path.getsize(temp_filename) < chunk_size
|
||||
):
|
||||
print(f"Waiting for transcoding to complete... filename: {temp_filename}")
|
||||
time.sleep(0.1) # Wait for 100ms before checking again
|
||||
|
||||
with open(temp_filename, "rb") as file:
|
||||
file.seek(0)
|
||||
return file.read(chunk_size)
|
||||
|
||||
audio_type = guess_mime_type(temp_filename)
|
||||
response = Response(
|
||||
generate(),
|
||||
206,
|
||||
mimetype=audio_type,
|
||||
content_type=audio_type,
|
||||
direct_passthrough=True,
|
||||
)
|
||||
response.headers.add("Content-Range", f"bytes {0}-{chunk_size}/{file_size}")
|
||||
response.headers.add("Accept-Ranges", "bytes")
|
||||
response.headers.add("X-Transcoded-Bitrate", bitrate)
|
||||
return response
|
||||
|
||||
|
||||
def send_file_as_chunks(filepath: str) -> Response:
|
||||
"""
|
||||
Returns a Response object that streams the file in chunks.
|
||||
"""
|
||||
# NOTE: +1 makes sure the last byte is included in the range.
|
||||
# NOTE: -1 is used to convert the end index to a 0-based index.
|
||||
chunk_size = 1024 * 512 # 0.5MB
|
||||
|
||||
# Get file size
|
||||
file_size = os.path.getsize(filepath)
|
||||
start = 0
|
||||
end = chunk_size
|
||||
|
||||
# Read range header
|
||||
range_header = request.headers.get("Range")
|
||||
if range_header:
|
||||
start = get_start_range(range_header)
|
||||
|
||||
# If start + chunk_size is greater than file_size,
|
||||
# set end to file_size - 1
|
||||
_end = start + chunk_size - 1
|
||||
|
||||
if _end > file_size:
|
||||
end = file_size - 1
|
||||
else:
|
||||
end = _end
|
||||
|
||||
def generate_chunks():
|
||||
with open(filepath, "rb") as file:
|
||||
file.seek(start)
|
||||
remaining_bytes = end - start + 1
|
||||
|
||||
retry_count = 0
|
||||
max_retries = 10 # 5 * 100ms = 500ms total wait time
|
||||
|
||||
while remaining_bytes > 0 or retry_count < max_retries:
|
||||
if retry_count == max_retries:
|
||||
print("💚 sending final chunk! ...")
|
||||
|
||||
pos = file.tell()
|
||||
chunk = file.read(os.path.getsize(filepath) - pos)
|
||||
|
||||
return chunk, pos, True
|
||||
|
||||
if remaining_bytes < chunk_size:
|
||||
time.sleep(0.25)
|
||||
retry_count += 1
|
||||
remaining_bytes = os.path.getsize(filepath) - file.tell()
|
||||
continue
|
||||
|
||||
chunk = file.read(min(chunk_size, remaining_bytes))
|
||||
if chunk:
|
||||
remaining_bytes -= len(chunk)
|
||||
return chunk, file.tell(), False
|
||||
else:
|
||||
# If no data is read, wait for 100ms before retrying
|
||||
time.sleep(0.25)
|
||||
retry_count += 1
|
||||
|
||||
# update remaining bytes
|
||||
remaining_bytes = os.path.getsize(filepath) - file.tell()
|
||||
print(f"▶ Remaining bytes: {remaining_bytes}")
|
||||
|
||||
return None, 0, True
|
||||
|
||||
data, position, is_final = generate_chunks()
|
||||
|
||||
audio_type = guess_mime_type(filepath)
|
||||
response = Response(
|
||||
response=data,
|
||||
status=206, # Partial Content status code
|
||||
mimetype=audio_type,
|
||||
content_type=audio_type,
|
||||
direct_passthrough=True,
|
||||
)
|
||||
|
||||
bytes_to_add = chunk_size if not is_final else 0
|
||||
response.headers.add(
|
||||
"Content-Range",
|
||||
f"bytes {start}-{position}/{os.path.getsize(filepath) + bytes_to_add}",
|
||||
)
|
||||
response.headers.add("Access-Control-Expose-Headers", "Content-Range")
|
||||
response.headers.add("Accept-Ranges", "bytes")
|
||||
return response
|
||||
|
||||
|
||||
def get_start_range(range_header: str):
|
||||
try:
|
||||
range_start, range_end = range_header.strip().split("=")[1].split("-")
|
||||
return int(range_start)
|
||||
|
||||
except ValueError:
|
||||
return 0
|
||||
|
||||
|
||||
class GetAudioSilenceBody(BaseModel):
|
||||
ending_file: str = Field(description="The ending file's path")
|
||||
starting_file: str = Field(description="The beginning file's path")
|
||||
|
||||
|
||||
@api.post("/silence")
|
||||
def get_audio_silence(body: GetAudioSilenceBody):
|
||||
"""
|
||||
Get silence paddings
|
||||
|
||||
Returns the duration of silence at the end of the current ending track and the duration of silence at the beginning of the next track.
|
||||
|
||||
NOTE: Durations are in milliseconds.
|
||||
"""
|
||||
ending_file = body.ending_file # ending file's filepath
|
||||
starting_file = body.starting_file # starting file's filepath
|
||||
|
||||
if ending_file is None or starting_file is None:
|
||||
return {"msg": "No filepath provided"}, 400
|
||||
|
||||
return get_silence_paddings(ending_file, starting_file)
|
||||
Reference in New Issue
Block a user