mirror of
https://github.com/Dvorinka/swingmusic-extended.git
synced 2026-06-03 20:13:02 +00:00
fix: recently added items sort order in the homepage
.ie. stop relying on folder last mod date, and use the latest file from the folder + bump watchdog to v4 + add WIP docs (stashed in .github/code.docs for now)
This commit is contained in:
+8
-14
@@ -1,19 +1,13 @@
|
|||||||
# What's New?
|
# What's New?
|
||||||
|
|
||||||
- Silence removal: Remove silence less than -40 dB between tracks
|
- New no sidebar layout
|
||||||
- Crossfade (labeled experimental on Firefox, because some tracks stutter when crossfade ends)
|
|
||||||
- Automatic preloading of next track, meaning reduced delay between tracks. Impact most noticable on reverse proxy.
|
|
||||||
- Quick settings on settings page
|
|
||||||
- CTRL + B to toggle right sidebar
|
|
||||||
- Remove from queue option in track context menu in Now playing page
|
|
||||||
- Add Playlists and favorites to Browse section of homepage
|
|
||||||
- Add right click menu option to search track on Youtube, Spotify, etc.
|
|
||||||
|
|
||||||
# Bug fixes
|
# Bug fixes
|
||||||
|
|
||||||
- Tracks not being removed from queue when you click the ✕ button
|
- Fix recently added items not filling row
|
||||||
- Consistent design on playlist list page
|
-
|
||||||
- Playlist cards are now larger on playlist list page
|
|
||||||
- Save queue as playlist now works as expected
|
# Development
|
||||||
- Keyboard shortcuts not working on first attempt
|
|
||||||
- Misc. code improvements
|
- Move to [waitress](https://docs.pylonsproject.org/projects/waitress/en/stable/) WSGI server
|
||||||
|
- Add documentation to `.github/docs`. Contributions are welcome!
|
||||||
|
|||||||
@@ -0,0 +1,8 @@
|
|||||||
|
# Databases
|
||||||
|
|
||||||
|
The databases store data for faster persistence and faster retrieval. There are 2 databases:
|
||||||
|
|
||||||
|
- `swing.db` - stores metadata
|
||||||
|
- `userdata.db` - stores user data. Think favorites, playlists and settings.
|
||||||
|
|
||||||
|
## More info on this should be added soon
|
||||||
@@ -0,0 +1,77 @@
|
|||||||
|
# Hashing
|
||||||
|
|
||||||
|
I didn't know what else to name this page, so I just named it hashing.
|
||||||
|
|
||||||
|
## What is hashing in this context?
|
||||||
|
|
||||||
|
From sources on the web, "hashing is the process of converting data — text, numbers, files, or anything, really — into a fixed-length string of letters and numbers". In our little case here, we would modify that to say something like: "hashing is the process of creating a short identifier (hash) for tracks, artists and albums".
|
||||||
|
|
||||||
|
We need a short hash because we want to perform searches faster while keeping URLs short. For the unique part, a special condition is added into the mix: the ability to return the same hash for soft duplicates (case-insensitive and non-alpha numeric character ignoring). For example, given the hash for artist name "Bob" is `foobar`, the other variation of it (eg. "bob", "BoB", "bob?") should give the same hash `foobar`.
|
||||||
|
|
||||||
|
This hashing method makes one costly assumption: Any non-alphanumeric characters are for decoration and can be done without. This of course is messed up, but it works for 99% of the tracks, albums and artists. The remaining 1% is the cost of doing business (standardization).
|
||||||
|
|
||||||
|
## Why do we need this?
|
||||||
|
|
||||||
|
These are a few use cases that I can remember without digging through the code.
|
||||||
|
|
||||||
|
### 1. Duplicate detection
|
||||||
|
|
||||||
|
First of all, when we refer to songs as duplicates, it means that regardless of the file quality, filepath or anything else, those songs share the same track title, album and artist despite the text case. This is how we treat tracks in the real world. Right?
|
||||||
|
|
||||||
|
When showing an album with duplicates, we filter out duplicates and show trackswith the highest bitrate, keeping your album page clean.
|
||||||
|
|
||||||
|
### 2. Downloading images from the internet
|
||||||
|
|
||||||
|
Swing Music uses Deezer as the artist image provider. When searching for artist images, the first result is not always the correct artist. Therefore, we use hashing to get the correct artist image from the list of results.
|
||||||
|
|
||||||
|
For this specific task, we also consider decoding artist names for special characters. (eg. `Chlöe` and `Chloe` are the same artist). The `unidecode` library is used for decoding.
|
||||||
|
|
||||||
|
## The steps
|
||||||
|
|
||||||
|
The structure of the hash function looks something like this:
|
||||||
|
|
||||||
|
```py
|
||||||
|
def create_hash(*args: str):
|
||||||
|
# steps here
|
||||||
|
return created_hash
|
||||||
|
```
|
||||||
|
|
||||||
|
The `args` are an arbitrary number of string to use to create a hash. For example, when creating a track hash (track identifier) you pass it the track title, artist and album in that order.
|
||||||
|
|
||||||
|
```py
|
||||||
|
create_hash('Speedometer', 'Post Malone', 'AUSTIN')
|
||||||
|
```
|
||||||
|
|
||||||
|
The hash function follows the following steps when creating the hash:
|
||||||
|
|
||||||
|
1. Removes non-alphanumeric characters from the args separately
|
||||||
|
2. joining them in that order and converting to lowercase
|
||||||
|
3. generating a `sha1` hash of the resulting string
|
||||||
|
4. selecting 10 chars from the hash (concatenation of the first 5 and last 5 chars)
|
||||||
|
|
||||||
|
## Caveats
|
||||||
|
|
||||||
|
Ed Sheran!!!
|
||||||
|
|
||||||
|
This guy sums all the problems that could arise with our hashing thing. Have you ever checked out his discography? Look at the image below.
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
Notice the last 3 albums in the image? Their titles are non-alphanumeric characters. This means that once they follow our little procedure up there, they'll all have the same album hash. Thus, they'll all appear in the same album.
|
||||||
|
|
||||||
|
I experienced this sometime back and I fixed by using an if statement that states something like this: if you strip an `arg` and get an empty string hash it as is.
|
||||||
|
|
||||||
|
```py
|
||||||
|
create_hash('Drunk', 'Ed Sheeran', '=')
|
||||||
|
# 👆 leave this one alone
|
||||||
|
```
|
||||||
|
|
||||||
|
Scenarios like these outside Ed Sheeran are rare, but not non-existent.
|
||||||
|
|
||||||
|
Here's another one:
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
These 2 totally different albums will have the same hash because the plus sign on the 2nd album will be stripped.
|
||||||
|
|
||||||
|
If you can help fix this, a pull request is more than welcome. The hashing logic is contained in the `app/utils/hashing.py` file.
|
||||||
@@ -0,0 +1,50 @@
|
|||||||
|
# Code Base Docs
|
||||||
|
|
||||||
|
Hello there
|
||||||
|
|
||||||
|
If you would like to contribute a feature or fix a bug in this project, you're more than welcome to do so. The workings of Swing Music and its code base is documented here.
|
||||||
|
|
||||||
|
## How Swing Music works
|
||||||
|
|
||||||
|
The Swing Music architecture looks something like this:
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
When you launch Swing Music, music is indexed and stored in the database. The same data is also loaded into memory for faster retrieval. This data is the made accesible by the help of a Flask REST API.
|
||||||
|
|
||||||
|
Most `READ` requests retrieve data from memory instead of the database. While `WRITE` requests update both the database and the memory.
|
||||||
|
|
||||||
|
### Indexing tracks
|
||||||
|
|
||||||
|
Swing music crawls the selected root dirs and finds all [supported files](https://github.com/swing-opensource/swingmusic/blob/f62fe0ac24d3cb356f43c31882fd60ba0976e28b/app/settings.py#L101). It extracts metadata such as title, artists and album from the files and stores them in the database. Along with the extracted metadata, the`trackhash` and `albumhash` properties are added to help with duplicate detection.
|
||||||
|
|
||||||
|
For more on how the `trackhash` and `albumhash` are generated and used, check out the page on [hashing](./hashing.md).
|
||||||
|
|
||||||
|
## The data folder
|
||||||
|
|
||||||
|
The data folder is used to store all the files and data used in Swing Music. This includes databases, images, etc. In Linux, it's usually under `~/.config/swingmusic`. The directory is resolved using the XDG Base Directory Specification (Check out `app/utils/xdg_utils.py`).
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
This folder contains a few folders inside:
|
||||||
|
|
||||||
|
- `/assets` - stores static assets used by the clients. THink logos, etc.
|
||||||
|
- `/images` - stores thumbnails, artist images and playlist covers.
|
||||||
|
- `/plugins` - stores data used by plugins
|
||||||
|
|
||||||
|
The directories are created in `app/setup/files.py` by the `create_config_dir` function.
|
||||||
|
|
||||||
|
## The databases
|
||||||
|
|
||||||
|
The databases store data for persistence. There are 2 databases:
|
||||||
|
|
||||||
|
- `swing.db` - stores metadata
|
||||||
|
- `userdata.db` - stores user data. Favorites, playlists, settings, etc.
|
||||||
|
|
||||||
|
For more on databases, see the [databases page](./databases.md).
|
||||||
|
|
||||||
|
## Stores
|
||||||
|
|
||||||
|
You might have noticed that Swing Music is very fast. Prolly faster than your average streaming server. This is because Swing Music does not read the database on each requests (at least not most of the time). All the data in the database is also loaded into memory.
|
||||||
|
|
||||||
|
For more on this topic, see the [stores page](./stores.md).
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
# Playlists
|
||||||
|
|
||||||
|
Playlists are stored in the user data database. You can find the schema at `app/db/sqlite/queries.py`.
|
||||||
|
|
||||||
|
The methods for manipulating playlists in the database can be found at `app/db/sqlite/playlists.py`.
|
||||||
|
|
||||||
|
The api routes for playlists can be found at `app/api/playlist.py`.
|
||||||
|
|
||||||
@@ -0,0 +1,59 @@
|
|||||||
|
# Stores
|
||||||
|
|
||||||
|
Stores load all the database data into memory when Swing Music is booted.
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
This is done for a few reasons:
|
||||||
|
|
||||||
|
1. To make things fast and snappy
|
||||||
|
2. Inference and post processing
|
||||||
|
|
||||||
|
## Making things fast
|
||||||
|
|
||||||
|
Prior to the Swing Music project, I went through quite a lot of music players. One of the reasons I despised them all, is how often they would freeze when I opened a huge Music folder of Music or searched for something in my library of 40, 000 music files. The ones that didn't freeze often had a very huge delay before giving me what I requested.
|
||||||
|
|
||||||
|
This is the main reason that stores are at the core of how Swing Music works. The main goal was to make all read operations have a time complexity of `~O(1)`. ie. all requests should feel like they take a constant time which should be near 1ms. We all know that's impossible, but 13ms is not that bad either for a library of 25, 000 songs. Right?
|
||||||
|
|
||||||
|
## Inference and post processing
|
||||||
|
|
||||||
|
You might have noticed that Swing Music tries to extract album version info from an album. Something like this:
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
Other inferences and post processing activities that Swing Music does include:
|
||||||
|
|
||||||
|
- Removing remaster info from tracks
|
||||||
|
- Removing producers from track titles
|
||||||
|
- Extracting featured artists from track titles, etc.
|
||||||
|
|
||||||
|
All these activities are done when tracks are loaded into memory, when you start the app. Inferences and post processing helps standardize things and make things clean and relatable.
|
||||||
|
|
||||||
|
For example: If you have 3 versions of the Fleetwood Mac album `Rumours` like me, going to that album page will show you the other two at the bottom of the track list.
|
||||||
|
|
||||||
|
> [!TIP]
|
||||||
|
> Psst! You can disable the above behaviors anytime from the settings page.
|
||||||
|
|
||||||
|
## What about memory usage?
|
||||||
|
|
||||||
|
You might be thinking that storing thousands of items in memory would lead to high memory usage ... and you are right. But the thing is, if you have memory, why not use it to make your life snapier?
|
||||||
|
|
||||||
|
## The disadvantage
|
||||||
|
|
||||||
|
One of the disadvantages of using memory stores is that the app can't behave like a typical web server. That is, you can't use a WSGI server to run multiple instances if you need them. I couldn't even get gunicorn to run in the same thread as the stores (cries in C major), but that's not the problem here.
|
||||||
|
|
||||||
|
I don't know how many people share their entire music collection with thousands of users out there on the internet, because that's the only time you'd require that kind of setup. Starting from `v1.4.8`, Swing Music should be able to handle 10 users sending a crazy amount of requests to the server all at once (60r/s).
|
||||||
|
|
||||||
|
Now, a fix to this wsgi server situation (I think), would to move to [Shared Memory](https://docs.python.org/3/library/multiprocessing.shared_memory.html). That way, you can load all the tracks into a shared memory location and multiple instances can read it and write to it, if needed.
|
||||||
|
|
||||||
|
I haven't tried out anything yet, so it's all hypothetical. I haven't met anyone who needs this kind of setup so it might not happen. Plus it's a lot of work.
|
||||||
|
|
||||||
|
## The stores
|
||||||
|
|
||||||
|
There are 3 stores:
|
||||||
|
|
||||||
|
1. Track store - `app/store/tracks.py`
|
||||||
|
2. Album store - `app/store/albums.py`
|
||||||
|
3. Artist store - `app/store/artists.py`
|
||||||
|
|
||||||
|
You can guess what kind of data each store holds. Each store has **class methods** which manipulate or retrieve data from it. Stores are uses as classes and thus not instantiated.
|
||||||
Binary file not shown.
|
After Width: | Height: | Size: 34 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 41 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 291 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 79 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 26 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 38 KiB |
@@ -165,12 +165,28 @@ def check_folder_type(group_: group_type) -> str:
|
|||||||
|
|
||||||
|
|
||||||
def group_track_by_folders(tracks: Track):
|
def group_track_by_folders(tracks: Track):
|
||||||
|
"""
|
||||||
|
Groups tracks by folder and returns a list of groups sorted by last modified date.
|
||||||
|
|
||||||
|
Uses generator expressions to avoid creating intermediate lists.
|
||||||
|
"""
|
||||||
|
|
||||||
|
# INFO: sort tracks by folder name, then group by folder name
|
||||||
tracks = sorted(tracks, key=lambda t: t.folder)
|
tracks = sorted(tracks, key=lambda t: t.folder)
|
||||||
groups = groupby(tracks, lambda t: t.folder)
|
groups = groupby(tracks, lambda t: t.folder)
|
||||||
|
|
||||||
|
# INFO: sort tracks by last modified date in descending order to get the most recent last modified date
|
||||||
groups = (
|
groups = (
|
||||||
{"folder": folder, "tracks": list(tracks), "time": os.path.getctime(folder)}
|
(folder, sorted(tracks, key=lambda t: t.last_mod, reverse=True))
|
||||||
for folder, tracks in groups
|
for folder, tracks in groups
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# INFO: Return a generator of the groups
|
||||||
|
groups = (
|
||||||
|
{"folder": folder, "tracks": list(tracks), "time": tracks[0].last_mod}
|
||||||
|
for folder, tracks in groups
|
||||||
|
)
|
||||||
|
|
||||||
# sort groups by last modified date
|
# sort groups by last modified date
|
||||||
return sorted(groups, key=lambda group: group["time"], reverse=True)
|
return sorted(groups, key=lambda group: group["time"], reverse=True)
|
||||||
|
|
||||||
|
|||||||
+4
-3
@@ -4,6 +4,7 @@ import random
|
|||||||
|
|
||||||
from app.db.sqlite.albumcolors import SQLiteAlbumMethods as aldb
|
from app.db.sqlite.albumcolors import SQLiteAlbumMethods as aldb
|
||||||
from app.models import Album, Track
|
from app.models import Album, Track
|
||||||
|
from app.utils.customlist import CustomList
|
||||||
from app.utils.remove_duplicates import remove_duplicates
|
from app.utils.remove_duplicates import remove_duplicates
|
||||||
|
|
||||||
from ..utils.hashing import create_hash
|
from ..utils.hashing import create_hash
|
||||||
@@ -14,7 +15,7 @@ ALBUM_LOAD_KEY = ""
|
|||||||
|
|
||||||
|
|
||||||
class AlbumStore:
|
class AlbumStore:
|
||||||
albums: list[Album] = []
|
albums: list[Album] = CustomList()
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def create_album(track: Track):
|
def create_album(track: Track):
|
||||||
@@ -35,7 +36,7 @@ class AlbumStore:
|
|||||||
global ALBUM_LOAD_KEY
|
global ALBUM_LOAD_KEY
|
||||||
ALBUM_LOAD_KEY = instance_key
|
ALBUM_LOAD_KEY = instance_key
|
||||||
|
|
||||||
cls.albums = []
|
cls.albums = CustomList()
|
||||||
|
|
||||||
print("Loading albums... ", end="")
|
print("Loading albums... ", end="")
|
||||||
tracks = remove_duplicates(TrackStore.tracks)
|
tracks = remove_duplicates(TrackStore.tracks)
|
||||||
@@ -172,4 +173,4 @@ class AlbumStore:
|
|||||||
"""
|
"""
|
||||||
Removes an album from the store.
|
Removes an album from the store.
|
||||||
"""
|
"""
|
||||||
cls.albums = [a for a in cls.albums if a.albumhash != albumhash]
|
cls.albums = CustomList(a for a in cls.albums if a.albumhash != albumhash)
|
||||||
|
|||||||
@@ -4,6 +4,7 @@ from app.db.sqlite.artistcolors import SQLiteArtistMethods as ardb
|
|||||||
from app.lib.artistlib import get_all_artists
|
from app.lib.artistlib import get_all_artists
|
||||||
from app.models import Artist
|
from app.models import Artist
|
||||||
from app.utils.bisection import UseBisection
|
from app.utils.bisection import UseBisection
|
||||||
|
from app.utils.customlist import CustomList
|
||||||
from app.utils.progressbar import tqdm
|
from app.utils.progressbar import tqdm
|
||||||
|
|
||||||
from .albums import AlbumStore
|
from .albums import AlbumStore
|
||||||
@@ -13,7 +14,7 @@ ARTIST_LOAD_KEY = ""
|
|||||||
|
|
||||||
|
|
||||||
class ArtistStore:
|
class ArtistStore:
|
||||||
artists: list[Artist] = []
|
artists: list[Artist] = CustomList()
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def load_artists(cls, instance_key: str):
|
def load_artists(cls, instance_key: str):
|
||||||
@@ -24,7 +25,9 @@ class ArtistStore:
|
|||||||
ARTIST_LOAD_KEY = instance_key
|
ARTIST_LOAD_KEY = instance_key
|
||||||
|
|
||||||
print("Loading artists... ", end="")
|
print("Loading artists... ", end="")
|
||||||
cls.artists = get_all_artists(TrackStore.tracks, AlbumStore.albums)
|
cls.artists.extend(
|
||||||
|
get_all_artists(TrackStore.tracks, AlbumStore.albums)
|
||||||
|
)
|
||||||
print("Done!")
|
print("Done!")
|
||||||
for artist in ardb.get_all_artists():
|
for artist in ardb.get_all_artists():
|
||||||
if instance_key != ARTIST_LOAD_KEY:
|
if instance_key != ARTIST_LOAD_KEY:
|
||||||
@@ -110,4 +113,4 @@ class ArtistStore:
|
|||||||
"""
|
"""
|
||||||
Removes an artist from the store.
|
Removes an artist from the store.
|
||||||
"""
|
"""
|
||||||
cls.artists = [a for a in cls.artists if a.artisthash != artisthash]
|
cls.artists = CustomList(a for a in cls.artists if a.artisthash != artisthash)
|
||||||
|
|||||||
+5
-3
@@ -4,14 +4,14 @@ from app.db.sqlite.favorite import SQLiteFavoriteMethods as favdb
|
|||||||
from app.db.sqlite.tracks import SQLiteTrackMethods as tdb
|
from app.db.sqlite.tracks import SQLiteTrackMethods as tdb
|
||||||
from app.models import Track
|
from app.models import Track
|
||||||
from app.utils.bisection import UseBisection
|
from app.utils.bisection import UseBisection
|
||||||
|
from app.utils.customlist import CustomList
|
||||||
from app.utils.remove_duplicates import remove_duplicates
|
from app.utils.remove_duplicates import remove_duplicates
|
||||||
from app.utils.progressbar import tqdm
|
|
||||||
|
|
||||||
TRACKS_LOAD_KEY = ""
|
TRACKS_LOAD_KEY = ""
|
||||||
|
|
||||||
|
|
||||||
class TrackStore:
|
class TrackStore:
|
||||||
tracks: list[Track] = []
|
tracks: list[Track] = CustomList()
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def load_all_tracks(cls, instance_key: str):
|
def load_all_tracks(cls, instance_key: str):
|
||||||
@@ -23,7 +23,7 @@ class TrackStore:
|
|||||||
global TRACKS_LOAD_KEY
|
global TRACKS_LOAD_KEY
|
||||||
TRACKS_LOAD_KEY = instance_key
|
TRACKS_LOAD_KEY = instance_key
|
||||||
|
|
||||||
cls.tracks = list(tdb.get_all_tracks())
|
cls.tracks = CustomList(tdb.get_all_tracks())
|
||||||
|
|
||||||
fav_hashes = favdb.get_fav_tracks()
|
fav_hashes = favdb.get_fav_tracks()
|
||||||
fav_hashes = " ".join([t[1] for t in fav_hashes])
|
fav_hashes = " ".join([t[1] for t in fav_hashes])
|
||||||
@@ -44,6 +44,7 @@ class TrackStore:
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
cls.tracks.append(track)
|
cls.tracks.append(track)
|
||||||
|
print(f"\n A: Current track count:, {len(cls.tracks)} \n")
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def add_tracks(cls, tracks: list[Track]):
|
def add_tracks(cls, tracks: list[Track]):
|
||||||
@@ -52,6 +53,7 @@ class TrackStore:
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
cls.tracks.extend(tracks)
|
cls.tracks.extend(tracks)
|
||||||
|
print(f"\n E: Current track count:, {len(cls.tracks)} \n")
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def remove_track_obj(cls, track: Track):
|
def remove_track_obj(cls, track: Track):
|
||||||
|
|||||||
@@ -0,0 +1,14 @@
|
|||||||
|
from typing import Iterator
|
||||||
|
|
||||||
|
|
||||||
|
class CustomList(list):
|
||||||
|
# TODO: I think SharedMemoryList implementation will be done here.
|
||||||
|
# This list should be used as a normal list without any changes in the stores.
|
||||||
|
|
||||||
|
def __getitem__(self, index):
|
||||||
|
# Do some shared memory stuff here
|
||||||
|
return super().__getitem__(index)
|
||||||
|
|
||||||
|
def __iter__(self) -> Iterator:
|
||||||
|
# Do some shared memory stuff here
|
||||||
|
return super().__iter__()
|
||||||
@@ -1,10 +1,12 @@
|
|||||||
"""
|
"""
|
||||||
This file is used to run the application.
|
This file is used to run the application.
|
||||||
"""
|
"""
|
||||||
import logging
|
|
||||||
import mimetypes
|
|
||||||
import os
|
import os
|
||||||
from flask import request
|
import logging
|
||||||
|
import psutil
|
||||||
|
import mimetypes
|
||||||
|
from flask import Response, request
|
||||||
|
|
||||||
import waitress
|
import waitress
|
||||||
import setproctitle
|
import setproctitle
|
||||||
@@ -14,7 +16,7 @@ from app.arg_handler import HandleArgs
|
|||||||
from app.lib.watchdogg import Watcher as WatchDog
|
from app.lib.watchdogg import Watcher as WatchDog
|
||||||
from app.periodic_scan import run_periodic_scans
|
from app.periodic_scan import run_periodic_scans
|
||||||
from app.plugins.register import register_plugins
|
from app.plugins.register import register_plugins
|
||||||
from app.settings import FLASKVARS, Keys
|
from app.settings import FLASKVARS, TCOLOR, Keys
|
||||||
from app.setup import run_setup
|
from app.setup import run_setup
|
||||||
from app.start_info_logger import log_startup_info
|
from app.start_info_logger import log_startup_info
|
||||||
from app.utils.filesystem import get_home_res_path
|
from app.utils.filesystem import get_home_res_path
|
||||||
@@ -76,6 +78,34 @@ def serve_client():
|
|||||||
return app.send_static_file("index.html")
|
return app.send_static_file("index.html")
|
||||||
|
|
||||||
|
|
||||||
|
prev_memory = 0
|
||||||
|
|
||||||
|
|
||||||
|
# INFO: For debugging memory usage
|
||||||
|
# @app.after_request
|
||||||
|
def print_memory_usage(response: Response):
|
||||||
|
# INFO: Ignore assets
|
||||||
|
if (
|
||||||
|
request.path.startswith("/img")
|
||||||
|
or request.path.endswith(".js")
|
||||||
|
or request.path.endswith(".css")
|
||||||
|
):
|
||||||
|
return response
|
||||||
|
|
||||||
|
process = psutil.Process(os.getpid())
|
||||||
|
global prev_memory
|
||||||
|
current_mem = process.memory_info().rss
|
||||||
|
diff = (current_mem - prev_memory) / 1024**2
|
||||||
|
prev_memory = current_mem
|
||||||
|
|
||||||
|
# INFO: Print memory usage (highlights if diff is more than 0.1 MB)
|
||||||
|
print(
|
||||||
|
f"\n{request.path} | TOTAL: {current_mem/1024**2} MB | DIFF: {TCOLOR.FAIL if diff > 0.1 else ''}{diff} MB{TCOLOR.ENDC if diff > 0.1 else ''} \n"
|
||||||
|
)
|
||||||
|
|
||||||
|
return response
|
||||||
|
|
||||||
|
|
||||||
@background
|
@background
|
||||||
def bg_run_setup() -> None:
|
def bg_run_setup() -> None:
|
||||||
run_periodic_scans()
|
run_periodic_scans()
|
||||||
@@ -106,4 +136,11 @@ if __name__ == "__main__":
|
|||||||
host = FLASKVARS.get_flask_host()
|
host = FLASKVARS.get_flask_host()
|
||||||
port = FLASKVARS.get_flask_port()
|
port = FLASKVARS.get_flask_port()
|
||||||
|
|
||||||
waitress.serve(app, host=host, port=port, threads=10, ipv6=True, ipv4=True,)
|
waitress.serve(
|
||||||
|
app,
|
||||||
|
host=host,
|
||||||
|
port=port,
|
||||||
|
threads=10,
|
||||||
|
ipv6=True,
|
||||||
|
ipv4=True,
|
||||||
|
)
|
||||||
|
|||||||
Generated
+32
-30
@@ -2092,38 +2092,40 @@ testing = ["coverage (>=5.0)", "pytest", "pytest-cover"]
|
|||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "watchdog"
|
name = "watchdog"
|
||||||
version = "3.0.0"
|
version = "4.0.0"
|
||||||
description = "Filesystem events monitoring"
|
description = "Filesystem events monitoring"
|
||||||
optional = false
|
optional = false
|
||||||
python-versions = ">=3.7"
|
python-versions = ">=3.8"
|
||||||
files = [
|
files = [
|
||||||
{file = "watchdog-3.0.0-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:336adfc6f5cc4e037d52db31194f7581ff744b67382eb6021c868322e32eef41"},
|
{file = "watchdog-4.0.0-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:39cb34b1f1afbf23e9562501673e7146777efe95da24fab5707b88f7fb11649b"},
|
||||||
{file = "watchdog-3.0.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:a70a8dcde91be523c35b2bf96196edc5730edb347e374c7de7cd20c43ed95397"},
|
{file = "watchdog-4.0.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:c522392acc5e962bcac3b22b9592493ffd06d1fc5d755954e6be9f4990de932b"},
|
||||||
{file = "watchdog-3.0.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:adfdeab2da79ea2f76f87eb42a3ab1966a5313e5a69a0213a3cc06ef692b0e96"},
|
{file = "watchdog-4.0.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:6c47bdd680009b11c9ac382163e05ca43baf4127954c5f6d0250e7d772d2b80c"},
|
||||||
{file = "watchdog-3.0.0-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:2b57a1e730af3156d13b7fdddfc23dea6487fceca29fc75c5a868beed29177ae"},
|
{file = "watchdog-4.0.0-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:8350d4055505412a426b6ad8c521bc7d367d1637a762c70fdd93a3a0d595990b"},
|
||||||
{file = "watchdog-3.0.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:7ade88d0d778b1b222adebcc0927428f883db07017618a5e684fd03b83342bd9"},
|
{file = "watchdog-4.0.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:c17d98799f32e3f55f181f19dd2021d762eb38fdd381b4a748b9f5a36738e935"},
|
||||||
{file = "watchdog-3.0.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:7e447d172af52ad204d19982739aa2346245cc5ba6f579d16dac4bfec226d2e7"},
|
{file = "watchdog-4.0.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:4986db5e8880b0e6b7cd52ba36255d4793bf5cdc95bd6264806c233173b1ec0b"},
|
||||||
{file = "watchdog-3.0.0-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:9fac43a7466eb73e64a9940ac9ed6369baa39b3bf221ae23493a9ec4d0022674"},
|
{file = "watchdog-4.0.0-cp312-cp312-macosx_10_9_universal2.whl", hash = "sha256:11e12fafb13372e18ca1bbf12d50f593e7280646687463dd47730fd4f4d5d257"},
|
||||||
{file = "watchdog-3.0.0-cp38-cp38-macosx_10_9_universal2.whl", hash = "sha256:8ae9cda41fa114e28faf86cb137d751a17ffd0316d1c34ccf2235e8a84365c7f"},
|
{file = "watchdog-4.0.0-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:5369136a6474678e02426bd984466343924d1df8e2fd94a9b443cb7e3aa20d19"},
|
||||||
{file = "watchdog-3.0.0-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:25f70b4aa53bd743729c7475d7ec41093a580528b100e9a8c5b5efe8899592fc"},
|
{file = "watchdog-4.0.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:76ad8484379695f3fe46228962017a7e1337e9acadafed67eb20aabb175df98b"},
|
||||||
{file = "watchdog-3.0.0-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:4f94069eb16657d2c6faada4624c39464f65c05606af50bb7902e036e3219be3"},
|
{file = "watchdog-4.0.0-cp38-cp38-macosx_10_9_universal2.whl", hash = "sha256:45cc09cc4c3b43fb10b59ef4d07318d9a3ecdbff03abd2e36e77b6dd9f9a5c85"},
|
||||||
{file = "watchdog-3.0.0-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:7c5f84b5194c24dd573fa6472685b2a27cc5a17fe5f7b6fd40345378ca6812e3"},
|
{file = "watchdog-4.0.0-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:eed82cdf79cd7f0232e2fdc1ad05b06a5e102a43e331f7d041e5f0e0a34a51c4"},
|
||||||
{file = "watchdog-3.0.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:3aa7f6a12e831ddfe78cdd4f8996af9cf334fd6346531b16cec61c3b3c0d8da0"},
|
{file = "watchdog-4.0.0-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:ba30a896166f0fee83183cec913298151b73164160d965af2e93a20bbd2ab605"},
|
||||||
{file = "watchdog-3.0.0-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:233b5817932685d39a7896b1090353fc8efc1ef99c9c054e46c8002561252fb8"},
|
{file = "watchdog-4.0.0-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:d18d7f18a47de6863cd480734613502904611730f8def45fc52a5d97503e5101"},
|
||||||
{file = "watchdog-3.0.0-pp37-pypy37_pp73-macosx_10_9_x86_64.whl", hash = "sha256:13bbbb462ee42ec3c5723e1205be8ced776f05b100e4737518c67c8325cf6100"},
|
{file = "watchdog-4.0.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:2895bf0518361a9728773083908801a376743bcc37dfa252b801af8fd281b1ca"},
|
||||||
{file = "watchdog-3.0.0-pp38-pypy38_pp73-macosx_10_9_x86_64.whl", hash = "sha256:8f3ceecd20d71067c7fd4c9e832d4e22584318983cabc013dbf3f70ea95de346"},
|
{file = "watchdog-4.0.0-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:87e9df830022488e235dd601478c15ad73a0389628588ba0b028cb74eb72fed8"},
|
||||||
{file = "watchdog-3.0.0-pp39-pypy39_pp73-macosx_10_9_x86_64.whl", hash = "sha256:c9d8c8ec7efb887333cf71e328e39cffbf771d8f8f95d308ea4125bf5f90ba64"},
|
{file = "watchdog-4.0.0-pp310-pypy310_pp73-macosx_10_9_x86_64.whl", hash = "sha256:6e949a8a94186bced05b6508faa61b7adacc911115664ccb1923b9ad1f1ccf7b"},
|
||||||
{file = "watchdog-3.0.0-py3-none-manylinux2014_aarch64.whl", hash = "sha256:0e06ab8858a76e1219e68c7573dfeba9dd1c0219476c5a44d5333b01d7e1743a"},
|
{file = "watchdog-4.0.0-pp38-pypy38_pp73-macosx_10_9_x86_64.whl", hash = "sha256:6a4db54edea37d1058b08947c789a2354ee02972ed5d1e0dca9b0b820f4c7f92"},
|
||||||
{file = "watchdog-3.0.0-py3-none-manylinux2014_armv7l.whl", hash = "sha256:d00e6be486affb5781468457b21a6cbe848c33ef43f9ea4a73b4882e5f188a44"},
|
{file = "watchdog-4.0.0-pp39-pypy39_pp73-macosx_10_9_x86_64.whl", hash = "sha256:d31481ccf4694a8416b681544c23bd271f5a123162ab603c7d7d2dd7dd901a07"},
|
||||||
{file = "watchdog-3.0.0-py3-none-manylinux2014_i686.whl", hash = "sha256:c07253088265c363d1ddf4b3cdb808d59a0468ecd017770ed716991620b8f77a"},
|
{file = "watchdog-4.0.0-py3-none-manylinux2014_aarch64.whl", hash = "sha256:8fec441f5adcf81dd240a5fe78e3d83767999771630b5ddfc5867827a34fa3d3"},
|
||||||
{file = "watchdog-3.0.0-py3-none-manylinux2014_ppc64.whl", hash = "sha256:5113334cf8cf0ac8cd45e1f8309a603291b614191c9add34d33075727a967709"},
|
{file = "watchdog-4.0.0-py3-none-manylinux2014_armv7l.whl", hash = "sha256:6a9c71a0b02985b4b0b6d14b875a6c86ddea2fdbebd0c9a720a806a8bbffc69f"},
|
||||||
{file = "watchdog-3.0.0-py3-none-manylinux2014_ppc64le.whl", hash = "sha256:51f90f73b4697bac9c9a78394c3acbbd331ccd3655c11be1a15ae6fe289a8c83"},
|
{file = "watchdog-4.0.0-py3-none-manylinux2014_i686.whl", hash = "sha256:557ba04c816d23ce98a06e70af6abaa0485f6d94994ec78a42b05d1c03dcbd50"},
|
||||||
{file = "watchdog-3.0.0-py3-none-manylinux2014_s390x.whl", hash = "sha256:ba07e92756c97e3aca0912b5cbc4e5ad802f4557212788e72a72a47ff376950d"},
|
{file = "watchdog-4.0.0-py3-none-manylinux2014_ppc64.whl", hash = "sha256:d0f9bd1fd919134d459d8abf954f63886745f4660ef66480b9d753a7c9d40927"},
|
||||||
{file = "watchdog-3.0.0-py3-none-manylinux2014_x86_64.whl", hash = "sha256:d429c2430c93b7903914e4db9a966c7f2b068dd2ebdd2fa9b9ce094c7d459f33"},
|
{file = "watchdog-4.0.0-py3-none-manylinux2014_ppc64le.whl", hash = "sha256:f9b2fdca47dc855516b2d66eef3c39f2672cbf7e7a42e7e67ad2cbfcd6ba107d"},
|
||||||
{file = "watchdog-3.0.0-py3-none-win32.whl", hash = "sha256:3ed7c71a9dccfe838c2f0b6314ed0d9b22e77d268c67e015450a29036a81f60f"},
|
{file = "watchdog-4.0.0-py3-none-manylinux2014_s390x.whl", hash = "sha256:73c7a935e62033bd5e8f0da33a4dcb763da2361921a69a5a95aaf6c93aa03a87"},
|
||||||
{file = "watchdog-3.0.0-py3-none-win_amd64.whl", hash = "sha256:4c9956d27be0bb08fc5f30d9d0179a855436e655f046d288e2bcc11adfae893c"},
|
{file = "watchdog-4.0.0-py3-none-manylinux2014_x86_64.whl", hash = "sha256:6a80d5cae8c265842c7419c560b9961561556c4361b297b4c431903f8c33b269"},
|
||||||
{file = "watchdog-3.0.0-py3-none-win_ia64.whl", hash = "sha256:5d9f3a10e02d7371cd929b5d8f11e87d4bad890212ed3901f9b4d68767bee759"},
|
{file = "watchdog-4.0.0-py3-none-win32.whl", hash = "sha256:8f9a542c979df62098ae9c58b19e03ad3df1c9d8c6895d96c0d51da17b243b1c"},
|
||||||
{file = "watchdog-3.0.0.tar.gz", hash = "sha256:4d98a320595da7a7c5a18fc48cb633c2e73cda78f93cac2ef42d42bf609a33f9"},
|
{file = "watchdog-4.0.0-py3-none-win_amd64.whl", hash = "sha256:f970663fa4f7e80401a7b0cbeec00fa801bf0287d93d48368fc3e6fa32716245"},
|
||||||
|
{file = "watchdog-4.0.0-py3-none-win_ia64.whl", hash = "sha256:9a03e16e55465177d416699331b0f3564138f1807ecc5f2de9d55d8f188d08c7"},
|
||||||
|
{file = "watchdog-4.0.0.tar.gz", hash = "sha256:e3e7065cbdabe6183ab82199d7a4f6b3ba0a438c5a512a68559846ccb76a78ec"},
|
||||||
]
|
]
|
||||||
|
|
||||||
[package.extras]
|
[package.extras]
|
||||||
@@ -2304,4 +2306,4 @@ testing = ["coverage (>=5.0.3)", "zope.event", "zope.testing"]
|
|||||||
[metadata]
|
[metadata]
|
||||||
lock-version = "2.0"
|
lock-version = "2.0"
|
||||||
python-versions = ">=3.10,<3.12"
|
python-versions = ">=3.10,<3.12"
|
||||||
content-hash = "0d48f9d8aa4f824ae49406fca1819314c745ca93dc25874741ac8fa611e0df57"
|
content-hash = "74ff4493630a31bcae0bf41764e6d9f6c84fb920fc2abc1eafd7f8eec1881f44"
|
||||||
|
|||||||
+1
-1
@@ -9,7 +9,6 @@ python = ">=3.10,<3.12"
|
|||||||
Flask = "^2.0.2"
|
Flask = "^2.0.2"
|
||||||
Flask-Cors = "^3.0.10"
|
Flask-Cors = "^3.0.10"
|
||||||
requests = "^2.27.1"
|
requests = "^2.27.1"
|
||||||
watchdog = "^3.0.0"
|
|
||||||
Pillow = "^9.0.1"
|
Pillow = "^9.0.1"
|
||||||
"colorgram.py" = "^1.2.0"
|
"colorgram.py" = "^1.2.0"
|
||||||
tqdm = "^4.65.0"
|
tqdm = "^4.65.0"
|
||||||
@@ -25,6 +24,7 @@ setproctitle = "^1.3.2"
|
|||||||
flask-restful = "^0.3.10"
|
flask-restful = "^0.3.10"
|
||||||
locust = "^2.20.1"
|
locust = "^2.20.1"
|
||||||
waitress = "^2.1.2"
|
waitress = "^2.1.2"
|
||||||
|
watchdog = "^4.0.0"
|
||||||
|
|
||||||
[tool.poetry.dev-dependencies]
|
[tool.poetry.dev-dependencies]
|
||||||
pylint = "^2.15.5"
|
pylint = "^2.15.5"
|
||||||
|
|||||||
Reference in New Issue
Block a user