264 lines
10 KiB
Markdown
264 lines
10 KiB
Markdown
# (Audio Analysis | redis ) == <3
|
|
|
|
|
|
Redilysis sends audio analysis to a redis server.
|
|
|
|
The idea is to share a single audio analysis to many Visual Jockey filters, in our case for lasers.
|
|
|
|
### Requirements and installation
|
|
|
|
You need:
|
|
* python 2.7 / 3+
|
|
* an audio card
|
|
* a redis server
|
|
|
|
#### Custom install on Linux
|
|
|
|
Note that it will almost certainly fail on Debian 9/10/11. This is due to mandatory LLVM's version when compiling the numba library for librosa.
|
|
|
|
```
|
|
apt install -y --no-install-recommends build-essential gcc pkg-config python-dev portaudio19-dev libsndfile1 pulseaudio
|
|
pip3 install librosa numpy pyaudio redis statistics
|
|
git clone https://git.interhacker.space/tmplab/redilysis.git
|
|
cd redilysis
|
|
python redilysis.py --help
|
|
```
|
|
You will need to have pulseaudio and a local redis server.
|
|
|
|
#### With Docker
|
|
|
|
Here is an example. It is a dirty method, but it works :)
|
|
|
|
```
|
|
docker run -it \
|
|
-e NUMBA_CACHE_DIR=/tmp \
|
|
-e PULSE_SERVER=unix:${XDG_RUNTIME_DIR}/pulse/native \
|
|
-v ${XDG_RUNTIME_DIR}/pulse/native:${XDG_RUNTIME_DIR}/pulse/native \
|
|
-v ~/.config/pulse/cookie:/root/.config/pulse/cookie \
|
|
--group-add $(getent group audio | cut -d: -f3) \
|
|
protonphoton/redilysis:latest \
|
|
python redilysis.py -i 172.17.0.1
|
|
```
|
|
|
|
Two things here:
|
|
|
|
1. We are using here the `172.17.0.1` IP address to connect to the redis server. Change according to your needs.
|
|
2. The dirty part: Docker uses pulseaudio through your personal user and environment variables...
|
|
|
|
|
|
### Redis Keys and Contents for end users
|
|
|
|
**Each _word in bold and italic_ is a key which you can query the redis server for.**
|
|
```
|
|
# Example:
|
|
$ redis-cli get spectrum_10
|
|
"[2.21, 0.56, 0.51, 0.32, 0.27, 0.21, 0.18, 0.17, 0.18, 0.23]"
|
|
```
|
|
|
|
**_rms_**
|
|
* **Mode** spectrum
|
|
* **Type** float number
|
|
* **Length** scalar(1)
|
|
* **Meaning** Represents the root-mean-square -a mean value- for all frequencies between ```C0``` and ```C9```, e.g. between 12Hz and 8,372Hz.
|
|
* **Use** A fairly basic information about the scene audio volume.
|
|
* **Example**
|
|
* ```"0.12"```
|
|
* The audio volume for the scene is pretty low.
|
|
* It is obtained by averaging the RMS of every audio frame during the capture.
|
|
|
|
***spectrum_10***
|
|
* **Mode** spectrum
|
|
* **Type** array of float numbers (0.0-10.0)
|
|
* **Length** 10
|
|
* **Meaning** Represents the audio volume for the 10 **octaves** between ```C0``` and ```C9```, e.g. between 12Hz and 8,372Hz.
|
|
* **Use** A simple and useful way to get a global idea of the sound landscape.
|
|
* **Example**
|
|
* ```"[2.21, 0.56, 0.51, 0.32, 0.27, 0.21, 0.18, 0.17, 0.18, 0.23]"```
|
|
* The audio volume for the `C4` octave is `spectrum_10[4]`.
|
|
* That value being `0.27` is pretty low meaning almost no audio volume for that octave.
|
|
* It is calculated by averaging the volume of the octave's notes, e.g. `C4, D4, D#4, E4, F4, F#4, G4, G#4, A4, A#4, B4`.
|
|
|
|
***spectrum_120***
|
|
* **Mode** spectrum
|
|
* **Type** array of float numbers (0.0-10.0)
|
|
* **Length** 120
|
|
* **Meaning** Represents the audio volume for the 120 **notes** between ```C0``` and ```C9```, e.g. between 12Hz and 8,372Hz.
|
|
* **Use** More detailed than spectrum_10, it allows to find the standing out notes of the audio landscape.
|
|
* **Example**
|
|
* ```"[5.55, 2.61, 2.49, 1.79, 2.09, 4.35, 1.99, 1.57, 1.47, 0.77, 0.91, 0.89, 0.85, 0.56, 0.53, 0.73, 0.53, 0.46, 0.43, 0.44, 0.27, 0.45, 0.7, 0.81, 0.98, 0.7, 0.71, 0.6, 0.83, 0.51, 0.32, 0.31, 0.33, 0.24, 0.25, 0.33, 0.39, 0.43, 0.51, 0.28, 0.27, 0.25, 0.38, 0.25, 0.27, 0.3, 0.2, 0.27, 0.35, 0.29, 0.34, 0.3, 0.27, 0.27, 0.22, 0.21, 0.21, 0.29, 0.22, 0.28, 0.18, 0.19, 0.25, 0.26, 0.25, 0.24, 0.2, 0.21, 0.19, 0.18, 0.19, 0.17, 0.2, 0.17, 0.18, 0.17, 0.15, 0.17, 0.19, 0.18, 0.21, 0.16, 0.16, 0.18, 0.15, 0.13, 0.14, 0.16, 0.2, 0.17, 0.17, 0.2, 0.18, 0.16, 0.18, 0.15, 0.15, 0.16, 0.16, 0.19, 0.19, 0.19, 0.17, 0.18, 0.17, 0.19, 0.23, 0.23, 0.2, 0.23, 0.24, 0.36, 0.34, 0.23, 0.22, 0.2, 0.19, 0.18, 0.21, 0.21]"```
|
|
* The audio volume for the `C2` note is `spectrum_10[23]` (12x2 - 1).
|
|
* That value being `0.81` is average meaning there is some audio volume for that octave.
|
|
|
|
***bpm_sample_interval***
|
|
* **Mode** bpm
|
|
* **Type** float
|
|
* **Length** scalar(1)
|
|
* **Meaning** Represents the duration in milliseconds of the interval at which beat detection sample are done, in milliseconds. Beat detection require longer sampling duration than spectrum. The former requires intervals superior to 1s, while a 0.1s interval is sufficent for the latter.
|
|
* **Use** This is useful only if you try to guess future beats.
|
|
* **Example**:
|
|
* `"3000.0"`
|
|
* Each audio sample used to detect the beats is 3s long.
|
|
|
|
***bpm_delay***
|
|
* **Mode** bpm
|
|
* **Type** float
|
|
* **Length** scalar(1)
|
|
* **Meaning** Represents the duration in milliseconds of the time taken to make the sample and analyze it, excluding the time spent saving it to redis. In other words, it is (bpm_sample_interval + bpm treatment time).
|
|
* **Use** This is useful only if you try to guess future beats.
|
|
* **Example**
|
|
* `"3197.49093056"`
|
|
* The capture + detection time for this tempo detection was 3.19s
|
|
* If the `bpm_sample_interval` is 3.0s, it took 0.19s to analyze and detect beats in the sample.
|
|
|
|
**_bpm_**
|
|
* **Mode** bpm
|
|
* **Type** float
|
|
* **Length** scalar(1)
|
|
* **Meaning** Represents the tempo of the audio landscape, in Beats Per Minute (BPM).
|
|
* **Use** A simple way to know how fast the music goes, in musical environments
|
|
* **Expiration** This is the only key with an expire value in milliseconds set using's Redis ```PEXPIREAT``` command.
|
|
* Each time this key is saved to redis, it is set to expire in 2 * bpm_sample_interval.
|
|
* For example the key will be set expire in 6 000ms if the audio sample lasts for 3s.
|
|
* This is useful to know for how long the key has been in the redis by using the Redis ```PTTL``` command.
|
|
* For example, if you run the following command ```redis-cli PTTL bpm``` which results in ```"3848"``` provided bpm_sample_interval is equal to ```3000.0```
|
|
* It means you can compute when the bpm key was saved to redis i.e. ```2 * 3000 - 3848 = 2352 milliseconds``` ago, plus the TCP transaction time of the redis query.
|
|
* **Example**
|
|
* `"126.05"`
|
|
* There are ~126 beats per minutes.
|
|
|
|
|
|
**_beats_**
|
|
* **Mode** bpm
|
|
* **Type** list
|
|
* **Length** variable
|
|
* **Meaning** Represents the beats positions in the same
|
|
* **Use** This is useful only if you try to guess future beats. And in this case, this is the key information.
|
|
* **Example**
|
|
* `"[0.34829932 0.81269841 1.20743764 1.60217687 2.00852608 2.48453515]"`
|
|
* After the audio captured, the first beat was detected at 0.35 seconds, the second beat ath 0.81, and so on until the last beat at 2.48
|
|
* Based on the BPM and this information, we can project future beats around 2.90, 3.30, 3.70, etc.
|
|
* See below for beat projection
|
|
|
|
|
|
### Calculating the next beats at user times
|
|
|
|
This computation requires the following values
|
|
* bpm_sample_interval
|
|
* bpm_delay
|
|
* bpm
|
|
* pttl_delta : Double the bpm_sample_interval minus (-) the PTTL value of the bpm key in redis.
|
|
* last_beat : last beat time in sample
|
|
|
|
Redis_latency times are not considered here, but could be
|
|
|
|
Examples are given based on previous values for redis keys.
|
|
|
|
0. **How many second per beats?**
|
|
* seconds_per_beat = 60 / bpm
|
|
* ```60 / 126.05 =~ 0.4760```
|
|
1. **When did the capture start?**
|
|
* total_delay = bpm_delay + pttl_delta
|
|
* ```3197.49093056 + 2352 =~ 5549.5 ``` i.e. the capture started 5.5 seconds ago
|
|
2. **When was the last beat (in milliseconds)?**
|
|
* last_beat_delay = bpm_delay - last_beat*1000 + pttl_delta
|
|
* ```3197.49093056 - (2.48453515*1000) + 2352 =~ 3064.96 ``` i.e. the last beat was 3.1 seconds ago
|
|
3. **How many beats were there between the last beat and the redis `get bpm` query?**
|
|
* count_past_beats = floor( (last_beat_delay / 1000) / seconds_per_beat)
|
|
* ```(3064.96/1000) / 0.4760 = 6.439``` e.g. there were at least 6 beats
|
|
4. **When are the next beats relative to the redis bpm key retrieval time (in milliseconds)?**
|
|
* next_beats = f(i){ i * seconds_per_beat * 1000 - last_beat_delay } where i >= count_past_beats
|
|
* ```f(i){ i * 0.4760 * 1000 - 3064.96 }``` where i >= 4
|
|
* f(6) = 6 * 0.4760 * 1000 - 3064.96 = -208.9600
|
|
* f(7) = 7 * 0.4760 * 1000 - 3064.96 = 267.0400
|
|
* The next beat is in 267 milliseconds after the time we got the redis key
|
|
* f(8) = 8 * 0.4760 * 1000 - 3064.96 = 743.0400
|
|
* etc. until you have ```f(i) > bpm_sample_interval```
|
|
|
|
|
|
|
|
|
|
### Running redilysis: Common parameters
|
|
|
|
Two modes are available (see below for SPECTRUM and BPM), so you might need to run two processes for full analysis.
|
|
|
|
Here are the commmon parameters for both modes.
|
|
|
|
|
|
**Get the help**
|
|
```
|
|
python redilysis.py -h
|
|
```
|
|
|
|
**Run with debug info**
|
|
```
|
|
python redilysis.py -v
|
|
```
|
|
|
|
**Get a list of audio devices**
|
|
```
|
|
python redilysis.py -L
|
|
```
|
|
|
|
**Run with a given audio device**
|
|
```
|
|
python redilysis.py -v -d 5
|
|
```
|
|
|
|
**Run with a sampling frequency of 0.5s**
|
|
```
|
|
python redilysis.py -v -s 0.5
|
|
```
|
|
|
|
**Connect to redis on address 192.168.2.20 and port 6379**
|
|
```
|
|
python redilysis.py -v -i 192.168.2.20 -p 6379
|
|
```
|
|
|
|
**Change the internals of capture: run at 22000Hz with 2200 frames per buffer and 2 channels**
|
|
```
|
|
python redilysis.py -v -r 22000 -f 2200 -c 2
|
|
```
|
|
|
|
|
|
|
|
### Running redilysis in Spectrum Mode
|
|
|
|
**Choosing the spectrum mode**
|
|
```
|
|
python redilysis.py -v -m spectrum -s 0.1
|
|
```
|
|
This is the default mode.
|
|
|
|
It performs some frequency analysis (Fast Fourier Transform) to detect "energy" in the human audition bandwidths.
|
|
|
|
It will record if there is sound and at which frequencies.
|
|
|
|
It can run at sub-second frequency (100ms) with no problem.
|
|
|
|
It reports realistic data: spectrum analysis is the easy part.
|
|
|
|
### Running redilysis in BPM Mode
|
|
|
|
**Choosing the BPM mode**
|
|
```
|
|
python redilysis.py -v -m bpm -s 3
|
|
```
|
|
**Choosing a minimum and maximum BPM**
|
|
```
|
|
python redilysis.py -v -m bpm -s 3 --bpm-min 100 --bpm-max 200
|
|
```
|
|
|
|
This mode is less sure that the spectrum mode.
|
|
|
|
It must absolutely run with multiple seconds interval to work well. Three seconds is a correct minimum.
|
|
|
|
It attempts to detect beats based on audio "jumps" in intensity and energy.
|
|
|
|
To correct a well-known error called the "octave error" where the detected tempo is twice/half or thrice/third of the real tempo, you can use the Min/Max BPM. When the calculated tempo is outside of the range, it will attempt to find more legitimate values.
|
|
|
|
|
|
# License
|
|
|
|
Released under the GPLv3 License
|
|
|