We separated the design and testing into the hardware (LED cube
construction and Arduino control) and software (music processing, music visualization, and
Raspberry Pi and mobile phone control) parts. The design concept is to perform real-time music
analysis visualization on the cube, and to provide user-friendly control between the mobile
interface and the LED cube. We also need to minimize the pins that are used by the LEDs to
realize the hardware buildup, and this helps enhance the performance of the music analysis
process.
Hardware Design
To control 216 RGB LEDs separately, we inevitably need to implement
multiplexing to achieve full control of the device in a limited amount of pins. Our original
plan is to use multiplexers to control all LEDs based on the reference of the previous student
work[1]. However, we need more than 14 pins (the total pins they used to control 512
single-colored LEDs) for our RGB LEDs, which may be impossible to fit on an Arduino Uno, not
even an Arduino Mega. Thus, we chose to use shift registers instead since they can be
daisy-chained to control more than eight output pins with three inputs.
LED cube construction
We used common cathode(-) RGB LEDs to build up the whole cube. For each
horizontal plane, which is also called a layer, the cathodes of the 36 LEDs are connected
together to ground. First, we soldered six layers of 6x6 LED planes separately, and then
inserted the wires of the desired length into the holes cut off on the base plane. Those wires
are the pillars to connect each LED in the same column of each plane together and to stabilize
the structure. Each LED has three wires connected to red, green, and blue signals. The most
important part is to make sure no LED connection is shorted in the soldering process.
Electronics
As mentioned above, the ground pins of each LED on the same plane are
connected together, and it is controlled by one N-channel MOSFET. The MOSFET is connected to an
output pin on a shift register, so the first shift register records the data to ground each
layer separately. As for the input signal of the LEDs, we only implement the green signal in the
current situation due to the time limit of this project. The wire of each green input signal of
an LED column is soldered to a wire, and the wire would be an output pin on a shift register.
Since we have 36 LED columns, we need 5 shift registers (8 output pins per each) to control all
of them. Shift registers can be connected to one another by the output enable pin (pin 9) with
the same clock signal. Thus, we chained the first shift register with the rest of them to
control all of the pins with only three digital pins on the Arduino.
Process and Testing
To better understand multiplexing, we first tried to control a 3x3 LED
array by six digital pins (3 inputs for each row and 3 outputs for each column).
Then, when we moved on to change our design to utilize shift registers, we tested the basic
control of 8 LEDs in one shift register before chaining all six of them together. Originally, we
planned to connect the shift register controlling the ground of each layer to a separate set of
pins. However, after seeing another reference[2] showing the feasibility to control all signal
data altogether, we chained all six shift registers together for the process. The only thing we
need to remember is the data sequence of each output pin to send out the right signal. Overall,
the decision to implement the electronics of one color first instead of building all three
colors helps a lot in the building and debugging process, and it also saves a lot of time to
have a functioning prototype by the deadline.
Software Design
The software for the music visualizer setup consisted of python packages
such as Pandas, Librosa, BlueALSA, FIFO and serial.
We divided the software design into parts: music processing, music visualizer, playback control,
and player & serial communication.
Music Processing
We firstly designed to set up a real-time audio analyzer as we play the
music. However, we wanted the system to be controlled by mobile phones. To achieve that, it was
necessary to be equipped with a sound card or microphone. In the end, we adopted a concise way.
We made the music playing and music analyzing separately, which means there is no need for us to
collect the real-time music signal. We download the music file in mp3 format, and save the
preprocessing file for the purpose of reading it while the mobile phone plays the music.
After we got the mp3 file, we analyzed the music by Librosa and transformed it to the spectrom.
Since we have a 6*6*6 cube, we designed to divide the spectrogram into
six pieces. For
convenience, we selected six frequencies from the spectrogram range from 100 to 8000. Then we
extract the loudness level of these frequencies by time, the result would be a matrix. we saved
the matrix in csv by Pandas for the purpose of reading while music playing.
Music Visualizer
We designed a class to read the music data. Before we generate the audio
analyzer, we need to define the name of the music for the program to find the required folder
whose name is the current playing music.
We will call some functions in the class later, while the music is playing. For example, we want
to know the decibel of the time T and the first frequency F. We input the two parameters of the
get_decibel function, the return value would be the decibel of the specified time and frequency.
Playback Control
In this part, we played the music on the mobile phone, and the bluetooth
transmitted the audio to the Raspberry Pi. The Raspberry Pi received streamed audio from the
A2DP (Advanced Audio Distribution Profile) protocol, we referred most of the script from
https://scribles.net/controlling-bluetooth-audio-on-raspberry-pi/#Ref01. Before we followed the
instructions, we were required to enable Audio Profile Sink Role and pair the mobile phone with
Raspberry Pi 4. Here is the
link:https://scribles.net/streaming-bluetooth-audio-from-phone-to-raspberry-pi-using-alsa/.
After pairing the devices, the key point was to capture the audio streams from Bluetooth devices
and play them on Raspberry Pi 4. The bluez-alsa-utils helped to do that. After we install
Bluetooth Audio ALSA Backend on internet, we simply run the script on terminal:
“bluealsa-aplay 00:00:00:00:00:00” and test whether the speaker made sounds as the mobile phone
was playing the music.
After it works, we run the code from the instructions and see some of the information of the
music playing.
pause
Playback Status: paused
play
Playback Status: playing
next
Music Info:
Title: Warriors
Artist: Imagine Dragons
Album: 2WEI
vol 100
The first, third and fifth lines were the control commands from the
terminal. The rest of the lines showed the information of the music while the playing status
changed. Since there has been a special loop calledGLib Main loop, we were unable to take any
extra actions in the main program. We decided to modify the program to send messages to the
other program using FIFO for the purpose of communication with different programs.
Since we need to send the information of the playing status to the other python program.
We applied os.mkfifo to create a fifo file when the program detected the status change.
Meanwhile, the other program (main program) kept reading the fifo file if it existed. When we
followed the method, we found that the two programs did not continue until both the programs
finished the read/write operations. However, as long as the while loop does not stop, the read
operation would not stop, the programs would be stuck. To solve the problem, we just simply made
the FIFO file deleted after reading. Then when the status changed, the message sender created a
new FIFO to send to the main program.
Player & Serial communication
For this part, we need to combine all the above programs to calculate the
correct output of the cube lights, and send the messages to Arduino via serial.
Time Computing
The basic idea for the calculation was:
Each time we press the pause and continue button, the main program will get the message from
FIFO, we can then update the playing time by the message. In addition, when the music changes,
it will also receive the name of new music. Therefore, we designed a loop to keep asking the
information from the FIFO, and if there is no update, then the program will compute the playing
time by itself. The basic idea for computing the time is every time it comes to the line
mentioned, it will add playtime to the difference of current time and last time. And then it
saves the current time as the last time for the preparation of the next computing.
Serial communication
In the serial communication, we applied a serial package which actually communicates with Arduino by UART. It is called Universal asynchronous receiver-transmitter, and the method is widely used by the communications between Raspberry Pi and Arduino. After connect Arduino to Raspberry Pi by USB, we run the code in python:
ser = serial.Serial('/dev/ttyACM0', 9600, timeout=1)
If there is no error, it shows the system has detected the device and the device is ready to transmit the data. Noted that UART can only transmit data in bytes, we transformed the data to bytes by python command
command = bytes(out_str, 'utf-8')
We also found a problem when we tried to implement the method. The data transmission would be slower after the music playing for around 30 seconds. The reason why it became slower was that the buffer has been full and it should pop out each element when new data came in, which dramatically reduced the transmission speed. To solve the problem, we reset the input and output buffer every time before we wrote the data to the serial.
