Linux Scripts

From TonyWiki
Jump to navigation Jump to search

Introduction

Over the years, I have build up several useful scripts to use on Linux systems. I'll try to document them here!

Misc Stuffs

Mirror all files from a website

Mirror all the files off of a website, with the full folder structure

wget -m -p -E -k -K -np https://example.com/

Mount Seagate external drive without UAS

Normally, you can't run smartctl on a Seagate external drive since they use UAS (USB attached SCSI), which might make it a bit faster for planar drives, but smartctl doesn't support reading SMART data over UAS.

First, unmount and unplug the drive if it is in use

Then, remove the usb-storage and uas modules

sudo modprobe -r usb-storage
sudo modprobe -r uas

(or alternatively, use rmmod)

sudo rmmod usb-storage
sudo rmmod uas

Next, before you plug the drive in (which will auto-load uas), you need to load the usb-storage module with the quirk to disable uas for your Seagate drive.

To do this, get the vendor and product id by doing lsusb (the xxxx:yyyy stuff)

Finally, reload the module with the quirk as follows

sudo modprobe usb-storage quirks=0bc2:331a:u

Where the xxxx:yyyy is the vendor and product id for your drive, and the :u tells it to not use uas.

Microphone Loopback

If you're recording something and want to be able to hear something, then you can load the pulseaudio loopback module.

To load the module and start loopback:

pactl load-module module-loopback latency_msec=1

To unload and stop playback:

pactl unload-module module-loopback

If it's looping the wrong device, you can change it using the 'pavucontrol' program.

Map Wacom tablet to one display

By default, connecting a graphic tablet such as a Wacom will map the full resolution of the X11 window to the tablet. If you have more then one display, it will mess with the aspect ratio.

The solution is simple. If you are on X11, this can be done with xinput. You can also use xsetwacom with libinput.

First check the list of input devices by doing

xinput

This will return the list of input devices. You want to search for the one that is the pen or stylus for your tablet, as it is the one that send the mouse inputs. Copy the id of this device.

Then, check the display name you want to map it to by running

xrandr

or using a display manager of your choice.

Finally, to map the input to the display, run the following

xinput map-to-output <input> <display>

For example, to map input device 10 to display DVI-D-0

xinput map-to-output 10 DVI-D-0

Working with Files

Merge multiple PDF's together

Merging multiple PDF's can be done using several programs. I used to use ImageMagick, but it can destroy the resolution. Instead, ghostscript and pdftk provide nice options.

Using pdftk:

pdfunite in1.pdf in2.pdf out.pdf

Make sure you put out.pdf otherwise it will overwrite the last file! This has no loss in resolution (unlike the ghostscript version).

Cannot unmount USB drive

If you try unmounting a disk as follows (wherever you mounted it)

sudo umount /run/media/username/FlashDrive

and you get an error umount: target is busy, it can be very annoying to find the process that is using something from the drive.

You can use `lsof` to check which processes are accessing the files.

lsof | grep '/run/media/username/FlashDrive'

This will show you which process is using which file, which can be useful.


FFMPEG Tricks

Compress all FLAC in subfolders to OPUS

Compress all FLAC audio files in subfolders to OPUS at 128 Kbit/s (near-transparent), while maintaining the metadata tags and album art.

find ./ -name '*.flac' -exec bash -c 'ffmpeg -i "$0" -f flac - | opusenc - --bitrate 128 --vbr --comp 10  "${0/flac/opus}" && rm -v "$0"' {} \;

We use find to select all the flac files, and run the bash files. The first runs ffmpeg and pipes the output to opusenc. The reason is if you use ffmpeg to save to opus files, the album art is lost, whereas when piped in to opusenc, it is encoded properly.

Visualize music into notes

Sometimes I get curious and want to break down a song to it's core melodies, FFMPEG has a cqt filter (like fft), that can convert an audio file into a visual based on the notes. I also realized you can do this in real-time with decent latency using ALSA.

ffmpeg -f alsa -ac 2 -i pulse -filter_complex "[0:a]showcqt=s=1280x720:fps=30:[out]" -map "[out]" -f sdl -
  • This will try capturing a microphone by default, use 'pavucontrol' to change the captured input to loopback for your speaker.
  • I tried to use '-f pulseaudio' instead, but it just ends up being very stuttery and with much higher latency, whereas the alsa version is so much smoother and lower latency.
  • You can change the resolution and fps are needed, I sometimes daisy-chain multiple effects.
  • Output as SDL has much lower latency from my testing. I used to pipe it into mpv with no buffer, but this is even faster


Older revision of this command I used to use as reference:

ffmpeg -f pulse -ac 2 -i default -filter_complex "[0:a]showcqt=s=1280x720:fps=30:[out]" -map "[out]" -r 30 -c:v rawvideo -f matroska - | mpv --no-cache --untimed --no-demuxer-thread -

This stutters a bit more, but might be preferred in some cases. It also uses pulseaudio directly. I switched to using rawvideo instead of other codecs, since rawvideo has a much lower latency, as there is no need for compression and decompression, which doesn't matter since we're piping it for playback anyway.

Stream desktop to Icecast

If you need to stream to many computers at once, Icecast is pretty useful. FFMPEG can directly stream to an Icecast server (alternatively, you can use similar settings in OBS). I capture the x11 window, then encode using VAAPI for reduced CPU usage.

ffmpeg -vaapi_device /dev/dri/renderD128 -video_size 1920x1080 -framerate 30 -f x11grab -i :0.0+0,0  -vf 'hwupload,scale_vaapi=w=1920:h=1080:format=nv12' -c:v h264_vaapi -f mpegts -content_type 'video/mp2t' "icecast://source:password@example.com:8000/name.ts"
  • The coordinates after the -i is width, height, left offset, top offset.
  • The video filters are to scale to 1080p, while maintaining the vaapi encoder.
  • I use mpegts to stream the video, though other file formats that allow streaming such as matroska also work. If you're streaming theora/vorbis, use ogg, for audio, use opus if you're encoding opus, so on.
  • I set the content type to video/mp2t, which makes the video stream properly detect. It also makes the m3u file download rather then play in the browser.

Useful Programs

(ncdu) Find out what is using the disk space

Sometimes, when your disk is almost full, you want to know what is using up the disk space. On Windows, tools like WinDirStat achieve this function.

On GNU/Linux, du will list the space of files used. However, the ncurses version is more useful

ncdu

It will print out in descending order the list of items using disk space.