milestones
woah
i don't get to update the site much but i do check on it frequently. it's running off a small rented server that started out as nothing more than a fresh ubuntu install and my ssh keys loaded onto it. this means i'm the only person maintaining it :P
adventures in brotli
i'm writing this post because earlier today i visited this site to just see if it was kicking - and lo, i got a 502 error. it was odd - i checked my DNS and everything was good. but the weird thing was... i could still ssh into the server, using the @zardo.us domain!
knowing i could ssh in meant the domain name was working correctly, and the server was managing its ports correctly.
at this point i've learned it's best to go ahead and start from the bottom. sudo reboot to restart the box and i waited briefly for it to cycle back. after refreshing the webpage and continuing to get a 'connection refused' error, i fired up an SSH tunnel - once again connecting to zardo.us immediately.
it's weird because it seems so simple, but what followed next is a good example of how all the random hobbies i've pursued over the last few years have all sort of coalesced into useful experience and information.
because i have spent so much time working with nginx, via ssh, on servers guarded by ufw... i'd become familiar enough with networking to understand that i was having an issue in which knowing about ports would be useful. http:// connections happen through a different port than SSH connections - the former by default happening via port 80, the latter port 22.
since my SSH port was apparently in working order, and sudo ufw status showed the firewall running on all cylinders, protecting the right ports... i deduced that the issue likely related to the webserver itself, nginx.
brotli!!!!
i keyed in sudo systemctl reload nginx. eureka!!! because the reply was that nginx could not be reloaded... because it wasn't running. weird, but hey maybe during all my webserver standup i forgot to run sysctl enable to make sure nginx started up on boot, and cycling the server's power caused nginx to simply not start.
so i ran systemctl start nginx - and got a failure message. odd, i thought, since i compiled nginx from the sourcecode myself.
here i had alot of help thanks to developing a habit of compiling software when and where i can. initially, this started after my first experience with compiling a piece of code - specifically after learning about configure flags. using flags, you can instruct the compiler to do a variety of things - depending on the target you're compiling for and the hardware it runs.
the obsessive part of me loved this because it meant i could bake my own versions of common open-source software, but tweak the flags to optimize the code for the box i would be using it on. 'release' versions of software generally avoid using too many flags like this and opt for a build that is the most 'generic' and therefore usable on the widest variety of systems. an example not based on any specific real event but plausible would be having a compiler that could compile code to take advantage of a new, proprietary cpu design on Intel chips that would cut down on loading times. however, this flag might not be used for a 'release' version because it would either break or bug out when used on systems using AMD chips.
so when i got this failure to start message, i knew right away the issue might relate to my build. the only thing that had changed from a working website to this, was my occasionally logging on to run sudo apt update.
and then i remembered that i had added an nginx launchpad repository maintained by the community with development builds - this might be different now but at the time, the mainline version of nginx avaliable in the apt repository was the last version before nginx rolled out support for modules that could be dynamically linked and loaded when compiled against the nginx source code.
so first i ran journalctl -xe as root to confirm a hypothesis. the error pointed to an issue with the Brotli module, which was showing as being compiled for the 1.19.6 version of nginx. that's the version i have, i thought. but... i ran nginx -v anyways.
i saw i was, in fact, running version 1.19.7!!
besides the practice and habit of compiling as mentioned earlier, i rolled my own nginx for an important reason: so i could install a newer version of nginx with module support, and then install a module to allow utilizing brotli for faster page load times.
and there it was. the solution was as simple as running wget to grab the source code for the correct codebase version, re-using the configure flags to generate a makefile, running make modules and linking the compiled brotli module to the nginx/modules directory.
reflecting on this
the thing was - this all happened so fast i didn't have time to really document this. it's why i didn't paste the actual errors returned when running systemctl reload, journalctl -xe, et cetera - this entire event unfolded over a period of maybe ten minutes, from the time i visited zardo.us and got the 502 bad gateway error, to running systemctl reload nginx with the updated brotli module and refreshing the website to see that it had returned.
small steps
i guess this post is something of a celebrating my own progress type of thing. but i think the reason i'm publishing this for an audience is because i know alot of people who want to get into development but there's always obstacles both real and imagined. and there's people who want to get into development but they are chasing money or security.
i've got opinions on all that but the point here is, at the time i bought my first raspberry pi, i was pretty broke. like the kind of broke where i agonized over actually buying the RPi for months, because for me sixty or so bucks was a month of groceries.
and man i was tired. warehouse labor jobs are hell, from the shitty hours to the shitty pay to the shitty benefits which don't help the shitty lower back.
but since forever i've always loved technology. not in some idealistic 'this will change the world' sort of way - just as a mystery or a puzzle to solve. knowing how things work.
so i bought a raspberry pi. at first i just wanted to run ubuntu and get myself a cool terminal with a stylish prompt. but then to do that i had to learn how to use things like npm, pip, and cargo. i wanted more and i learned about zsh and oh-my-zsh. i wanted more!!! and i learned about fonts and glyphs and powerline symbols and how terminals render text. by this point i realized how easy it was to get shit done via commandline, so i severed ways with the RPi hdmi connection and went headless. i learned how to use ssh. i wanted a cool hacker tunnel session vibe so i learned how to use screen and tmux. i missed irc so i learned how to run irssi, which in turn helped me learn more about tmux and screen.
i wanted to try out projects on my RPi and i learned that alot of them are essentially webserver frontends that serve as easy GUI panels that translate user choices into terminal commands. it made me wonder if i could point these home servers outward, so i learned about networking, DNS services, security and at some point i found myself running websites out of my bedroom.
the takeaway
over the course of a few years my natural curiosity and a readily-avaliable mini-computer for testing shit bounced me around from one thing to another. i'd lurk github and see interesting projects - and i learned about a variety of build systems in the process. it's cool to clone a git repo, run ls, see a telltale file like Makefile or Cargo.toml, and know exactly what command to run. and if it doesn't work? it's even cooler working out the correct way to build a project using nothing but the documentation on the system itself - whether it be perusing a README or a man topic.
i've used a github project to decompile proprietary firmware to re-assemble into a jailbroken custom OS for my android phone. i've built a docker VM that cross-compiles Rust crates, with dependencies that are no longer maintained, for targets that are no longer supported by gcc.
guess what i am trying to say is - compiling a custom module for my RYO webserver running from a $5/month virtual server i built on a fresh ubuntu install was something i never intended to be able to do. it's something that happened as a sort of accidental byproduct of just playing around with computers on my own dime and time.
and that is pretty cool so i just wanted to type all this as a dedication to a single moment in time. i read alot of posts by folks interested in becoming developers but not knowing where to begin. front-end? back-end? "full stack"? web? system? mac? linux? and my bearded zen-master reply would be: if you're interested then follow your interest. in my experience there's no clear divisions between all these terms. i once talked with an employed Java programmer who worked for Apple, who had no idea what the concept of architectures like 'x86' or 'arm64' even meant. it haunts me because the dude paid money to somehow not learn something and how it relates to the something he did learn about.
but then again i have no idea what the concept of 'turning all this experience into financial reward' means, so maybe the joke is on me.
~,