State of Project Segfault: June 2023
This month we had some unfortunate events such as the IN Node DDoS and the failed Zen migration. However, we still got stuff done and recovered from our failures. We also had a new member join our team this month.
Server News
IN Node DDoS
Near the beginning of this month, we observed a DDoS attack targeting the IN Node.
Due to Arya being away, the node was forced to be shut down, resulting in a downtime that lasted three days.
However, we were prepared for this and redirected traffic to our nodes in other regions. This allowed us to minimize the downtime and keep services available.
CrowdSec
To reduce the impact of DDoS attacks, we deployed CrowdSec on all of our nodes.
It is a state of the art DDoS protection tool, that blocks IPs on conditions that may be triggered.
For example, if an IP is making too many ssh connections, CrowdSec can automatically add a rule to drop traffic originating from that bad IP.
As for the "Crowd" in the name, CrowdSec maintains a shared database of abusive IPs, just like AbuseIPDB, which is used to infer whether the IP might be a first time offender or a known one which could be insta-blocked.
(Failed) Zen Migration
On June 10th, we started migrating VMs from Soleil Levant to the new Zen node.
The process went smoothly, but took several hours since several terabytes of data were needed to be transferred over a 1Gbit link.
By the time all the VMs were migrated and initially turned on, everything was fine. However, we had to reallocate memory to the VMs to fit within 128GB. Unfortunately, things got worse. Soon, we experienced database slowdowns which affected many of our services including Matrix. We attempted to fix this problem in which the root cause very slow drive I/O times. While we managed to make small improvements, the bad disk speeds were still a problem. We were all considering migrating back to Soleil, and we were forced to after a drive in our ZFS pool failed.
About the electricity consumption, apparently the price was a lot lower than the French average for MrLeRien and hence we would be able to continue on Soleil Levant.
Debian 12 Migration
We updated all our servers to Debian 12, not much to say here :)
Upcoming US Node migration
As our server credit for our US Node runs out, we plan to move it to another provider in about 1-2 months time. Due to us regularly exceeding bandwidth limits, the server credit is running out much earlier than expected.
Member News
Resignation of Akis
Unfortunately, Akis resigned this month due to inactivity and because of disagreements with the rest of the team. We wish him the best of luck!
New member joins our team
This month, we introduce a new member: py_. He has extensive experience with UI design and web development. We are excited to see what he can do to benefit the project.
Instance News
psf.lt Dendrite
We are running a backup Matrix homeserver on psf.lt on the IN Node, using the Dendrite Matrix server implementation. The primary purpose of this homeserver is for communication while our primary EU node is down. It is currently closed for registration, but you may ask us for an account.
SimplyTranslate
Google Translate and DeepL are two of the most popular machine translation options out there, but both unfortunately collect user data. SimplyTranslate is a privacy frontend that lets you use both of these engines, including ICIBA and Reverso. It even shows you the TTS that Google generates.
You can use our SimplyTranslate instance at https://translate.projectsegfau.lt or https://tl.psf.lt
Safetwitch
We now host an instance of SafeTwitch, a new privacy frontend for Twitch.
You can check it out at https://safetwitch.projectsegfau.lt / https://tw.psf.lt
Kbin
Because of the recent Reddit API changes, many now-former Reddit users have been found Kbin to be the next best alternative. Kbin is a decentralized content aggregator, much like Reddit. So for you now-former Reddit users, we now host a Kbin instance!
Why not Lemmy? We chose Kbin because it's already fully compatible with Lemmy. And because Lemmy was giving us too many issues for to handle.
You can use our Kbin instance at https://kbin.projectsegfau.lt or https://kb.psf.lt
Conclusion
Although this month may have not been the smoothest, we managed to push through and improved our infrastructure, as well as releasing new instances. Massive thanks to everyone who has been supporting us thus far!
See you next month :)
-Devrand, Arya and py_