Infrastructure

Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode

An in-depth analysis of server reliability issues encountered with Hetzner's AX162 server line, which initially showed a 16x higher crash rate than its predecessor. Through extensive debugging and multiple hardware iterations, the issues were eventually resolved with newer motherboard versions, demonstrating the risks of early hardware adoption.

14 years of systemd

Lennart Poettering's FOSDEM keynote discusses systemd's 14-year evolution from an init system to a comprehensive suite of Linux OS building blocks, highlighting its growth to 150 binaries, adoption by major distributions, and future focus on boot integrity, IPC improvements, and image-based deployments.

searchcode.com’s SQLite database is probably 6 terabytes bigger than yours 2025/02/16 (1949 words)

A developer details the migration of searchcode.com's database from MySQL to SQLite, resulting in what might be the world's largest SQLite database at 6.4TB. The migration involved implementing BTRFS compression, upgrading to a powerful server with an Intel Xeon CPU, and successfully maintaining performance across all operations.

Siren Call of SQLite on the Server

An analysis of using SQLite in server-side production environments reveals that while technically capable, it introduces unnecessary complexity compared to traditional client-server databases. Despite SQLite's reliability and performance, implementing features like high availability and backups requires additional infrastructure, potentially negating its main benefits.

Larry Ellison wants to put all US data in one big AI system

Larry Ellison advocates for centralizing all national data, including genomic information, into a single Oracle database system for AI analysis, suggesting this approach would improve healthcare and government services. Speaking at the World Governments Summit in Dubai, the Oracle tycoon promoted the idea of comprehensive data unification while dismissing privacy concerns, just months after endorsing real-time population surveillance.

DOGE as a National Cyberattack - Schneier on Security

A new Department of Government Efficiency (DOGE) has gained unprecedented access to critical US government systems, including Treasury, USAID, and OPM, bypassing essential security protocols and potentially exposing sensitive data. The breach involves uncleared personnel making system modifications while dismantling security measures, creating vulnerabilities that could be exploited by foreign adversaries.

How We Migrated Checkly From Heroku to AWS

A detailed account of Checkly's migration from Heroku to AWS in 2022, driven by PostgreSQL upgrade limitations and maintenance window constraints. The migration involved careful planning, data replication strategies, and resulted in a successful transition with minimal downtime, offering valuable insights for similar cloud platform migrations.

Backblaze Drive Stats for 2024

Backblaze manages 305,180 drives with a focus on data drives performance and failure rates throughout 2024, revealing an overall improvement in annualized failure rates from 1.70% in 2023 to 1.57% in 2024. The company has successfully integrated new high-capacity 24TB Seagate drives while phasing out older 4TB models, demonstrating their ongoing storage optimization strategy.

Meta’s Hyperscale Infrastructure: Overview and Insights

Meta's hyperscale infrastructure emphasizes rapid deployment, technology openness, and shared resources across global datacenters, enabling efficient processing of billions of user requests through an integrated system of PoPs, CDNs, and datacenter regions. The infrastructure employs innovative approaches to boost developer productivity through continuous deployment and serverless functions, while reducing hardware costs through global resource optimization and hardware-software co-design.

QUIC action: patching a broadcast address amplification vulnerability

A group of researchers discovered a broadcast amplification vulnerability in Cloudflare's QUIC implementation, where a single packet to a broadcast IP address could trigger multiple responses from server workers. The vulnerability, which has been fully patched, highlighted how broadcast functionality combined with SO_REUSEPORT socket options can create significant amplification risks in UDP-based services.

Cloudflare incident on February 6, 2025

A 59-minute Cloudflare R2 storage outage occurred on February 6, 2025, causing widespread service disruptions across multiple Cloudflare products due to human error during phishing site remediation. The incident resulted in 100% failure rates for R2 operations and affected dependent services like Stream, Images, and Cache Reserve, though no data was lost or corrupted. Cloudflare has implemented immediate safeguards and is developing additional system-wide controls to prevent similar incidents.

Cities Can Cost Effectively Start Their Own Utilities Now

An analysis of PG&E's electricity rates reveals significant price disparities, with PG&E charging 40 cents per kilowatt hour compared to other utilities charging around 17 cents. Cities could potentially save residents $800-$1200 annually by creating their own utility companies, with a detailed case study of Walnut Creek demonstrating how municipal utilities could operate more efficiently and reduce costs by approximately 25%.

February 2025: Starlink in the Falkland Islands - a national emergency situation!

Starlink users in the Falkland Islands are facing service terminations as their 60-day roaming period expires, despite widespread adoption by hundreds of residents and a successful petition backed by 70% of the population. The service remains technically illegal due to Sure International's monopoly and lack of official government approval, even for users who paid the £5,400 FIG VSAT license fee. A resolution is being delayed until April 2024, prompting discussions about declaring a National Emergency to expedite the approval process.

The origins of 60-Hz as a power frequency

The global adoption of 50 Hz and 60 Hz power frequencies stems from decisions made by Westinghouse (Pittsburgh) and AEG (Berlin) engineers in 1891, which continue to impact electrical systems worldwide. The evolution of these frequency standards spans from early experimental phases through lighting systems development to power systems interconnection, shaping the electrical infrastructure across the USA, Europe, and Japan.

Baltic Grid Frequency

Engineers collaborate across Europe to develop a web application for monitoring power grid frequency data from the Baltic region, using Sympower's tech stack and AWS infrastructure. The initiative, sparked by the Baltic Desynchronisation Event, evolves from a quick experiment into a structured project with front-end and back-end development, data storage, and potential expansion to include voltage monitoring.

F-Droid Awarded Open Technology Fund’s FOSS Sustainability Grant | F-Droid - Free and Open Source Android App Repository

F-Droid received a $396,044 grant from Open Technology Fund to address sustainability challenges and enhance its infrastructure. The funding will support code refactoring, legal strategies, localization improvements, and infrastructure enhancements to ensure the platform's long-term viability in delivering FOSS-based, privacy-focused apps globally.