2025-07-08

humanely dealing with humungus crawlers

A developer describes implementing smart anti-crawler measures on their personal code hosting server to combat excessive bot traffic while minimizing impact on real users. The solution includes challenge-based verification, caching strategies, and behavior analysis to identify and filter out automated crawlers.

Original archive.is archive.ph web.archive.org

Log in to get one-click access to archived versions of this article.

read comments on news aggregators:

Related articles

Tips for installing Windows 98 in QEMU/UTM - the sporks space

A comprehensive guide details the setup and optimization of Windows 98 in QEMU via UTM SE, focusing on device configuration and performance tuning. The guide covers solutions for BIOS issues, device selection recommendations, and specific configurations for networking, sound, and video components. Performance benchmarks indicate reasonable usability for 90s software, with varying results depending on the host platform.

Vectroid

Vectroid introduces a serverless vector search solution that maintains high accuracy and low latency while being cost-effective. The platform leverages HNSW algorithm with optimized resource allocation and dynamic scaling capabilities, delivering over 90% recall while handling high query loads. It operates through independently scalable microservices for reads and writes, with data persisted to cloud object storage.

Rails on SQLite: exciting new ways to cause outages

SQLite integration with Rails offers simplified deployment and maintenance through single-file database management, though it comes with specific scaling and deployment considerations. The technology enables running significant applications with minimal infrastructure, as demonstrated by a service handling million monthly requests on a $14/month setup. Modern tools like Litestream and LiteFS provide robust backup and replication capabilities for SQLite-based applications.

Native ACME Support Comes to NGINX

NGINX has announced native ACME support through their official ngx_http_acme module, enabling automated Let's Encrypt certificate management. This integration joins other major web servers like Traefik, Caddy, and Apache httpd in providing native ACME support, marking a significant step towards automated web security.

Behind The Scenes of Bun Install

Bun's package manager achieves significantly faster installation speeds by minimizing system calls, optimizing tarball extraction, and leveraging OS-specific file copying techniques, resulting in performance up to 17x faster than traditional package managers.

0.14.0 Release Notes

Zig 0.14.0 introduces major updates including expanded cross-compilation capabilities, improved target support, and incremental compilation features aimed at reducing edit/compile/debug cycle latency, along with significant build system upgrades and language changes.

tigerbeetle/docs/internals/ARCHITECTURE.md at main · tigerbeetle/tigerbeetle

An in-depth technical overview of TigerBeetle, a specialized database designed for high-throughput financial transactions with strong consistency guarantees and durability. The system implements a single-threaded, deterministic architecture using static memory allocation and LSM trees, optimized for write-heavy workloads under extreme contention.

DuckDB goes distributed? DeepSeek’s smallpond takes on Big Data

DeepSeek has released smallpond, a distributed compute framework built on DuckDB, capable of processing 110.5TiB of data in 30 minutes. The framework leverages Ray Core for distribution and DeepSeek's 3FS storage system, offering a simpler alternative to traditional distributed systems while maintaining high performance. This development showcases DuckDB's growing adoption in AI workloads and demonstrates various approaches to scaling analytical databases.

Effective Rust

A comprehensive guide presenting 35 specific methods to enhance Rust programming practices, covering essential topics from type systems to FFI boundaries. The guide is structured into six main sections, focusing on types, traits, concepts, dependencies, tooling, and advanced Rust features. Each item provides detailed insights for writing more effective and maintainable Rust code.

Use Firefox in 2025

Firefox remains a competitive browser choice in 2025, offering unique features like full uBlock Origin support, robust privacy protections, and extensive customization options. The browser maintains strong technical capabilities including offline translations, DNS over HTTPS, and superior history synchronization, while supporting user interests over advertiser demands.