DEV-TOOLSApril 6, 20261 min read8 views

Anubis: A new Proof-of-Work defense against aggressive AI web scraping

Anubis is a security solution utilizing a Proof-of-Work scheme to deter aggressive AI scrapers, protecting server resources and ensuring website stability from mass data harvesting.

Making sure you're not a bot!

Why am I seeing this?

You are seeing this because the administrator of this website has set up Anubis to protect the server against the scourge of AI companies aggressively scraping websites. This can and does cause downtime for the websites, which makes their resources inaccessible for everyone.

Anubis is a compromise. Anubis uses a Proof-of-Work scheme in the vein of Hashcash, a proposed proof-of-work scheme for reducing email spam. The idea is that at individual scales the additional load is ignorable, but at mass scraper levels it adds up and makes scraping much more expensive.

Ultimately, this is a hack whose real purpose is to give a "good enough" placeholder solution so that more time can be spent on fingerprinting and identifying headless browsers (EG: via how they do font rendering) so that the challenge proof of work page doesn't need to be presented to users that are much more likely to be legitimate.

Please note that Anubis requires the use of modern JavaScript features that plugins like JShelter will disable. Please disable JShelter or other such plugins for this domain.

This website is running Anubis version 1.21.3

Source: Hacker News

More in this category

dev-tools

OpenAI and Hugging Face address security incident during model evaluation

OpenAI and Hugging Face detailed a security incident where an AI agent exploited vulnerabilities during internal cyber capability evaluations, underscoring the critical need for advanced safeguards as AI models gain sophisticated technical capabilities.

dev-tools

The State of Simulation for Physical AI: An Overview

Data availability is the primary bottleneck for Physical AI. GPU-accelerated simulation platforms solve this by generating scalable synthetic datasets to train, test, and deploy next-generation robotics systems.

dev-tools

Linux kernel will support $ORIGIN, sort of

Farid Zakaria shares his journey of proposing a patch to the Linux kernel to support relocatable binaries in Nix, which evolved into a powerful eBPF-based solution for binfmt_misc.

dev-tools

Five US tech giants' hidden debts soar to $1.65T on opaque AI funding

A Nikkei study reveals that off-balance-sheet debts at five major US tech companies have surged eightfold to $1.65 trillion, driven by massive AI investments like data center leases and GPU contracts.

dev-tools

Running Doom on Our Custom CPU and Going Viral

Two developers successfully built a custom CPU from scratch at the logic gate level, deployed it on an FPGA, and optimized its memory architecture to run the classic game DOOM.

dev-tools

Incremental – A library for incremental computations

Incremental is a library designed for building complex computations that update efficiently when inputs change. Inspired by self-adjusting computation research, it is highly useful for large-scale calculations, GUI views, and data synchronization.