corntoole / Cornelius Toole

There are five people in corntoole’s collective.

Huffduffed (274)

  1. Pluralistic: Podcasting “How To Think About Scraping” (25 Sept 2023) – Pluralistic: Daily links from Cory Doctorow

    Today's links

    Podcasting "How To Think About Scraping": How to preserve the benefits of web-scraping while targeting the real harms.

    Hey look at this: Delights to delectate.

    This day in history: 2003, 2008, 2013, 2018, 2022

    Colophon: Recent publications, upcoming/recent appearances, current writing projects, current reading

    Podcasting "How To Think About Scraping" (permalink)

    This week on my podcast, I read my recent Medium column, "How To Think About Scraping: In privacy and labor fights, copyright is a clumsy tool at best," which proposes ways to retain the benefits of scraping without the privacy and labor harms that sometimes accompany it:

    What are those benefits from scraping? Well, take computational linguistics, a relatively new discipline that is producing the first accounts of how informal language works. Historically, linguists overstudied written language (because it was easy to analyze) and underanalyzed speech (because you had to record speakers and then get grad students to transcribe their dialog).

    The thing is, very few of us produce formal, written work, whereas we all engage in casual dialog. But then the internet came along, and for the first time, we had a species of mass-scale, informal dialog that was also written, and which was born in machine-readable form.

    This ushered in a new era in linguistic study, one that is enthusiastically analyzing and codifying the rules of informal speech, the spread of vernacular, and the regional, racial and class markers of different kinds of speech:

    The people whose speech is scraped and analyzed this way are often unreachable (anonymous or pseudonymous) or impractical to reach (because there's millions of them). The linguists who study this speech will go through institutional review board approvals to make sure that as they produce aggregate accounts of speech, they don't compromise the privacy or integrity of their subjects.

    Computational linguistics is an unalloyed good, and while the speakers whose words are scraped to produce the raw material that these scholars study don't give permission, they probably wouldn't object, either.

    But what about entities that explicitly object to being scraped? Sometimes, it's good to scrape them, too.

    Since 1996, the Internet Archive has scraped every website it could find, storing snapshots of every page it found in a giant, searchable database called the Wayback Machine. Many of us have used the Wayback Machine to retrieve some long-deleted text, sound, image or video from the internet's memory hole.

    For the most part, the Internet Archive limits its scraping to websites that permit it. The robots exclusion protocol (AKA robots.txt) makes it easy for webmasters to tell different kinds of crawlers whether or not they are welcome. If your site has a robots.txt file that tells the Archive's crawler to buzz off, it'll go elsewhere.


    Since 2017, the Archive has started ignoring robots.txt files for news services; whether or not the news site wants to be crawled, the Archive crawls it and makes copies of the different versions of the articles the site publishes. That's because news sites – even the so-called "paper of record" – have a nasty habit of making sweeping edits to published material without noting it.

    I'm not talking about fixing a typo or a formatting error: I'm talking about making a massive change to a piece, one that completely reverses its meaning, and pretending that it was that way all along:

    This happens all the time, with major news sites from all around the world:

    By scraping these sites and retaining the different versions of their article, the Archive both detects and prevents journalistic malpractice. This is canonical fair use, the kind of copying that almost always involves overriding the objections of the site's proprietor. Not all adversarial scraping is good, but this sure is.

    There's an argument that scraping the news-sites without permission might piss them off, but it doesn't bring them any real harm. But even when scraping harms the scrapee, it is sometimes legitimate – and necessary.

    Austrian technologist Mario Zechner used the API from the country's super-concentrated grocery giants to prove that they were colluding to rig prices. By assembling a longitudinal data-set, Zechner exposed the raft of dirty tricks the grocers used to rip off the people of Austria.

    From shrinkflation to deceptive price-cycling that disguised price hikes as discounts:

    Zechner feared publishing his results at first. The companies whose thefts he'd discovered have enormous power and whole kennelsful of vicious attack-lawyers they can sic on him. But he eventually got the Austrian competition bureaucracy interested in his work, and they published a report that validated his claims and praised his work:

    Emboldened, Zechner open-sourced his monitoring tool, and attracted developers from other countries. Soon, they were documenting ripoffs in Germany and Slovenia, too:

    Zechner's on a roll, but the grocery cartel could shut him down with a keystroke, simply by blocking his API access. If they do, Zechner could switch to scraping their sites – but only if he can be protected from legal liability for nonconsensually scraping commercially sensitive data in a way that undermines the profits of a powerful corporation.

    Zechner's work comes at a crucial time, as grocers around the world turn the screws on both their suppliers and their customers, disguising their greedflation as inflation. In Canada, the grocery cartel – led by the guillotine-friendly hereditary grocery monopolilst Galen Weston – pulled the most Les Mis-ass caper imaginable when they illegally conspired to rig the price of bread:

    We should scrape all of these looting bastards, even though it will harm their economic interests. We should scrape them because it will harm their economic interests. Scrape 'em and scrape 'em and scrape 'em.

    Now, it's one thing to scrape text for scholarly purposes, or for journalistic accountability, or to uncover criminal corporate conspiracies. But what about scraping to train a Large Language Model?

    Yes, there are socially beneficial – even vital – uses for LLMs.

    Take HRDAG's work on truth and reconciliation in Colombia. The Human Rights Data Analysis Group is a tiny nonprofit that makes an outsized contribution to human rights, by using statistical methods to reveal the full scope of the human rights crimes that take place in the shadows, from East Timor to Serbia, South Africa to the USA:

    HRDAG's latest project is its most ambitious yet. Working with partner org Dejusticia, they've just released the largest data-set in human rights history:

    What's in that dataset? It's a merger and analysis of more than 100 databases of killings, child soldier recruitments and other crimes during the Colombian civil war. Using a LLM, HRDAG was able to produce an analysis of each killing in each database, estimating the probability that it appeared in more than one database, and the probability that it was carried out by a right-wing militia, by government forces, or by FARC guerrillas.

    This work forms the core of ongoing Colombian Truth and Reconciliation proceedings, and has been instrumental in demonstrating that the majority of war crimes were carried out by right-wing militias who operated with the direction and knowledge of the richest, most powerful people in the country. It also showed that the majority of child soldier recruitment was carried out by these CIA-backed, US-funded militias.

    This is important work, and it was carried out at a scale and with a precision that would have been impossible without an LLM. As with all of HRDAG's work, this report and the subsequent testimony draw on cutting-edge statistical techniques and skilled science communication to bring technical rigor to some of the most important justice questions in our world.

    LLMs need large bodies of text to train them – text that, inevitably, is scraped. Scraping to produce LLMs isn't intrinsically harmful, and neither are LLMs. Admittedly, nonprofits using LLMs to build war crimes databases do not justify even 0.0001% of the valuations that AI hypesters ascribe to the field, but that's their problem.

    Scraping is good, sometimes – even when it's done against the wishes of the scraped, even when it harms their interests, and even when it's used to train an LLM.


    Scraping to violate peoples' privacy is very bad. Take Clearview AI, the grifty, sleazy facial recognition company that scraped billions of photos in order to train a system that they sell to cops, corporations and authoritarian governments:

    Likewise: scraping to alienate creative workers' labor is very bad. Creators' bosses are ferociously committed to firing us all and replacing us with "generative AI." Like all self-declared "job creators," they constantly fantasize about destroying all of our jobs. Like all capitalists, they hate capitalism, and dream of earning rents from owning things, not from doing things.

    The work these AI tools generate sucks, but that doesn't mean our bosses won't try to fire us and replace us with them. After all, prompting an LLM may produce bad screenplays, but at least the LLM doesn't give you lip when you order to it give you "ET, but the hero is a dog, and there's a love story in the second act and a big shootout in the climax." Studio execs already talk to screenwriters like they're LLMs.

    That's true of art directors, newspaper owners, and all the other job-destroyers who can't believe that creative workers want to have a say in the work they do – and worse, get paid for it.

    So how do we resolve these conundra? After all, the people who scrape in disgusting, depraved ways insist that we have to take the good with the bad. If you want accountability for newspaper sites, you have to tolerate facial recognition, too.

    When critics of these companies repeat these claims, they are doing the companies' work for them. It's not true. There's no reason we couldn't permit scraping for one purpose and ban it for another.

    The problem comes when you try to use copyright to manage this nuance. Copyright is a terrible tool for sorting out these uses; the limitations and exceptions to copyright (like fair use) are broad and varied, but so "fact intensive" that it's nearly impossible to say whether a use is or isn't fair before you've gone to court to defend it.

    But copyright has become the de facto regulatory default for the internet. When I found someone impersonating me on a dating site and luring people out to dates, the site advised me to make a copyright claim over the profile photo – that was their only tool for dealing with this potentially dangerous behavior.

    The reasons that copyright has become our default tool for solving every internet problem are complex and historically contingent, but one important point here is that copyright is alienable, which means you can bargain it away. For that reason, corporations love copyright, because it means that they can force people who have less power than the company to sign away their copyrights.

    This is how we got to a place where, after 40 years of expanding copyright (scope, duration, penalties), we have an entertainment sector that's larger and more profitable than ever, even as creative workers' share of the revenues their copyrights generate has fallen, both proportionally and in real terms.

    As Rebecca Giblin and I write in our book Chokepoint Capitalism, in a market with five giant publishers, four studios, three labels, two app platforms and one ebook/audiobook company, giving creative workers more copyright is like giving your bullied kid extra lunch money. The more money you give that kid, the more money the bullies will take:

    Many creative workers are suing the AI companies for copyright infringement for scraping their data and using it to train a model. If those cases go to trial, it's likely the creators will lose. The questions of whether making temporary copies or subjecting them to mathematical analysis infringe copyright are well-settled:

    I'm pretty sure that the lawyers who organized these cases know this, and they're betting that the AI companies did so much sleazy shit while scraping that they'll settle rather than go to court and have it all come out. Which is fine – I relish the thought of hundreds of millions in investor capital being transferred from these giant AI companies to creative workers. But it doesn't actually solve the problem.

    Because if we do end up changing copyright law – or the daily practice of the copyright sector – to create exclusive rights over scraping and training, it's not going to get creators paid. If we give individual creators new rights to bargain with, we're just giving them new rights to bargain away. That's already happening: voice actors who record for video games are now required to start their sessions by stating that they assign the rights to use their voice to train a deepfake model:

    But that doesn't mean we have to let the hyperconcentrated entertainment sector alienate creative workers from their labor. As the WGA has shown us, creative workers aren't just LLCs with MFAs, bargaining business-to-business with corporations – they're workers:

    Workers get a better deal with labor law, not copyright law. Copyright law can augment certain labor disputes, but just as often, it benefits corporations, not workers:

    Likewise, the problem with Clearview AI isn't that it infringes on photographers' copyrights. If I took a thousand pictures of you and sold them to Clearview AI to train its model, no copyright infringement would take place – and you'd still be screwed. Clearview has a privacy problem, not a copyright problem.

    Giving us pseudocopyrights over our faces won't stop Clearview and its competitors from destroying our lives. Creating and enforcing a federal privacy law with a private right of action will. It will put Clearview and all of its competitors out of business, instantly and forever:

    AI companies say, "You can't use copyright to fix the problems with AI without creating a lot of collateral damage." They're right. But what they fail to mention is, "You can use labor law to ban certain uses of AI without creating that collateral damage."

    Facial recognition companies say, "You can't use copyright to ban scraping without creating a lot of collateral damage." They're right too – but what they don't say is, "On the other hand, a privacy law would put us out of business and leave all the good scraping intact."

    Taking entertainment companies and AI vendors and facial recognition creeps at their word is helping them. It's letting them divide and conquer people who value the beneficial elements and those who can't tolerate the harms. We can have the benefits without the harms. We just have to stop thinking about labor and privacy issues as individual matters and treat them as the collective endeavors they really are:

    Here's a link to the podcast:

    And here's a direct link to the MP3 (hosting courtesy of the Internet Archive; they'll host your stuff for free, forever):

    And here's the RSS feed for my podcast:

    (image: syvwlch, CC BY-SA 2.0, modified)

    Hey look at this (permalink)

    Dumping 2 Million Tires In The Ocean To "Help" Fish (h/t Salim Fadhley)

    Writers Strike Ends: WGA, AMPTP Agree to Deal After 146-Day Strike

    This day in history (permalink)

    #20yrsago Epic micropayments rant

    #20yrsago Michael Moore’s comprehensive response to criticisms of Bowling for Columbine

    #20yrsago WKRP in Cincinnati redacted to save on license fees

    #15yrsago Rockbox 3.0: revive old iPod with free/open software

    #15yrsago Judge says that “attempted copyright infringement” is bogus

    #15yrsago HOWTO Make a giant spherical metalamp out of dozens of cheap Ikea lamps

    #15yrsago China’s IP address shortage, two perspectives

    #15yrsago World’s largest wargaming table art installation

    #10yrsago More details, new video showing Iphone fingerprint reader pwned by Chaos Computer Club

    #10yrsago Not Your Ordinary Wolf Girl: fast YA novel with wonderful characters

    #10yrsago Godspeed You! Black Emperor condemns music contest they won, vows to use money to buy instruments for prisoners

    #10yrsago Love Song for Internet Trolls

    #10yrsago Adding some evidence to copyright’s “evidence-free zone”

    #10yrsago Beijing’s “mystery rooms”: single-room funhouses

    #10yrsago Easyjet tells law professor he can’t fly because he tweeted critical remarks about airline

    #10yrsago The Coldest Girl in Coldtown: dangerous, bloody vampire YA novel

    #5yrsago Big Tech is building a $80B capex wall around its empire

    #5yrsago A CRISPR-based hack could eradicate malaria-carrying mosquitoes

    #5yrsago There’s a literal elephant in machine learning’s room

    #5yrsago To fix Canadian copyright, let creators claim their rights back after 25 years

    #5yrsago The world’s richest families got MUCH richer, thanks to the stock market

    #5yrsago DNA ancestry tests are bullshit

    #5yrsago Incredibly sensible notes on software engineering, applicable to the wider world

    #5yrsago Hank Green’s “An Absolutely Remarkable Thing”: aliens vs social media fame vs polarization

    #5yrsago Jewelry in the shape of gerrymandered US congressional districts

    #5yrsago Facebook reminds America’s cops that they’re not allowed to use fake accounts

    #5yrsago Record numbers of people have downloaded and used the Democrats’ mobile app for doorknocking canvassers

    #5yrsago Canada’s legal weed stock-bubble is a re-run of the dotcom bubble

    #1yrago Billionaire grifters hate her: Cometh the Hour, Cometh the Ida M Tarbell

    #1yrago McKinsey and Providence colluded to force poor patients into destitution

    Colophon (permalink)

    Today's top sources:

    Currently writing:

    A Little Brother short story about DIY insulin PLANNING

    Picks and Shovels, a Martin Hench noir thriller about the heroic era of the PC. FORTHCOMING TOR BOOKS JAN 2025

    The Bezzle, a Martin Hench noir thriller novel about the prison-tech industry. FORTHCOMING TOR BOOKS FEB 2024

    Vigilant, Little Brother short story about remote invigilation. FORTHCOMING ON TOR.COM

    Moral Hazard, a short story for MIT Tech Review's 12 Tomorrows. FIRST DRAFT COMPLETE, ACCEPTED FOR PUBLICATION

    Spill, a Little Brother short story about pipeline protests. FORTHCOMING ON TOR.COM

    Latest podcast: Plausible Sentence Generators

    Upcoming appearances:

    Launch for "The Internet Con" and Brian Merchant's "Blood in the Machine," Chevalier's Books (LA), Sept 27

    An Evening with VE Schwab (Boise), Oct 2

    Wired Nextfest (Milano), Oct 7-8

    The Internet Con at Moon Palace Books (Minneapolis), Oct 15

    26th ACM Conference On Computer-Supported Cooperative Work and Social Computing keynote (Minneapolis), Oct 16

    41st annual McCreight Lecture in the Humanities (Charleston, WV), Oct 19

    Seizing the Means of Computation (Edinburgh Futures Institute), Oct 25

    Recent appearances:

    Words, Images, & Worlds

    Against Enshittification | Medium Day 2023

    The Jim Rutt Show

    Latest books:

    "The Internet Con": A nonfiction book about interoperability and Big Tech (Verso) September 2023 ( Signed copies at Book Soup (

    "Red Team Blues": "A grabby, compulsive thriller that will leave you knowing more about how the world works than you did before." Tor Books Signed copies at Dark Delicacies (US):

    and Forbidden Planet (UK):

    "Chokepoint Capitalism: How to Beat Big Tech, Tame Big Content, and Get Artists Paid, with Rebecca Giblin", on how to unrig the markets for creative labor, Beacon Press/Scribe 2022

    "Attack Surface": The third Little Brother novel, a standalone technothriller for adults. The Washington Post called it "a political cyberthriller, vigorous, bold and savvy about the limits of revolution and resistance." Order signed, personalized copies from Dark Delicacies

    "How to Destroy Surveillance Capitalism": an anti-monopoly pamphlet analyzing the true harms of surveillance capitalism and proposing a solution. (print edition: (signed copies:

    "Little Brother/Homeland": A reissue omnibus edition with a new introduction by Edward Snowden:; personalized/signed copies here:

    "Poesy the Monster Slayer" a picture book about monsters, bedtime, gender, and kicking ass. Order here: Get a personalized, signed copy here:

    Upcoming books:

    The Lost Cause: a post-Green New Deal eco-topian novel about truth and reconciliation with white nationalist militias, Tor Books, November 2023

    The Bezzle: a sequel to "Red Team Blues," about prison-tech and other grifts, Tor Books, February 2024

    Picks and Shovels: a sequel to "Red Team Blues," about the heroic era of the PC, Tor Books, February 2025

    Unauthorized Bread: a graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2025

    This work – excluding any serialized fiction – is licensed under a Creative Commons Attribution 4.0 license. That means you can use it any way you like, including commercially, provided that you attribute it to me, Cory Doctorow, and include a link to

    Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.

    How to get Pluralistic:

    Blog (no ads, tracking, or data-collection):

    Newsletter (no ads, tracking, or data-collection):

    Mastodon (no ads, tracking, or data-collection):

    Medium (no ads, paywalled):

    (Latest Medium column: "How To Think About Scraping: In privacy and labor fights, copyright is a clumsy tool at best

    Twitter (mass-scale, unrestricted, third-party surveillance and advertising):

    Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):

    "When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla

    Like this:Like Loading…

    —Huffduffed by corntoole

  2. DEF CON 31 - An Audacious Plan to Halt the Internet’s Ensh*ttification - Cory Doctorow

    The enshittification of the internet follows a predictable trajectory: first, platforms are good to their users; then they abuse their users to make things better for their business customers; finally, they abuse those business customers to claw back all the value for themselves. Then, they die. It doesn't have to be this way. Enshittification occurs when companies gobble each other up in an orgy of mergers and acquisitions, reducing the internet to "five giant websites filled with screenshots of text from the other four" (credit to Tom Eastman!), which lets them endlessly tweak their back-ends to continue to shift value from users and business-customers to themselves. The government gets in on the act by banning tweaking by users - reverse-engineering, scraping, bots and other user-side self-help measures - leaving users helpless before the march of enshittification. We don't have to accept this! Disenshittifying the internet will require antitrust, limits on corporate tweaking - through privacy laws and other protections - and aggressive self-help measures from alternative app stores to ad blockers and beyond!

    Original video:
    Downloaded by on Sat Sep 16 19:33:24 2023 Available for 30 days after download

    —Huffduffed by corntoole

  3. Daniel Burka at dConstruct 2022

    Daniel Burka is a Canadian digital designer who has previously worked in the agency world, at Silicon Valley startups, and even venture capital. But now he’s doing truly meaningful work, designing for busy healthcare workers in low-income countries.

    Daniel previously spoke at dConstruct back in 2008, when he gave a talk called Designing for Interaction.

    Daniel spoke at the final dConstruct, a special one-off anniversary edition held in 2022. The theme of the day was design transformation.

    —Huffduffed by corntoole

  4. Matt Webb at dConstruct 2022

    Matt Webb is a technologist, product designer, and writer who defies categorisation. He has headed up a design studio, co-founded a start-up, and now consults on super-clever machine learning stuff. His blog is brilliant.

    Matt previously spoke at dConstruct back in 2007, when he gave a talk called The Experience Stack.

    Matt spoke at the final dConstruct, a special one-off anniversary edition held in 2022. The theme of the day was design transformation.

    —Huffduffed by corntoole

  5. Base Training Fundamentals: 3 Key Ingredients

    Base training helps build your foundation for a fast, healthy season!

    Season Planner Worksheet: Subscribe: More base training:

    Base training is critical for endurance runners. This valuable phase of training helps build your foundation of aerobic development, strength, and neuromuscular fitness. Without base training, your season won't be as successful.

    Plan your season right with our free worksheet:

    Jason Fitzgerald is a USATF running coach, 2:39 marathoner, and the host of the award-winning Strength Running Podcast. He's the 2017 Men's Running Magazine's Influencer of the Year and his work has appeared in Runner's World, Health Magazine, The Washington Post, Lifehacker, and other major media.

    Visit to see their award-winning blog, free email courses, and full library of training programs and coaching services.

    Podcast: Twitter: Instagram: Facebook:

    Original video:
    Downloaded by on Tue Nov 23 21:04:57 2021 Available for 30 days after download

    —Huffduffed by corntoole

  6. Productive Talk Compilation: 8-episode podcast with GTD’s David Allen | 43 Folders

    Download MP3 of "Productive Talk Compilation" As promised, here's the single-file compilation of the Productive Talk podcast interviews I did with David Allen, the author of Getting Things Done. The final version's eight episodes clock in at a

    —Huffduffed by corntoole

  7. Panic Podcast - Episode 6: The Story of Playdate

    Find out how an offhand idea to mark the 15th anniversary of a software company launched a decade-long saga of twists, turns, and mini-boss battles that led to the creation of a handheld gaming console as surprising and unique as its creators. Pre-order day is finally here, but it’s dangerous to go alone. Take this adorable yellow box, and let’s get crankin’… on the story of Playdate.

    —Huffduffed by corntoole

  8. Ashley Williams — Why the #wasmsummit Website isn’t written in Wasm

    WebAssembly is not here to kill JavaScript. In fact, to be successful, it must not. But let me back up.

    WebAssembly is an exciting new technology that has the ambition to change how and what we program for not only the web, but everywhere. In the case of the web platform, WebAssembly's promise has led many to declare that WebAssembly's entrance means the death of JavaScript. This belief is not only reactionary, but deeply short-sighted, and likely to threaten the successful wide-spread adoption of WebAssembly.

    In this talk, we'll use the WebAssembly Summit website to discuss the uses and misuses of WebAssembly on the web. We'll explore the historical and material conditions of the web, past and present, to understand how and why the web changes and what its current trajectory is. With this understanding, we'll explore how WebAssembly can navigate this unique moment and discuss the practical implications of the specification's growth and better tooling as WebAssembly searches for its place in the web platform and beyond.

    Original video:
    Downloaded by on Sat Dec 19 23:07:33 2020 Available for 30 days after download

    —Huffduffed by corntoole

Page 1 of 28Older